TechnologyTop Stories
Can ChatGPT pass the CFA exam? AI researchers tried to find out
Relax, Wall Street. chatgpt It is still a long way from passing the Chartered Financial Analyst exam and threatening the jobs of financial professionals across the world.
A team of JPMorgan Chase & Co. researchers and university academics tested whether OpenAI’s ChatGPT and GPT-4 chatbots would have a chance of passing the first two levels of the exam. It typically takes humans four years to complete all three levels of testing, which can lead to higher pay and better job opportunities.
“Based on estimated passing rates and average self-reported scores, we concluded that ChatGPT would likely not be able to pass CFA Level I and Level II under all testing settings,” the researchers wrote in the 11-page report. ” 4 will have a good chance of passing CFA Level I and Level II if indicated.
The researchers include academics and six employees of JPMorgan’s AI research organization, including Samina Shah and Antony Papadimitriou.
CFA Institute, which provides credentials, has spent years refining its tests to ensure that professionals looking to advance their careers are familiar with the forces driving automation. The institute announced that it will add questions on artificial intelligence and big data analysis methods to its exams in 2017.
Chris Wiese, managing director of education at the CFA Institute, acknowledged that larger language models will have the ability to answer some exam questions correctly.
Wiese said, “Although multiple-choice exams and essay questions are excellent ways to assess learning and understanding in a safe environment, the day-to-day in finance does not simply present itself as a series of short, standalone questions. ” “That’s why to become a CFA charterholder, we also require 4,000 hours of qualifying work experience, a minimum of two references, a strong ethical guideline and, soon, completion of practical skills modules.”
He said the company is also looking at using large language model technology to aid CFA candidates’ learning.
Every few months, thousands of candidates sit for three different levels of the exam. Recipients of the charter typically spend more than 300 hours studying for each level of the exam.
Pass rates for the exam have declined in recent years, with the average pass rate for the first level of the exam reaching 37% in August, compared to an average of 43% in 2018.
common errors
Level I of the CFA consists of 180 multiple-choice questions, while Level II consists of case studies and 88 multiple-choice questions. The researchers found that both major language models struggled more at Level II, no matter the type of sign used.
However, in Level I, both ChatGPT and GPT-4 performed best in the sections of the test focused on derivatives, alternative investments, corporate issuers, equity investments, and ethics. However, both chatbots performed relatively poorly on those focused on financial reporting and portfolio management.
In Level II, ChatGPT struggled compared to GPT-4 on sections focused on alternative investments and fixed income instruments, while ChatGPT performed better in areas involving portfolio management and economics.
Most of ChatGPT’s errors were knowledge-based, while GPT-4 had the most computation-related errors.
“One error type that GPT-4 made more frequently than ChatGPT was logic errors,” the researchers found. “It appears that, with GPT-4’s greater ability to reason, it is more likely to ‘talk itself’ into incorrect lines of reasoning.”
A team of JPMorgan Chase & Co. researchers and university academics tested whether OpenAI’s ChatGPT and GPT-4 chatbots would have a chance of passing the first two levels of the exam. It typically takes humans four years to complete all three levels of testing, which can lead to higher pay and better job opportunities.
“Based on estimated passing rates and average self-reported scores, we concluded that ChatGPT would likely not be able to pass CFA Level I and Level II under all testing settings,” the researchers wrote in the 11-page report. ” 4 will have a good chance of passing CFA Level I and Level II if indicated.
The researchers include academics and six employees of JPMorgan’s AI research organization, including Samina Shah and Antony Papadimitriou.
CFA Institute, which provides credentials, has spent years refining its tests to ensure that professionals looking to advance their careers are familiar with the forces driving automation. The institute announced that it will add questions on artificial intelligence and big data analysis methods to its exams in 2017.
Chris Wiese, managing director of education at the CFA Institute, acknowledged that larger language models will have the ability to answer some exam questions correctly.
Wiese said, “Although multiple-choice exams and essay questions are excellent ways to assess learning and understanding in a safe environment, the day-to-day in finance does not simply present itself as a series of short, standalone questions. ” “That’s why to become a CFA charterholder, we also require 4,000 hours of qualifying work experience, a minimum of two references, a strong ethical guideline and, soon, completion of practical skills modules.”
He said the company is also looking at using large language model technology to aid CFA candidates’ learning.
Every few months, thousands of candidates sit for three different levels of the exam. Recipients of the charter typically spend more than 300 hours studying for each level of the exam.
Pass rates for the exam have declined in recent years, with the average pass rate for the first level of the exam reaching 37% in August, compared to an average of 43% in 2018.
common errors
Level I of the CFA consists of 180 multiple-choice questions, while Level II consists of case studies and 88 multiple-choice questions. The researchers found that both major language models struggled more at Level II, no matter the type of sign used.
However, in Level I, both ChatGPT and GPT-4 performed best in the sections of the test focused on derivatives, alternative investments, corporate issuers, equity investments, and ethics. However, both chatbots performed relatively poorly on those focused on financial reporting and portfolio management.
In Level II, ChatGPT struggled compared to GPT-4 on sections focused on alternative investments and fixed income instruments, while ChatGPT performed better in areas involving portfolio management and economics.
Most of ChatGPT’s errors were knowledge-based, while GPT-4 had the most computation-related errors.
“One error type that GPT-4 made more frequently than ChatGPT was logic errors,” the researchers found. “It appears that, with GPT-4’s greater ability to reason, it is more likely to ‘talk itself’ into incorrect lines of reasoning.”
#ChatGPT #pass #CFA #exam #researchers #find
