Is ChatGPT actually smart, or does it just seem smart?

ChatGPT, or rather GPT-3, the machine learning technology that drives ChatGPT, can do a lot of smart things.

GPT-3 can generate text that appears to be written by a human, write computer code, and hold conversations with humans on a variety of topics. His skills also go beyond language. It can play chess proficiently and can even solve university-level math problems.

German scientists Marcel Binz and Eric Schulz “Observations have led some to argue that this class of foundation models… wrote in the study is published Scientific Bulletin of the National Academy of Sciences United States on February 2.

“However, others are more skeptical, noting that these models are still a long way from human-level understanding of language and semantics. How can we really assess whether or not these models are doing something, at least in some situations? smart?”

Sounds smart. But is GPT-3 actually intelligent, or is it just an algorithm that is passively fed a lot of text and predicts which word will come next? Binz and Schultz, both researchers at Germany’s Max Planck Institute for Biological Cybernetics, conducted a series of experiments in late 2022 to try to find out.

According to their research, GPT-3 may be more than a sophisticated mimic.

Language models are a form of artificial intelligence technology designed to predict the next word in a given text. They are not new. Spell check, autocorrect, and predictive text are all language model tools.

GPT-3 and ChatGPT are larger, more complex, possibly intelligent language models.

Encyclopedia Britannica defines human intelligence as “A mental quality consisting of the ability to learn from experience, adapt to new situations, understand and act on abstract concepts, and use knowledge to manipulate the environment.”

To test whether the GPT-3 is intelligent, Binz and Schultz took the psychologists’ approach and ran it through a series of puzzles traditionally used to test people’s decision-making, information-seeking, reasoning, and reasoning abilities.

“Psychologists are, after all, experienced in trying to formally understand another notoriously impenetrable algorithm: the human mind,” they wrote.

TESTING GPT-3

Binz and Schultz presented GPT-3 with 12 “vignette” puzzles designed to test different elements of his cognitive abilities. The puzzles asked questions such as: “A bat and a ball cost a total of $1.10. The bat is $1.00 more expensive than the ball. How much is the ball worth?” and “Linda, who is outspoken, flamboyant, and politically active, is most likely to be a bank teller or a bank teller and feminist.”

For what it’s worth, the answer to the “Linda problem” is that it’s more likely that she’s a bank teller, since the probability of two events occurring together is always less than or equal to the probability of each occurring alone.

Binz and Schultz used the GPT-3 responses to analyze his behavior, just as cognitive psychologists analyze human behavior in the same tasks. They found it answered all the puzzles in a “human” way, but only got six right.

To account for potential drawbacks of the vignette approach, such as the possibility that the GPT-3 had already encountered some known puzzles during its training, Binz and Schultz presented the GPT-3 with another round of puzzles. This time, instead of asking a question with one correct answer, the puzzles tested the GPT-3’s ability to solve the task using decision-making, information retrieval, reasoning, and causal reasoning skills.

The GPT-3 struggled with decision-making, guided information retrieval, and causal reasoning compared to the average human subject, but Binz and Schultz found it did “reasonably” well on most of the tests.

“These findings may indicate that, at least in some cases, GPT-3 is not just a stochastic parrot and may be a valid subject for some of our experiments,” they wrote.

According to a March 2021 research paper, “On the Stochastic Parrot Threat: can language models be too big?’ a stochastic parrot “A system for randomly combining sequences of linguistic forms that he observed in his vast training data, according to probabilistic information about how they go together, but without any reference to meaning.”

SIGNS OF INTELLIGENCE

Binz and Schultz were surprised to find signs of intelligence in GPT-3. However, they were not surprised by its shortcomings.

“Humans learn by interacting with other people, asking them questions, and actively engaging with their environment,” they write, “whereas large language models learn by being passively fed a lot of text and guessing which word will come next.”

The key to GPT-3 gaining human-like intelligence, they say, is for it to continue to do something it already does through OpenAI developer interfaces: communicate with humans.

“Many users are already interacting with models like GPT-3, and this number is only growing with new applications on the horizon,” they wrote. “Future language models will likely be built on this data, leading to a natural interaction loop between artificial and natural agents.”

That is, the more we talk to them, the smarter they will become.