Nathaniel Whittemore

This paper introduces a quantifiable framework to address this, defining AGI as matching the cognitive versatility and proficiency of a well-educated adult.

This group, then, has grounded their analysis in Cattell-Horne-Carroll theory, one of the more well-accepted models of human cognition.

Applying the theory, the researchers split AI performance into 10 distinct categories.

Reading and writing, math, reasoning, working memory, memory storage, memory retrieval, visual, auditory, speech, and knowledge.

Now, you'll note that these categories cover some of the general performance categories, things like reading and writing or math, but it also addresses a model's ability to learn and apply its intelligence to topics outside of its training data.

Each of these categories has multiple subcategories that can be assessed individually.

In fact, assessment was one of the main focuses of this paper.

Researchers wrote, "...applications of this framework reveal a highly jagged cognitive profile in contemporary models."

While proficient in knowledge-intensive domains, current AI systems have critical deficits in foundational cognitive machinery, particularly long-term memory storage.

Each category was equally weighted and given a score out of 10, and researchers measured GPT-4 and GPT-5 to demonstrate the framework.

GPT-4 scored 27%, while GPT-5 achieved a 58%.

You can see from the two sets of results mapped out on a chart that while GPT-5 only made minor progress in knowledge, it made significantly more progress in reading and writing as well as math.

What's more, GPT-5 scored in multiple categories where GPT-4 was entirely deficient.

This included reasoning, working memory, memory retrieval, visual, and auditory.

And while those areas of intelligence are developing in the latest models, they're still very nascent compared to, for example, math.

Dan Hendricks, the director of the Center for AI Safety, commented, "...people who are bullish about AGI timelines rightly point to rapid advancements like math."

The skeptics are correct to point out that AIs have many basic cognitive flaws.

Hallucinations, limited inductive reasoning, limited world models, no continual learning.

There are many barriers to AGI, but they each seem tractable.

It seems like AGI won't arrive in a year, but it could easily arrive this decade.