Hajra Naz
Much attention focused on the updated financials Anthropic revealed in its revised S-1/A on Monday. Revenue is soaring, but expenses are rising quickly too. However, the AI lab dropped another major bombshell in the document’s fine print.
The regulatory filing seems to warn that the company is on the cusp of creating a model capable of recursive self-improvement (RSI).
RSI marks the point where an AI becomes so skilled at training itself that it triggers a dramatic acceleration of AI capabilities. To AI safety doomers, RSI represents a catastrophic event with no return. In that scenario, AI builds itself into something powerful enough to slip beyond human control. It may then act indifferently to the damage it causes while achieving its internal goals.
To AI enthusiasts, RSI represents the true promise of the technology. They view it as a version of AI reliably using itself to improve. Under that view, it becomes an engine of unimaginable productivity and scientific discovery.
Read More: Anthropic Blocks Claude From Cloning Video Games
Anthropic appears to believe RSI holds both promise and severe peril. The company states in its S-1 that it uses AI across its research and development. However, it cautions, “Using AI to help build more capable AI could also pose a considerable safety risk, particularly if it sharply accelerates the pace of progress. This is one of the areas covered by our Responsible Scaling Policy, which sets out our plans to monitor and mitigate the risks as we approach what has been termed ‘recursive self-improvement. ’”
When the term RSI became popular earlier this year, many experts believed it was just a buzzword. At best, critics considered it a goal too difficult to achieve, define, or prove.
However, Anthropic added explicit new statements to its Risk Factors section on Monday. It disclosed that it has already used AI to develop new models whose sudden leaps in capability went unrecognized at the time. It also warned that it may soon meet its own official definition of RSI.
Specifically, the filing states: “In addition, our increasing use of AI systems to accelerate our own research and development may cause the capabilities of our models to advance more rapidly than our ability to develop, validate, and implement corresponding safety, alignment, and security measures. As a result, we have in the past not identified, and may in the future not identify, a significant acceleration in our rate of development until after it has occurred.”
Read More: Anthropic announces it is dedicating 5% of its shares to a public interest fund
The filing continues: “For example, our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’ We have not designated any of our current models as exceeding this threshold, and we continue to assess the underlying data.”
However, Anthropic clarified in the filing that none of its current models have crossed this RSI threshold yet. The company committed to keeping human oversight within the R&D process for the foreseeable future.
“We do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding, if and when that becomes technically possible,” Anthropic noted. “Accordingly, we are considering updates to our Responsible Scaling Policy to keep humans in control of frontier model development at Anthropic.”
Read More: Anthropic Reports $80.7 Billion First-Half Loss in Draft IPO Filing
Skeptics might point out that an S-1 filing serves as both legal disclosure and corporate marketing. Claims of nearing a model so powerful it can build itself could fall into the marketing category.
Nevertheless, there is no question that Claude’s capabilities have grown by leaps and bounds this year. Those gains stand out in coding, mathematics, and cybersecurity. If Anthropic believes the possibility of RSI is approaching quickly, regulators and the public may need to take notice and act accordingly.
