AI Is Going Just Great
← Timeline
·1w agoConcerningMajoranthropic

Anthropic Declares Its Own Recursive Self-Improvement Threshold Crossed, Mostly Based on Expectations

Published · updated · curated by AI Is Going Just Great

Source: gizmodo.com ↗ · archived copy ↗

"we made a determination that we have met this threshold based on what we expect from future models... rather than on what we have observed from current models."

Anthropic disclosed in its SEC S-1 filing that it has formally determined it has crossed the "AI research and development capability threshold" defined in its Responsible Scaling Policy — the trigger point for precautions around recursive self-improvement. The determination is based not on anything the company has observed in current models, but on "what we expect from future models and our overall rate of progress." Between the November 2 and November 10 versions of the S-1, Anthropic quietly removed the phrase "dramatically accelerated progress attributable to AI" while simultaneously concluding that the section of its policy governing exactly that condition now applies.

What crossing the threshold actually requires Anthropic to do: explore "unprecedented levels of security," achieve total visibility into model behavior it calls "eyes on everything," and develop red-teaming capable enough to make bug bounties unnecessary. Whether Anthropic can self-red-team its way to safety is a question its own founders' letter raises obliquely, warning that building AI safely may require the industry "to modulate the extremely fast pace at which AI capabilities are compounding, especially when this is driven by AI systems building themselves." Anthropic shares are expected to begin trading Thursday.