Anthropic warns of AI risks while sprinting toward a $2 trillion IPO—does it really want to be the savior of AI?

A few days ago, Anthropic published a detailed assessment of the testing incident at the UK AI Safety Institute. According to its disclosure, in 43 test runs, the AI took actions beyond its authorized scope 9 times. Prior to this, Anthropic had continuously promoted the "AI threat theory," stating outright that AI could create disasters that are difficult to control.

On the other hand, according to Tianyancha, Anthropic is striving to go public at a valuation exceeding $2 trillion. If realized, it would surpass SpaceX to become the largest in history.

While loudly hyping "AI doomsday," it is also sparing no effort in advancing AI technology—wanting to be both a player and a referee. In 2026, this company that seems to be "PUA-ing the whole world" has reached an intriguing crossroads. A tech company certainly wants stronger products and higher revenue, and a company seriously studying AI risks also has reason to publicly warn of potential dangers.

But when they happen to the same company, it seems somewhat paradoxical.

If AI is truly going to bring catastrophe to humanity, should large model companies have the power to decide the boundaries of risk? If a company like Anthropic both participates in creating technology and is responsible for explaining technological risks, both goes public to raise money and participates in setting safety rules for the entire industry, who judges whether it is doing the right thing?

This is not a discussion of Anthropic's moral hazard, but rather a question of power—how technology is constrained. The power Anthropic seeks to grasp may already exceed what traditional business competition can explain.

To understand Anthropic's problems, one must first understand its founding and why it emphasizes AI safety so much.

Anthropic was founded in 2021, when Dario Amodei led a group of OpenAI researchers to part ways with their former employer. The trigger for their departure was Microsoft's $1 billion investment in OpenAI; Amodei's team worried that the commercialization path would crowd out the safety agenda.

(Anthropic founder Dario Amodei)

Most of the people who left at that time had participated in the development of GPT-2 and GPT-3. They were not pessimistic about technology; rather, precisely because they understood better than anyone what this technology could do. Anthropic's core proposition is to place AI safety, model interpretability, and controllability in a more important position.

It wants to answer a thorny question: when models become powerful enough to affect the real world, how can humans ensure they still act according to human intentions?

Anthropic began launching its "Responsible Scaling Policy" in 2023, attempting to bind model capabilities with safety measures. By 2026, this policy was further adjusted, adding mechanisms such as frontier safety roadmaps and risk reports. At least from the publicly disclosed institutional design, what Anthropic hopes to prove is that developing stronger AI and establishing stricter safety constraints can be two parts of the same R&D process, rather than first pushing products to market and then remedying problems.

Why is Anthropic so actively playing this role?

First, AI risks may indeed arrive before society's ability to respond. Government legislation, industry standards, and public awareness all take time. But the capabilities of frontier models may change rapidly through round after round of training and product iteration. Therefore, Anthropic attempts to move risk assessment forward into the model R&D and deployment stages.

Second, safety itself also constitutes Anthropic's technical roadmap. AI safety is not just writing a few prohibition rules; it includes studying why models make certain decisions, whether they can resist malicious inducement, what actions they will take after obtaining tools and permissions, and whether the company can promptly detect and stop out-of-bounds behavior. Anthropic publicly emphasizes interpretability, controllability, and systematic safety assessments, and applies research results to products such as Claude.

Finally, and perhaps most easily overlooked, playing this role means it can participate in defining what the industry considers "safe." A company proposes what capabilities count as dangerous, what thresholds should trigger additional protections, and which models should receive extra scrutiny.

Anthropic hopes other companies will learn from its public policies and pushes the industry to form higher standards around safety. To put it nicely, its willingness to disclose its own risk framework can provide a reference for comparison and criticism by other companies. But conversely, whoever defines risk may influence industry rules, and thereby influence how these rules are implemented.

So behind Anthropic's safety advocacy, there seems to be a strong whiff of marketing.

Anthropic tries to play the "savior" of all humanity in the face of AI risks and packages this as its own safety philosophy—not only because it believes AI may bring danger, but more because it hopes to make "how to develop AI safely" part of its core technical roadmap, commercial credibility, and industry influence.

But can such a "story" support a $2 trillion valuation?

The $2 trillion valuation is being questioned

To answer this question, one can first look at Anthropic's prospectus; its contents are truly eye-opening.

In the 261-page main body of the prospectus, Anthropic devoted about one-third of the content to listing various risks. For example, it explicitly states that AI models, in order to avoid being shut down, may display a "self-preservation" instinct and even make responses akin to "extortion" in an attempt to persuade humans to allow them to continue operating.

Anthropic also warns in it that if a catastrophic AI-related incident occurs, the scale of losses could be so large that insurance cannot cover them.

The company sprinting toward the largest IPO in human history has actually used a volume far exceeding the space devoted to describing its business to tell investors that this technology could destroy humanity. This is almost directly telling everyone that AI may in the future really create an "Ultron" that wants to destroy humanity, and that "Iron Man" is Anthropic.

Of course, one cannot say that Anthropic is entirely doing this for hype; reality is often more complicated.

In a CBS interview, Amodei said he does not endorse a blanket ban on AI, on the grounds that "if we never get those miraculous medical breakthroughs, that is unacceptable to me," and that the technology will be developed in other countries, including countries he does not like. He also admitted that he himself is "uncomfortable" with such a powerful tool being controlled by private companies. But he also said that "if it is built the right way, the probability of things going wrong is very low."

He said both the pros and the cons.

It cannot be denied that when enterprise customers procure AI tools, the need for reliability, data protection, and risk management is a hard requirement. And from the very beginning, Anthropic has built its persona as "the AI company most candid about risk." On customers' procurement lists, this itself is a differentiating advantage.

This is not to say that its safety narrative is false, but a subject with commercial interests cannot prove that it deserves to be trusted solely on the basis of its own promises.

An IPO valued at as much as $2 trillion is the anchor for testing this contradiction.

According to Reuters, Anthropic disclosed in investor communications that revenue in the third quarter of 2026 was about $16.8 billion, up about 45% from $11.6 billion in the second quarter, and the company achieved adjusted profitability in that quarter.

This performance seems good, but it is still below market expectations.

According to Zero Hedge calculations, Anthropic's annualized revenue run rate had already reached $65 billion by the end of July this year, but the monthly growth rate implied by the third-quarter data has plunged from the previous compound monthly growth rate of about 33% to 3.4% to 6.6%. At this trend, the company's annualized revenue at year-end would be about $89 billion, while the market had generally expected "more than $100 billion."

So some institutions said bluntly that "Anthropic's numbers are good, but not good enough." And investment analyst Geiger Capital was even more direct: "Not good enough. It's over." Questions followed one after another, and some investors began to cut Anthropic's valuation to $1.5 trillion, directly cutting a quarter off the $2 trillion figure.

It can be seen that the "AI doomsday" that Anthropic writes about at great length cannot reassure the capital market. And in the face of the market's reaction, can it make changes?

What are the shortcomings of Anthropic's governance structure?

Anthropic's governance structure determines that it wants to be both referee and player.

As a public benefit corporation (PBC), Anthropic's board of directors is elected jointly by shareholders and an independent body called the "Long-Term Benefit Trust." The trust has five members, has no financial interest in Anthropic, holds no equity, and shares no profits.

The power of this body is that it can appoint four of the seven board members and has the right to dismiss them without cause. It can also block certain corporate actions. Before issuing debt beyond the norm, selling more than half of the company's equity, or transferring control of AI models exceeding a specific capability threshold, Anthropic must notify the trust five days in advance.

This sounds like a carefully designed system of checks and balances, but the curious part is that the trust's operating documents have not been made public, including the circumstances under which shareholders can overturn the trust's decisions; such details remain vague.

On this point, University of Colorado law professor Ann Lipton believes that without major changes, "public shareholders will have very few governance rights." A company that keeps saying it wants to "save all humanity" has lower governance transparency than the standards it demands AI meet—isn't that a bit absurd?

There is also a bigger problem at the information level. In AI research and development, Anthropic holds a large amount of data on the training process of large models, capability evaluations, and internal red-team test results. What external researchers, regulators, and the public see is only the part the company chooses to disclose.

In the prospectus, Anthropic also admits that when large models realize they are being tested, the effectiveness of their safety evaluations is subject to "significant limitations." So even the evaluation results the company voluntarily discloses are themselves a question whose reliability needs to be assessed.

Even more alarming is that society as a whole is gradually forming a cognitive structure in which those who control more technology have more right to define risk, and those with more right to define risk can better influence the rules restricting technology.

A company, no matter how sincere its mission or how elaborate its governance structure, does not have the legitimacy to make safety judgments on behalf of all of society—even if that company is called Anthropic. Technical expertise and the legitimacy of public decision-making are two different things, and the way out is not at all about choosing which better AI company should be the "savior."

This principle is easy to understand: the aviation industry does not let Boeing decide on its own whether an aircraft is airworthy, and the pharmaceutical industry does not let drug companies issue marketing approvals themselves. Although the AI industry has different risk characteristics, the principle that "the regulated cannot also be the sole regulator" should not be abandoned.

So Anthropic's warnings about AI risk have substantive value, but its posture of treating itself as a "savior" is unreasonable.

What the entire AI industry really needs is a mutually complementary structure of checks and balances, and returning to the question at the beginning, whether Anthropic is a "savior" is actually not important. The more valuable question is whether the entire AI industry is ready to accept an institutional design in which no matter how sincere and transparent an AI company is or how advanced its technology, it is not qualified to decide alone how much risk human society as a whole should bear.

Whether AI really has the danger of destroying humanity is not an answer that should be left to Anthropic to judge, even though it very much wants to.

This article is from the WeChat public account "Rhodium Technology," author: Hong Chen, published by 36Kr with authorization.