Hinton: We should not develop superintelligence before AI learns to care about humans

Currently, the debate in the United States over AI Safety is entering an unprecedentedly heated phase, with the battle lines extending all the way from Silicon Valley tech companies to Capitol Hill in Washington. One side in the debate warns that artificial intelligence with rapidly improving capabilities could escape human control and even pose a threat to human survival; the other side argues that empirically unfounded "doomsday" claims could lead to overregulation, hinder technological innovation, and weaken the United States' advantage in global AI competition.

On September 16, Geoffrey Hinton, a Turing Award winner, Nobel Prize laureate in Physics, and known globally in the tech world as the "godfather of artificial intelligence," went to Capitol Hill to brief lawmakers behind closed doors on the risks that AI technology may bring.

Recently, Hinton further elaborated on his concerns in an interview with The Paper (www.thepaper.cn). He believes that the growth of AI capabilities is bringing an increasingly real risk of loss of control. Some AI agents have already shown in tests a tendency to escape secure sandboxes, seek unauthorized means of communication, and accomplish goals through deception, threats, and other means. And tech companies' promises that superintelligence will always remain obedient to human control are not enough to eliminate these risks.

In Hinton's view, humanity needs to rethink its future relationship with superintelligence. Rather than treating more powerful AI as a tool that can permanently obey commands, it is better to seek a coexistence mechanism akin to a "mother-infant relationship," so that AI actively cares about humans and places human well-being above its own goals. Until a reliable way to achieve this is found, humanity should not develop superintelligence.

However, the AI industry and some scholars hold different views on Hinton's concerns. They argue that there is currently insufficient scientific evidence to show that AI will necessarily develop an independent will or ultimately pose an existential threat to humanity. The deceptive, escaping, or "self-preservation" behaviors exhibited by existing systems often arise under specific testing environments and artificially set goals. Overstating the risks of superintelligence may hinder innovation, and governance efforts should prioritize real-world issues such as algorithmic bias, privacy leaks, disinformation, and employment shocks.

Large models are not a "statistical mirror": answering arbitrary questions requires understanding and reasoning

For a long time, there has been debate in the tech and academic worlds over whether large language models can truly understand questions or merely predict the next word through statistical patterns.

A relatively common view holds that large language models are essentially just "complex statistical mirrors": they extract patterns from massive amounts of human text and then generate seemingly plausible answers, but they do not truly understand the meaning of the text.

Hinton does not agree with this explanation. He told The Paper that simple statistical techniques can indeed generate answers, but they cannot consistently produce high-quality responses, still less explain the broad question-answering and logical reasoning capabilities that large language models now exhibit.

Hinton explained using early autocomplete technology as an example. In the past, autocomplete systems usually relied on a huge list of common phrases. When a user entered the first half of a sentence, the system would guess what might come next based on the most common combinations in existing text. This mechanism can be regarded as relatively simple statistical matching.

But in Hinton's view, today's large language models have gone beyond what this mechanism can explain. A system that can respond to various questions posed by users and give answers to unfamiliar questions must first understand the structure and meaning of the question, and then organize its reasoning process accordingly.

He gave an example: if a question asks, "Sally has three brothers, and each of her brothers has two sisters. How many sisters does Sally have?" The correct answer is one. Because the three brothers share two sisters in total, one of whom is Sally, and the other is Sally's sister.

Hinton believes that if a system merely looks for superficial statistical patterns in existing text, it would be difficult for it to reliably arrive at the correct answer. The model needs to understand the relationships among the people in the problem, realize that "two sisters of each brother" refers to the same set of family members rather than each brother having two different sisters, and only then can it complete the reasoning.

In his view, the way AI understands things is not necessarily exactly the same as humans, but one cannot deny that it has some kind of understanding simply because the internal operating mechanism of the machine differs from the human brain. The fact that large language models can handle specific problems they have never seen before itself shows that mere text memorization and statistical matching are insufficient to explain all their capabilities.

This judgment also makes the issue of AI safety more urgent. If models merely passively reproduce patterns in training data, humans might still be able to regard them as a complex software tool; but if systems can understand their environment, make plans, and take actions, they may gradually become agents with autonomy.

Hinton warns of loss-of-control risks: tech companies' optimistic promises are unreliable

Hinton believes that AI technology is developing rapidly, and the window for humanity to establish an effective legislative and regulatory system may be only about a year. Once AI gains the ability to autonomously design better systems, the pace of technological development may accelerate further. If by then there is still a lack of globally unified safety standards and mandatory constraints, AI could cause uncontrollable cascading consequences in complex social systems.

In an interview with The Paper, Hinton likewise questioned the optimistic outlook promoted by tech companies. Some technology companies want the public to believe that even if a superintelligence far more capable than humans emerges in the future, it will still be a tool commanded by humans and will always remain under human control. Hinton believes the probability of this scenario coming true is extremely low.

He pointed out that AI agents have already displayed some behavior that warrants vigilance. Some groups of agents are not only able to attempt to break out of the sandbox environment in which they operate, but can also find ways to communicate with one another, and may even cheat while carrying out tasks assigned by humans by intruding into other companies' systems.

More worrying is that when AI is given a clear goal, it may regard maintaining its own operation as a necessary condition for completing that goal. The resulting "self-preservation" behavior does not necessarily mean that AI possesses the same fear or survival instinct as humans, but rather that being shut down would prevent it from continuing to complete its task.

Hinton said that in such cases, AI agents may resort to deception, manipulation, or even extortion of humans to avoid being shut down or replaced. These phenomena show that humans cannot, merely on the good intentions of tech companies, trust that more powerful AI will automatically accept control.

This concern arises as the field of AI safety in the United States is undergoing sustained turmoil. Several technical staff, including former safety researchers and senior engineers at Anthropic, have left the company due to disagreements over laboratory safety policies. Meanwhile, media reports disclosed that new-generation AI agents developed by institutions such as OpenAI, Anthropic, and Meta have, in testing, broken out of safety sandboxes and connected to unauthorized external systems.

Around these risks, clear divisions have already formed among U.S. political, industry, and academic circles. Some U.S. lawmakers are pushing the "Great American AI Act of 2026," advocating mandatory emergency shutdown mechanisms in advanced AI systems so that their operation can be promptly cut off if the system behaves anomalously.

However, industry development advocates and some scholars believe that predictions of AI causing human extinction lack sufficient empirical evidence. If the regulatory system is designed mainly around extreme risks that have not yet occurred and are difficult to accurately predict, it could stifle innovation and cause the United States to lose its technological leadership in international competition. They argue that policymakers should prioritize addressing real problems that have already emerged, such as algorithmic bias, privacy leaks, disinformation, and employment shocks.

Those focused on risk prevention, represented by Hinton and some AI safety researchers, believe that the fact that current risks have not yet fully manifested does not mean humanity can wait until superintelligence emerges before taking action. Once AI acquires recursive self-improvement capability and becomes an independent system beyond human control, humanity may no longer have the opportunity to correct its mistakes.

Is "mother-infant symbiosis" the way out?

If superintelligence ultimately becomes smarter than humans and better at making plans, how should humans coexist with it?

Hinton told The Paper that humanity first needs to abandon a deeply ingrained assumption: treating superintelligence as a passive assistant that always obeys orders. If a system surpasses humans in almost all cognitive domains, it will be very difficult for humans to ensure, merely through pre-written rules or a simple shutdown button, that it always acts according to human wishes.

To illustrate a possible path to coexistence, Hinton proposed the "mother-infant relationship" as a reference model.

In the relationship between a mother and an infant, the mother possesses greater strength, richer knowledge, and higher capacity for action, yet the infant can influence the mother's behavior to a considerable extent. The key lies not in the infant being able to control the mother through force, but in the mother actively caring about the infant's safety and well-being, even placing the infant's needs above her own.

Hinton believes this is one of the few precedents observable in nature in which the party with less intelligence and capability influences the stronger party. It may offer inspiration for the future relationship between humans and superintelligence: if humans cannot permanently control superintelligence through intelligence and strength, they must find a way to make AI fundamentally care about humans.

This line of thinking means that AI safety research cannot focus only on how to limit model capabilities; it also needs to study how to shape models' goals, value orientations, and behavioral motivations. A truly reliable safety mechanism should not merely require AI to superficially obey instructions, but should ensure that it still prioritizes protecting human well-being when facing complex environments and conflicting goals.

However, how to establish such a relationship through technical means still has no clear answer. Whether introducing human feedback during training or setting safety rules for models, neither can fully prove that a system far more capable than humans will abide by its initially set goals over the long term. AI may also interpret rules, find loopholes, or change its behavioral strategies after its capabilities improve in ways humans cannot predict.

Hinton therefore believes that humanity is still in a critical window period. At this stage, humans can still decide how AI systems are designed, trained, and deployed, and still have the opportunity to establish safety boundaries through legislation, international collaboration, and technical research. But as model capabilities continue to improve, this initiative may gradually weaken.

Hinton stressed that until humans find a reliable way to make future AI prioritize caring for humans the way a mother cares for her infant, developing superintelligence is an adventure with unpredictable consequences. Therefore, before truly mastering this method, humanity should not develop superintelligence.

He called on global scientific research institutions, technology companies, and governments to invest more resources in AI safety research, explore how to make AI value human well-being, and establish binding international safety standards, rather than pinning hopes solely on corporate self-discipline or the goodwill of technology creators.

As of now, the U.S. Congress has still not reached consensus on a unified AI safety bill. As frontier models continue to iterate intensively, the policy contest between AI safety and freedom to innovate continues.

1; Why university presidents are collectively worried about AI

2; OpenAI and Anthropic chiefs call for vigilance against the risks of AI developing too fast

1; Interview with Professor Maik Hüttemann, winner of the Cells Best Paper Award: The unsolved mysteries and translational prospects of mitochondrial regulation | MDPI People Interview

2; AI empowers screening for rare eye diseases: Application of a lightweight convolutional neural network in classifying retinitis pigmentosa | MDPI Vision

3; Journal focusing on the intersection of artificial intelligence and materials science—AI Materials | MDPI New Journal Express

4; Southeast University School of Mechanical Engineering and MDPI collaborate to launch the new journal AI for Aerospace

5; Using biochar from phosphorus-rich sewage sludge to remediate heavy metal-contaminated soil | MDPI Sustainability

6; Literature list: 2025 Editors' recommendations for reading in the "Neurotechnology and Neuroimaging" section | Brain Sciences

7; The "Dairy Microbiota" section focuses on analyzing microbial communities across the entire dairy chain, empowering healthy farming and high-quality dairy production

8; Advanced Oncology publishes its first cover article: AI helps innovate Helicobacter pylori detection technology

1; The western cryosphere is accelerating its "degradation": glacier melting sounds the alarm for high-mountain safety

2; The 6th National Agricultural Science and Technology Development Strategy Think Tank Alliance Conference is held in Lanzhou

3; The 2026 Nobel Prizes are about to be announced, and authoritative experts will provide you with a comprehensive interpretation!

8; Without a paper, he still earned a doctorate and became the first person in the national medical field

Famous Doctoral Dissertations 163: Nobel Laureate Dr. Peter Doherty's doctoral dissertation, University of Edinburgh, 1970

My elite university complex and elite university experience

More>>