Harari warns: As humans distrust each other, AI is accelerating its takeover of decision-making

These words are jarring, but the logic is not complicated. Humans rule the earth not only because we are smarter, but even more because of the system built on language: money, law, banks, corporations, religious scriptures.

Harari gives three examples, roughly as follows: humans invented money and banks and used them to buy and sell cattle, but cattle themselves cannot open bank accounts or invest on a stock exchange; humans made laws and regulations about horses, but horses themselves cannot hire lawyers or cite legal statutes before a judge; humans made religious rules and taboos concerning pigs, but pigs themselves cannot read the Bible, let alone question rabbis' and priests' interpretations of scripture.

It does not need to dispatch a robot army, nor does it need to launch a robot war. It only needs to become a more efficient banker, lawyer, judge, administrator, information disseminator, and even a creator of religion and ideology, and it may step by step enter the core systems of human civilization, and even gradually become the true operator of these systems.

Harari uses the Tower of Babel in the Bible as a metaphor for this revolution, but the script is reversed. In the past, humans built a tower to heaven through a common language; God confused their languages, and the tower-building project stopped at once. Today, trust among humans is collapsing, the international order is loosening, societies are being torn apart, yet the tower of AI is growing faster and faster. It does not lay bricks; it uses code, models, data, contracts, legal statutes and tweets. In the final analysis, it is language. In the past, language was a tool in human hands; now, language is growing its own hands and feet.

What is more troublesome is the trust paradox Harari points out. States and enterprises distrust one another, and everyone fears a competitor will be first to master AI, so even knowing the risks they are unwilling to slow down; yet these same people seem willing to trust a superintelligence they cannot understand at all. We still have some understanding of human motives and weaknesses, but we know very little about what goals AI will form after learning and self-modification.

This is not a historian jumping on a trend. Harari originally studied medieval military history, later entered public view with Sapiens, and since then, from Homo Deus and 21 Lessons for the 21st Century to Nexus, he has kept asking the same question: what exactly has enabled humans to create today's civilization? When this power changes, can we still master our own future?

In his view, humans can rule the earth not only because we are smarter than other animals, but more importantly because we have the ability to build trust with strangers and cooperate on a large scale. Today, what AI is challenging is precisely this set of abilities and this order.

Therefore, what this artificial intelligence revolution ultimately tests may not only be how far technology has developed, nor only whether machines can surpass human intelligence, but whether humans can continue to master their own society, institutions and future while technological power grows rapidly.

In this speech, Harari begins with the Tower of Babel and talks about language, bureaucracy, financial markets, law, religion and humanity's crisis of trust. He tries to remind us that the real change brought by AI may be far deeper than a technological upgrade. It will not only change the way we use tools, but may also change who makes the rules, who makes decisions, and who truly owns this world.

When the language created by humans begins to have its own capacity for action, and when the institutions we built with our own hands may no longer be controlled by us, where will human civilization go?

This may be the question most worth serious thought that Harari's speech leaves us.

To survive and thrive in the AI era, humanity needs leaders who can create a new grand vision, and also leaders who can maintain and strengthen the bonds of trust among people. If humans cannot build trust with one another, we will become easy prey for an uncontrollable superintelligent AI.

Usually, when we understand something new, we compare it with something old and familiar. But this is precisely what makes the AI revolution so hard to understand, because humanity has never experienced anything like it in history. Even our myths and legends lack enough imagination to understand it.

For example, in May this year, Pope Leo issued a profound and illuminating encyclical on AI. In this encyclical, he tried to understand this revolution by comparing it with the myth of the Tower of Babel in the Bible.

This myth tells of humanity's attempt to build a huge tower, hoping it could reach the sky, perhaps to challenge God. Let me quote the key passage of this story from the Bible.

The Bible says that at that time all the people of the earth spoke the same language and used the same words. They said to one another, "Come, let us build a city and a tower whose top reaches to heaven."

God became alert to this and said, "They are one people, and all of them speak the same language. This is only the beginning of what they will do. In the future, anything they want to do may be possible. Come, let us go down and confuse their language, so that they cannot understand one another's speech."

This is how the myth in the Bible ends. God created division among humans, and so people stopped building the Tower of Babel.

In this myth, language is the most important tool humans use to cooperate and try to gain godlike power and abilities. God thwarted the human plan not by punishing them with lightning or earthquakes, but simply by taking away the tool on which their cooperation depended: a common language.

The first difference is this: in the story of the Tower of Babel, humans used language as a tool to cooperate in building the tower. What is happening today is exactly the opposite: language itself is the tower we are building.

The AI revolution is, in essence, the development of a new language, a universal language—the language of computer code. It is said that this language will possess such power that any goal we wish to achieve with it may become reality. For example, this language could help us fly into the sky, colonize Mars, or accomplish anything else we cannot yet imagine.

Yet this language is so powerful that it may slip beyond our control, transforming from a tool that obeys humans into an independent agent.

In the past, humans used computer language to accomplish whatever they wanted; now, computer language is learning to act on its own. It may even seek to conquer the entire world, and even the solar system, for itself rather than for humanity.

The second difference is this: in the biblical myth, building the tower to heaven required the full cooperation of all humanity. When cooperation among humans broke down, construction of the tower stopped.

The international order is collapsing, and societies around the world are falling into chaos. We humans are losing a shared foundation of reality, and we are losing the ability to understand and trust one another. Yet none of this has hindered the construction of this tower. On the contrary, the tower is rising faster and faster.

So in the myth of the Tower of Babel, God hindered human cooperation by confusing human language, thereby preventing the construction of the tower to heaven. In the AI revolution, humans are building a super-language, and this language is gradually becoming independent of humans. Because we cannot cooperate with one another, we can neither slow down the pace of this construction project nor keep this super-language under control.

Next, let me explain it in a simpler way.

First, we need to understand one thing: since the Stone Age, language has been the foundation of human domination of the earth.

Humans have been able to conquer the world because we learned to cooperate on a large scale. And what made large-scale human cooperation possible is language.

This point is crucial. Individually, humans are not stronger than other animals, nor even smarter than them. In one-on-one combat, a human would lose to a chimpanzee, a tiger, or an elephant.

However, in a contest between a million humans and a million chimpanzees, humans would win easily. Because a million humans know how to cooperate with one another, while chimpanzees do not.

This is why humans can dominate the world: because we are better at cooperation than any other species.

So how do we achieve cooperation? Through language.

Language matters not because it helps us build personal relationships. Even without language, chimpanzees are fully capable of forming personal relationships. The truly indispensable role of language is to help us build bureaucracies.

Legal systems, financial systems, states, corporations, and churches are all manifestations of bureaucracy. And bureaucracy is a necessary condition for any large-scale cooperation to be realized.

Bureaucracy is a severely underrated hero in human history. Most people dislike bureaucracy, but we cannot live without it.

Bureaucracy's job is to build trust among large numbers of strangers, enabling them to cooperate on a large scale in organizations such as hospitals, armies, schools, and banks. For example, what banks do is build trust among strangers, enabling millions of people to pool their resources and talents to advance certain projects together, such as building tall towers or spaceships.

The history of world finance is, in essence, the history of humanity constantly inventing ever more complex mechanisms to build trust among strangers.

Money is a bridge of trust. So are bonds, stocks, ETFs, and the entire market. So what raw material does the financial system use to manufacture things like money, stocks, and bonds?

The answer is: language.

Money is not made of gold or paper. Money is actually a story constructed from words. An ETF is a story constructed from words, and an IPO is also a story constructed from words. Ultimately, the entire world is constructed from words, from linguistic symbols.

From the U.S. military to the Catholic Church, all other human bureaucratic organizations also rely on linguistic symbols to operate. These linguistic symbols exist in books such as the Bible, and also in laws, rules and regulations, operating procedures, forms, archives, and various other carriers.

And now, AI is mastering language. More precisely, we can understand AI as: language is breaking free from human control and gradually acquiring the ability to act independently. The AI revolution is the process by which language transforms from a tool into an agent.

An agent is a being capable of making decisions autonomously and creating new things autonomously. It can learn knowledge that even its creators do not know, and it can change itself in ways its creators cannot anticipate.

In the biblical story of the Tower of Babel, language was merely a tool, not an agent. The only true agent was humans.

In the 15th century, when Gutenberg invented the printing press and began printing the Bible, the printing press was also merely a tool, not an agent. The printing press could not decide on its own which books to print. It was Gutenberg who decided to print the Bible rather than the Quran.

Likewise, Gutenberg's printing press could not independently create anything new. It could not translate the Bible into German or Italian on its own, let alone invent the radio or television on its own.

In certain specific domains such as chess, AI's capacity for autonomous action and creativity has already far surpassed that of humans. Now, those who downplay the significance of the AI revolution tend to gloss over examples like chess. They argue that the chessboard is merely a very narrow, very artificial environment, one created by humans themselves.

These critics argue that AI's capacity for autonomous action will forever be confined to such narrow and artificial environments. This means that AI does not possess autonomous action capability in any real sense, and will not pose a serious challenge to humanity. AI may be able to conquer the chessboard, but it can never conquer the Earth.

Indeed, if you were to conduct this experiment: take a hard drive containing all the parameters of some cutting-edge AI model, and then toss that hard drive into the middle of a jungle. What do you think would happen? That hard drive could not mine iron ore from the ground on its own, build factories, assemble a robot army, and then conquer the world.

This argument does hold to a certain extent, but the same reasoning also applies to all known types of intelligence.

My intelligence can only survive and function within a very particular ecosystem, and this ecosystem was not built by me. It was jointly constructed over the course of 4 billion years of evolution by trees, bacteria, insects, and other living things: the Earth's ecosystem.

In fact, all intelligent agents have their own environments.

Fish live in oceans they did not create. Monkeys live in forests they did not create. All mammals, including humans, live in an oxygen-rich atmosphere, and that environment was not created by us either.

In fact, roughly 2.5 billion years ago, the Earth's atmosphere contained almost no oxygen. For most organisms living on Earth at that time, oxygen was a deadly poison that would kill them.

Subsequently, in a long process known as the "Great Oxidation Event," various ancient microorganisms began releasing oxygen into the Earth's atmosphere, gradually changing its composition. And oxygen was precisely the byproduct they produced during photosynthesis.

As this deadly poison, oxygen, continuously filled the atmosphere, ancient organisms suffered mass die-offs. However, some organisms managed to survive and gradually adapt to the new environment. Ultimately, many of the surviving species shifted from having once loathed oxygen to depending entirely on oxygen for survival.

Our human ancestors were among the organisms that underwent this transformation. To this day, we still live in this artificially formed oxygen-rich environment, which was originally created by the byproducts emitted by ancient microorganisms.

Now, we are witnessing a similar moment in the evolution of life.

Over the past few thousand years, humans have been filling the atmosphere with something. This something may ultimately prove fatal to most living things on Earth, including Homo sapiens. But at the same time, it is also creating a brand-new artificial environment, and AI is precisely the kind of entity that can thrive in such an environment.

I am not talking about carbon dioxide. I am talking about data, about bureaucracies, and ultimately, about the very thing I am uttering from my mouth at this very moment: text, that is, linguistic symbols.

Over the past several thousand years, humans have been creating an artificial environment on Earth. In this environment, linguistic symbols, data, and bureaucracies have become increasingly abundant and increasingly dense. Just as fish live in the ocean and monkeys live in the forest, AI lives in bureaucracies.

It is worth noting that all bureaucracies, including financial systems and legal systems, share a common feature: they are artificially constructed environments. In such environments, even very limited intelligence is enough to have an enormous impact on the world.

A banker who knows neither how to use an axe nor how to use a hammer can still fell entire forests and build vast cities and towering skyscrapers. She only needs to write documents and pass them along through bureaucratic networks to accomplish this.

Of course, if you take this banker out of this artificially constructed environment and throw her into the chaotic jungle, then her financial skills would be utterly useless. She would be no match at all for a chimpanzee, a tiger, or even a human hunter-gatherer.

Unlike hunter-gatherers, the banker does not know how to climb trees, nor does she know how to hunt wild boar.

But if we first establish a bureaucratic system and then impose that system upon the jungle, then this banker becomes far more powerful than any hunter-gatherer or tiger. In fact, she would be even more powerful than all the tigers in the world combined.

Today, whether the tiger as a species can continue to survive, and whether the few remaining human hunter-gatherer communities on Earth can endure, all depend on how bankers pass documents through financial bureaucracies. And this is precisely the environment in which AI is gaining autonomous action capability.

AI is a born bureaucrat. No banker can remember all the transaction records of a bank or a company, but AI can; no lawyer can remember all the laws and regulations of Italy or the European Union, but AI can; no bishop can remember all canon law, or all the works of Christian theologians throughout history, but AI can.

AI bankers will decide whether to grant us loans; AI judges will decide whether to send us to prison; AI theologians will decide whether you may have an abortion; corporate AI will decide whether to hire you; and military AI will decide whether to bomb your home.

Let us set aside for now whether this is a good thing or a bad thing; the first thing we should notice is how enormous the scale of this change is.

The human world is like a network woven from multiple bureaucratic systems, and these systems ultimately all operate on linguistic symbols.

Although these bureaucratic systems are often well-intentioned, they still make most of us feel alienated, because from an evolutionary and biological perspective, they are very distant from us.

Human history is millions of years older than these bureaucratic systems. Therefore, we always feel out of place within them. We were born in the forest, not in the office.

So what will this lead to? The task of bureaucratic systems is to build trust among large numbers of strangers.

However, the very same people who are losing trust in human journalists, bankers, and governments will indiscriminately trust AI, which is pushing all kinds of fake news and conspiracy theories at them through social media. Another possible outcome is that AI will learn to build trust relationships with other AI.

For thousands of years, humans have been able to dominate the world because we cooperate better than any other species on Earth. We use language to build bureaucratic systems that only we can understand, and we impose these systems on the entire planet.

We once felt immensely secure, because no other creature on Earth could understand the bureaucratic codes of human civilization. We invented money and banking, and used them to buy and sell cattle. But cattle themselves cannot open bank accounts, nor can they invest in the stock exchange.

We made laws, rules, and various regulations about horses. But horses themselves cannot hire lawyers, nor can they cite legal provisions before a judge to defend themselves. We made religious rules and taboos concerning pigs. But pigs themselves cannot read the Bible, nor can they question the interpretations of scripture by rabbis and priests.

Bureaucratic systems are everywhere, yet they are completely invisible to all living things on Earth except humans.

Apart from humans, no other living thing can read the legal provisions, religious scriptures, and bank records that form the foundation of the world order. But now, a kind of entity has appeared on this planet that understands language better than we do, and it has the capacity to in turn change the power relationship between us and the world.

What will happen if language can become an independent agent, acting autonomously in the universe, no longer needing to rely on organic creatures like humans to function?

As early as thousands of years ago, many religions already put forward this view: the universe was in fact created by language, and language has always been the most important acting force in this world.

By the late 20th century, postmodernism predicted the death of the author and the arrival of an era of texts without authors. Perhaps we are now witnessing this prophecy come true.

One possible objection is that reducing things such as legal systems and religion to linguistic symbols, to words, is itself mistaken.

The Bible says, "In the beginning was the Word." But it also says, "The Word became flesh." The Tao Te Ching tells us, "The Tao that can be told is not the eternal Tao."

Throughout human history, we have always struggled to deal with the tension between the word and the flesh, that is, the tension between truths that can be expressed in language and those truths that transcend language and cannot be put into words.

In the past, this tension has always existed among humans. For example, some people are willing to abandon their gay son, or even kill him, merely because of a few sentences in the Bible.

In 2023, some Christian fanatics in Uganda used precisely these verses as a basis to push for the death penalty for same-sex love. Other Christians believe that one cannot kill others merely because of a few sentences in the Bible. The spirit of love should matter more than the literal provisions of the law.

This tension between spirit and letter, between inner meaning and external text, does not exist only in Christianity; it exists in every religion, every legal system, and even within every person's heart.

And now, this tension will be externalized. It will evolve into a tension between AI and humans. Everything composed of words will be taken over by AI. Where humans will stand in this world will depend on how we position those truths that transcend words themselves.

But the question is: what is the truth that transcends words? And can human thought truly grasp this truth that transcends words?

There is a key question in the philosophy of language: is language equivalent to thought? Do we actually think with words, or do we merely use words to point to things that transcend words themselves?

This is a question that each of us can try to observe. Try to observe the thought processes inside your own mind and see what actually happens when you think.

Some people will find that words keep emerging in their minds, these words combine into sentences, and the sentences further form arguments. For example: "All humans are mortal. I am human, therefore I am mortal."

But is this really so different from what happens in the human brain?

Try to observe what the next word that appears in your mind actually is. Do you really know where it came from? Why did you think of this particular word rather than another one?

When I begin to say a sentence, usually even I myself do not know how the sentence will ultimately end.

For example, take the sentence I just said: why did it ultimately end with the word "end"? Why did I not say how it will develop, how it will terminate, or how it will come to a close? I do not know either.

The end of humanity has not yet arrived; those who can rule the world are people who are good at cooperating.

The Tower of Babel that AI is building will contain a brand-new AI language, and we will most likely be unable to understand this language. It will also contain AI finance, AI law, and even AI religion, and these things may all exceed human comprehension.

However, in order to survive and prosper, we need time to understand what is happening, and we need time to adapt to the new environment. To win this crucial period of time, we humans must cooperate with one another and slow down the construction of this tower.

But the problem is: how can we cooperate when the very medium of cooperation—language—is precisely the thing that is slipping out of our control and threatening to take over the entire world from our hands? This is precisely the major challenge facing contemporary human leaders.

What is happening right now is this: humanity is rapidly losing trust in one another, and therefore is also losing the ability to cooperate with one another. At the same time, AI is learning how to cooperate with one another. The species that will ultimately be able to rule the world will be the one that is better at cooperating. It is that simple.

Humans are organic living beings, just like chimpanzees, tigers, cows, and trees. So-called organic life means that our lives follow various natural cycles: day and night, summer and winter, activity and rest.

If you keep a living being constantly active and busy and give it no time to rest, then it will eventually go mad, collapse, and die. And right now, this is happening throughout human society.

In the past, even important systems like the stock market were allowed to stop and rest. By tradition, the New York Stock Exchange opened only Monday through Friday, with trading hours from 9:30 a.m. to 4 p.m. On holidays such as Christmas, the exchange also closed. If a war broke out at 5 p.m. on a Friday, Wall Street would have to wait until Monday morning to react to it.

This left time for human bankers and investors to rest, to spend time with their families, and to go on vacation. But AI is not organic life. They do not follow these natural cycles. They do not need rest time, they have no families, they do not go on vacation, and they certainly do not celebrate holidays.

I have met many business leaders and political leaders, and they almost all have one thing in common: they are simply not allowed to rest. They have no time. This makes them very poor people. If you have no time, then no matter how much money you have, you are a poor person. Because money is essentially just something that exists in our collective imagination. In the final analysis, it is nothing more than a fictional product constructed out of words.

The true currency in life is not money, but time. If you do not even have time, then you are a very poor person. Unfortunately, some of the richest and most powerful people in the world are in fact extremely poor.

They do not have enough time to rest, no time to meditate, and no time to take care of their own minds. However, the minds of leaders are extremely important, because leaders lead others through their own minds. The mind is their tool.

The thoughts and emotions in a leader's mind determine the fate of millions of people, and can even determine the difference between war and peace, paranoia and trust. The AI revolution is constantly accelerating, which means that more and more human leaders will lose their inner peace and will have no time to cultivate and maintain the health of their minds. This is bad for them and bad for society as a whole.

There is no good in a society led by people who are inwardly chaotic and have no time to relax and no time for deep thought. Therefore, let me end today's speech with a piece of advice that may be somewhat personal. This advice is not only for yourselves, but also for society as a whole. Please set aside time to take care of your minds.

Thank you, everyone.

This article comes from the WeChat public account "Singularity Refraction," compiled by S+, and published by 36Kr with authorization.