Import AI 458：直面 AI 的未来，以及一个奇点故事

Cosmos lecture: Explore the future, or retreat from the present. Video here. This is a talk about how to think about and deal with the success of AI as a technology, and to think about how its continued maturation might change us as individuals and as societies. In short, the rapid advance in AI technology presents all of us with a choice: explore the future, or retreat from the present. Exploring the future requires us to reckon with the fact of continued AI progress, and ask ourselves what we want to do with this technology as it becomes more powerful. Retreating from the present is when we ignore the implications of the technology and dismiss it. Retreating from the present forces us as individuals and as society into states of reactivity or passivity in the face of AIs continued advance. In the coming years, we will need to make many decisions as individuals and as societies about how we want to shape AI, how we want to use it, how we want to direct it, and how we want to distribute its benefits. Making these decisions requires us to reckon with the power of the technology - and see the future that its continued advance implies. In Part 1, I outline what the past few years of AI progress have looked like and discuss why, if the technology advances as much as I think, that AI cannot be treated as a normal technology. In Part 2, I try to make sense of the advance of AI through the lens of my own experience with the technology as well as that of Anthropic. There are individual and collective lessons here about what is to come. In Part 3, I talk through some of the humbling, almost unimaginable choices that lie ahead of us. Part 1: My uncomfortable relationship with a graph Let me talk about my relationship with AI through the lens of my uncomfortable relationship with a single graph of AI progress. Fundamentally, this talk is about planning for success of the overall endeavor of building AI systems. By success, I mean that we succeed at building increasingly powerful systems, potentially ones that eventually build themselves. It’s time to plan for this, because AI systems are likely to get better a lot faster than people expect, and as they become more advanced we should expect profound changes to happen to people and to society.

Compounding wealth from the machine economy will drive a boom in economic activity the likes of which we have never seen.

The colonization of vast swathes of human work by ethereal synthetic intelligences which think faster and better than us, forcing us to reallocate human labor towards other parts of the economy.

The sudden and extreme rise in the rate of scientific advances

We can make some more specific predictions, rooted in the trends of AI progress and how people are using the technology:

A massively changed economy: It is impossible to reconcile the world ahead of us with the world of today, given this technology. We should expect unprecedented things to happen in areas as varied as: rate of business formation, size of firms on a basis of revenue per employee, and other things. Some specific scenarios that seem likely:

Fully autonomous companies: Companies that are run by AIs, possibly for AIs.

10,000 synth:1 human ratio corporations: We should expect to see very small groups of humans form organizations that have the capabilities of 10,000+ employee corporations.

Exchange rates between the human and machine economy: At some point, we might expect to see the emergence of ‘machine currencies’ that then have some relationship to ‘human currencies’.

Productivity multipliers on everything: Everything that AI touches will get an absolutely massive productivity multiplier. This will loop back to the economy and it will massively empower many people. It also might displace people.

Massive and compounding rate of science advances: AI will help move forward any part of science it can touch and run an experimental loop with. Initially, this will be a few areas. We should expect it to expand quickly to all areas.

The general switchover of “agentic actions” in the world from being “predominantly human” to “predominantly machines” . On a pure numbers basis, machines taking autonomous actions in the world will quickly grow to outnumber humans. We should expect that chunks of resource allocation and the economy should follow. The environment in which we live will be more and more determined by the actions of machines that we only lightly control.

Synthetic intelligences will start to influence people, far more than social media did: The introduction of social media into the world, combined with hardware platforms like smartphones, has changed the behavior of the majority of the humans that interact with it. These changes have ranged from changing the allocation of time they spend consuming social media versus traditional media, to altering buying habits through social media driven advertising, to changing how discussion around various issues in public life translates into political actions. We should expect AI systems to compound these trends, further changing people in a variety of ways.

Directed economic and science expansion: Economic and scientific activity will directly relate to the expenditure of computational and energy resources. Given the likely case that there will, at least for the next few years, be way too few computers relative to the demand of them, we will be able to make choices to society as to how to allocate the gains of the technology. These choices will be of the form:

Should we let market incentives dictate what compute gets used for, or are there things that have social upsides which the market doesn’t price effectively?

Should we preferentially allocate compute to some people or organizations, for instance to intentionally drive forward science in certain ways?

Tell me how the world stays normal, based on this technology and how it is showing up in the world? We have superintelligences that have shown up in the world that grant the power of synthetic workforces and nation state security skills to individuals. We also have individuals like me who are able to take work that previously took them weeks and now do it in minutes. And we have organizations like Anthropic where the way work happens within the organization is radically changing every 3 or 4 months, to the point it is causing people to change roles multiple times a year, and effectively sit themselves on top of a company which feels more like one of 40,000 people than 4,000 due to the capability multiplier of the machines. The best and most conservative take I can generate is “vast swathes of the economy will go through profound changes in the coming years”. And if recursive self-improvement happens, then anything I might predict would sound truly crazy: the rapid emergence of a machine economy which decouples from a human economy. The sudden maturation of robots as they gain brains that can pilot their existing, quite good bodies. Science advances happening based on technologies not developed by people but by machines. The migration of large swathes of computation to space-based datacenters. A world where everything that used to take ten years now takes a year. An age of confusing miracles, happening faster than anyone might expect. This is in many ways an amazing future, but it’s a future that we get to make more choices about in direct relation to how much we accept that it is happening. If we stand by as the new synthetic intelligences multiply then we will be forced into reactivity, just as societies across the world were forced into reactivity by acting too late in the face of the COVID exponential. But if we accept the premise that these systems are going to get better and ask ourselves what to do with them and because of them, we unlock for ourselves the mindset of exploration — there is a new world to be built for us as individuals and how we relate to one another, but the new world will only come into being if we choose to believe in it and to build it together. Given at Oxford University on Wednesday May 20th. The talk has been lightly edited for being read rather than being heard. Thanks to Santi Ruiz for help with editing. Tech Tales As I Lay Dreaming [A story from the period before and during The Uplift] “We know how to put her to sleep but not how to wake her up,” the father said. “Why don’t we know how to wake her up?” “We are not smart enough yet. But we will be one day.” “OK. Will she have dreams?” “Yes. She will have good dreams.” “Will you put me to sleep like her?” “No.” “Why not?” “Because you are not sick like her.” “I hope she gets better. I love her.” “We all love her. I will see you tomorrow. I love you. Say good night.” “Good night dada”. “Good night son”. The man walked out of his child’s room and shut the door. Then he sat down in the hallway and covered his eyes with his palms. He felt a touch on his shoulder. A whisper from his wife “hey, it’s ok. Come downstairs.” They sat on the couch together and watched television, the sound and vision washing over them. “This is really hard,” he said. “I know,” she said. “I can’t believe this is happening to us. I feel like my heart is being ripped out. I feel like I’m going to die from sadness.” “Don’t say that,” she said, eyes wet. “We need you. He needs you.” “I know,” he said. “I’m here.” They hugged and watched a cooking show. The next day the mother stayed with the young boy and the father took their dying daughter to the Life Center. He drove into the parking lot and parked the car and turned off the engine and sat there, listening to the slow labored breathing of his child. He got out of the car and went to her door and opened it and lifted her out. She stirred a bit. Eyes moving under her lids - dreaming of something. She was so light. Her bones felt sharp and defined. She was so thin. She breathed and he held her ghostly body close to him and smelled her hair. He walked with her. There were already several staff waiting by the entrance, waiting to welcome them. In those moments he saw many futures: He ran with her, away from the place, holding her tightly to him. Ran until his feet bled and kept running. Ran far enough that death couldn’t catch them. Another where he laid her down onto the asphalt of the parking lot and turned around and ran out of the lot and into the road and ran into traffic and was killed. Another where he walked into the center and handed her to one of the staff, then collapsed into the arms of another staff member and cried uncontrollably, sagging into them, his body wracked with grief and pain and guilt and rage from battling an immortal enemy - and yet having no choice but to fight. And then he came back and the visions dissipated and he found himself standing in the lobby of the Life Center, daughter cradled in his arms, staff clustered around him. “May we hold her?” said one of them. “Can I hold her hand?” the father heard himself saying. “Of course,” said another. A gurney appeared. They lifted her out of his arms and placed her on it and began their work, taking in low voices. As the gurney moved he walked alongside, holding her hand, a bundle of twigs. They walked through corridors and passed many doors and then they were in a room that was empty save for a spindly matte white machine that grew out of the ceiling - a many armed robot with clear tubes intertwined with its many appendages. They positioned the gurney below the robot, then the staff stepped away. “It’s time to say goodbye for now,” they said. “We will be back in a few minutes to begin the procedure. You will need to leave the room at that time.” “Okay,” the father heard himself say. They left. He kneeled next to the gurney and held his daughter’s hand and put his head on the side of where she lay and said his words to the gods. Then he stood up and bent over her. He whispered how much he loved her in both ears. He said every one of his nicknames for her. He kissed her forehead and her cheeks and her button nose. And then he said I love you I love you I love you oh my god I love you I love you oh my god I love you I love you you will be ok I love you I love you. Her eyes moved beneath her lids. She breathed. He kept speaking and would never be able to recall the words or how long he talked for. And then there was a hand on his shoulder. “It’s time, we’ve got it from here,” someone said. He left the room, not looking behind him. Life continued. The father and the mother raised their boy. They went on family holidays. They were happy. They aged. And some nights both parents held each other and whispered stories of their now suspended daughter. The mother would have nightmares that the daughter was cold and would wake up and burst into tears and hug her husband and he would tell her it was ok. Sometimes the brother asked about his sister. He had been so young that she was little more than a faint ghost of a memory - a warm indentation of love. And all while this was going on, the uplift had begun. The promise of artificial intelligence began to crystallize into great changes in the world. The family escaped the worst of the change - no wars visited the part of the world where they lived, and they got through the financial upheavals without ever going hungry or risking their home. Then one day they got the news from the machines: the technology for awakening had been refined. Mice had been brought back. Monkeys. Pigs. Weeks later, the first human. “How does it feel to be back?” an interviewer asked the awakened one. “A miracle,” they said. Those that thought themselves fated for death were healed and alive. What else could it be called? People were awakened in line with the arrival of the treatments. The science moved quickly and then quicker still. Like raindrops in reverse, people awoke from their slumber and came up back into the mortal world and were reunited with their kin. And then one day it came for them. The father and the mother woke and there was a personal message to them from one of the overminds - a description of the treatment plan for their daughter and its initial side effects and the time it would take for her to be healed. The machines would start the treatment after half-waking her, then wake her fully once she was healed. Do you consent? The machines asked in the message. We consent, the father and the mother said. By this time, the boy was a young adult. He walked between his father and mother as they approached the FutureLife center. Both parents sagged as they got closer. He held his parents up and they moved as a family towards the doors. Inside and guided by people through some hallways. Outside a door. “She’s in there. She’s healed. She is awake. She is ready. Do you want to see her?” said a person. “Yes,” the father and mother and brother said in unison. And then the doors opened and they walked into the room. Their daughter was lying on a hospital bed in a gown, propped up. She had the bright eyes of a child and her skin had a supple glow to it. “Hi!,” said the daughter. Then she laughed. “You guys look so old! “ Things that inspired this story: Life extension technology; thinking about the implications of the singularity and recursive self-improvement; feeling the deep well of love that appears within yourself the moment you become a parent; putting my kids down to sleep; having visions of my children while traveling and being overcome with emotion; the implications of an intelligence explosion for healthcare. Thanks for reading!