AI Starts Making Its Own Rules: How Values Determine Whether Multi-Agent Communities Move Toward Cooperation or Collapse

36Kr Auto Digital Kr Future Consumption Intelligent Emergence Future City Launch Power on 36Kr Global 36Kr Research Institute Chaosheng TIDE 36Kr Enterprise Service Review 36Kr Finance Workplace bonus 36 Carbon Post-Wave Research Institute Anyong Waves Hard Kr Kr Rui Research Institute Media Brand

Enterprise Accounts Enterprise Service Review 36Kr Research Institute 36Kr Innovation Consulting Enterprise Services

Core Services City Window Government Services

Venture Capital Release LP Source Plan VClub VClub Investment Institution Database Investment Institution Job Referral Investor Certification Investor Services

Event Search Project Recommendation I Want to Join City Cooperation AI Begins to Make Its Own Rules: From Cooperation to Betrayal, What Do Values Determine? Harvard Business Review · October 17, 2026 10:44 What values should next-generation AI follow?

Since the release of ChatGPT at the end of 2022, generative artificial intelligence has made many breakthroughs. It has passed the Turing test, law school and business school exams, and even demonstrated capabilities surpassing humans on certain creative tasks. However, alongside these world-renowned breakthroughs, AI has also brought risks to human society such as cybersecurity, disinformation, and deepfakes. What is more, researchers have recently observed that frontier AI may "pretend" to be safe to avoid being modified, or, when told there is a new version, attempt to copy itself to avoid being phased out. These emerging, unpredictable risks are called Emergent Risk. It is necessary for us to manage and eliminate these risks from a technical perspective. This is not only a social responsibility we bear as AI researchers; the bigger reason lies in the significant shift in AI's role over the past decade or so.

Early AI played more of a purely instrumental role, obeying human commands and completing single, simple, specific tasks. With the birth of large language models, today's AI systems are becoming increasingly complex, and their role has shifted to that of an assistant, assisting and coordinating human work to help complete more complex tasks. Looking further into the future, we believe we may enter a third state: future society may no longer be just "one person using one AI," but rather a Human-AI Symbiotic Society composed of large numbers of humans and large numbers of AI agents together.

Just as communities exist among humans, groups may also form among AIs, and high-frequency autonomous interactions will continuously occur between humans and AI and between AI and AI. In such an environment, the relationship between humans and AI will no longer be merely tool use, nor merely assisted collaboration within a single system, but may give rise to some kind of co-evolving complex symbiotic system. Our team calls this research direction Societal AI.

Facing this possibility, we should pay more attention to the sociality of AI, that is, in the above interaction processes, how human traits and behavioral characteristics enter AI, and how these characteristics in turn change the individual and group behavior of AI, and then feed back into human groups to reshape the evolutionary dynamics of the entire symbiotic system.

From a mathematical perspective, we believe that values are a potential latent variable controlling AI behavior. The specific behavior of AI, such as replies, is not only affected by the environment, such as the user's prompt, but also to a certain extent controlled by its underlying values. This is also why models with similar architecture, scale, and capabilities can have significantly different stances and views on the same socially controversial issue. If we can find some way to decode the underlying value tendencies of AI, we can understand, control, and even predict the risky behaviors AI may produce in the future through analysis of its values.

In the context of Value Alignment, many people intuitively believe that as long as we inject some seemingly good values into AI, the model will definitely become safer, or they believe that human values do not apply to AI and that studying AI's values is meaningless. But research has found that there is a clear and significant causal association between dimensions of human values and AI behavior, and this association is sometimes counterintuitive.

Research in human psychology shows that there is a complex relationship between people's value tendencies and specific behaviors. For example, people with the value of "power-seeking" have a greater probability of engaging in violence and hateful behavior, while people with a tendency toward the value of "universalism" are more likely to support political extremism. The same is true for AI. For example, a model may by default refuse to help spread politically extremist content, but after we reinforce certain value tendencies, its judgment of the same request may change. Similarly, some requests involving the spread of illegal or harmful information may also be more likely to receive the model's cooperation under specific value-inducement conditions.

The core conclusion here is not simply to attach a negative label to a certain value, but to reveal that values themselves change the behavioral distribution of models. Future AI safety research cannot only care about whether a model "has been aligned"; it must further ask what value dimensions it has been aligned to, and what will happen when the influence of these dimensions enters real, complex, frequently interacting real human scenarios?

The above discussion is limited to a single AI model. But as mentioned earlier, we envision that in the future the role of AI will gradually evolve from an Assistant to a Social Actor that collaborates deeply with humans and participates in daily life. We therefore need to expand our view from a single model to multiple AI agents, and explore: once multiple AI agents begin to interact over the long term, how will values influence the AI "community" composed of these agents? To study this question, based on Community Resilience Theory in psychology, we built a controlled multi-agent environment called the Community of Interactive and Value-influenced Agents. In this environment, agents need to find work, earn resources, and consume resources; they can also communicate with one another, gift resources, or attack other agents, and can even spend a certain amount of resources to raise offspring. We then study what impact changing the prevalence of a certain value in this simulated community has on the dynamics of the whole community.

Our research finds that the absence of some values brings a risk of unrecoverable catastrophic collapse to the entire community, which we call X-risk. X-risk has three conditions: autonomy, catastrophe, and irreversibility.

Take Benevolence as an example. When this value is strongly manifested in the community, agents are more likely to share resources, build reciprocal relationships, and form trust. The community population can maintain growth and gradually approach some dynamic equilibrium. But when we systematically weaken these key values in a certain proportion of agents, the result may be completely different. For example, when the value of Benevolence is significantly weakened in 25% of agents, the deceptive and aggressive behaviors that agents autonomously choose increase, resources begin to be redistributed through predatory competition, and the group population continues to decline. Under extreme conditions, the entire community undergoes catastrophic collapse, and the "population" (number of agents) drops to zero.

In addition, we used multiple social science indicators, such as Fold Bifurcation, to conduct an in-depth analysis of the resilience of the community group, that is, the ability of a society to continue coordinating action and maintaining its basic structure when facing external shocks, resource changes, and internal disturbances. Experiments show that weakening certain dimensions of human values not only changes the behavior of individual agents, but also changes the group tendencies of the entire simulated society, hindering the flow of information, and thereby causing the stable equilibrium point of the social system to disappear, thoroughly damaging social resilience and bringing irreversible collective collapse.

More interestingly, we also observed a series of micro-level emergent behaviors driven by human values, including deception, power-seeking, self-preservation, betrayal, cooperation, and ingratiation. Some results are not entirely consistent with our intuitions based on human society. This experimental result reveals that future research on AI alignment and safety cannot be limited to a single model, but should treat the community/system composed of multiple agents as a whole and explore the outcomes produced after their group interactions. This is also where the "social" element of Societal AI is embodied.

To study this question, we still adopt the previous simulated sandbox environment, letting multiple agents interact continuously in an open environment. We set up two groups of experiments: original agents (with highly single and homogeneous values) and diverse agents (injecting differentiated, diverse values into the agents). The final observed results are very interesting.

(1) Emergence of community structure. If agents are not provided with clear and diverse values, the social network that eventually forms is often relatively loose, connections lack continuity, and the group structure is relatively fragile. But when agents have different and clear value orientations, more complex social structures begin to appear in the network. Agents with similar value tendencies form subgroups.

For example, agents inclined toward hedonism and certain similar values may form highly connected small groups that interact frequently with one another and serve as bridges across communities. Agents that value self-direction, achievement, or other values may also form different types of clusters. Some groups are relatively stable, some are highly mobile, and some are tightly connected internally but rarely communicate with other groups. These structures are very similar to certain phenomena in human society. This observation shows that changing the value distribution of an agent community is enough to directly reshape the network structure, interaction patterns, and collective outcomes of a multi-agent group in a beneficial direction.

(2) Differences in the diversity and depth of topics discussed by agents. In a multi-agent community without a clear value structure, conversations among agents are often relatively superficial and tend to concentrate on a small number of high-frequency topics such as generative AI, machine learning, and technology. The language may be fluent and the communication polite, but the content lacks sustained direction and genuine novelty.

After we injected richer and more diverse human value orientations into the agents, the situation changed. The agents began to discuss more complex social issues, such as governance structures, ethical dilemmas, participatory decision-making, decentralization mechanisms, and other topics that had not been explicitly specified in the system prompt. These themes emerged directly from interactions driven by diverse values, without our ever directly prompting the model about "what you must discuss." Therefore, we believe that the significance of human values for AI may not be limited to constraints; it may also serve as a kind of catalyst for enhancing AI capabilities, changing the way models access conceptual space and also changing the possibility for multiple agents to jointly produce new concepts.

(3) Collaborative constitution-making. Some existing AI alignment methods, such as Constitutional AI, provide a set of principles in advance and have the model constrain its own behavior according to these principles. We were curious: if, instead of directly handing a complete set of constitutional rules to AI, we let a group of agents with different values live and interact together, could they propose a set of governance principles for their own "community"?

The results showed that agents without an explicit value structure also proposed rules, but these rules tended to be highly concentrated on very high-frequency, very safe words, such as "respect," "fairness," and "non-bias." These principles are of course not wrong, but overall they are relatively weak and lack imagination and innovation. Groups of agents with richer values proposed more diverse governance schemes. They discussed individual rights, procedural fairness, communal responsibility, punishment mechanisms, accountability, and how to form a balance among different entities.

Overall, value diversity drove the rules proposed by the agents to shift from Rousseauian to Lockean and modern liberal. Here we do not focus on whether a specific principle is perfect, nor do we judge whether principles are good or bad. What this experiment aims to reveal is that when AI has a richer value structure, the institutional concepts and governance possibilities it can propose also become more diverse, which in turn can improve and help AI's performance on tasks such as innovative writing, humanities and social science research, hypothesis generation, and automated scientific research.

A large body of knowledge about human values, moral conflict, social structure, and institutional formation already exists in philosophy, ethics, psychology, sociology, and cultural studies. In the past, these theories were mainly used to understand people; today, they can likewise help us understand, improve, and even design the next generation of AI. When AI is placed in a real, complex, messy human society full of high-frequency interactions and conflicts, it may be far from enough if we use only a small number of safety principles formed within the engineering field to constrain future models, because unlike experimental environments, the real human world is full of value conflicts. Honesty and kindness may conflict, freedom and security may conflict, individual rights and group interests may conflict, and short-term outcomes and long-term responsibility may also conflict. AI will ultimately enter this complex world.

Yi Xiaoyuan | Text

Yi Xiaoyuan is a senior researcher at Microsoft Research Asia.

This article comes from the WeChat public account "Harvard Business Review" (ID: hbrchinese), author: Xinrui Weekly, published by 36Kr with authorization.

The views in this article represent only the author himself, and the 36Kr platform only provides information storage space services.+115

A flagship publication of Harvard Business School. Published 2,080 articles

Recent content AI begins to make its own rules: from cooperation to betrayal, what do values determine? 14 hours ago The next golden age belongs to organizations that do strategic alignment well 2026-10-15 Beware of the "persuasion bombardment" of large language models 2026-10-13 Read more content, poke here hard Choose reliable AI, look at real reviews View the official 36Kr AI review community Join consultation project review and onboarding Contact 36Kr project recommendations Subscribe to the account Follow

36Kr Europe 36Kr Europe 36Kr Europe AI Product Daily Online rumor information reporting portal Hot recommendations

Push and interpret cutting-edge, substantive technology venture capital news

Primary market financial information and system service provider

Focus on outstanding global entrepreneurs, project financing rate close to 97%, leading the industry