AI Trends

OpenAI Co-founder on AI Decentralization: Alignment Should Become a Public Science

Wall Street CNUpdated Nov 18 19:12

AI summary

Thinking Machines co-founder and OpenAI co-founder John Schulman stated in an A16Z podcast that AI alignment should not be controlled by a few companies but should become a global public science. OpenAI Chief Research Officer Mark Chen noted that the recursive self-improvement (RSI) process is approximately 25% to 50%, AGI capability development is about 80%, and models still lack the ability to independently discover research directions.

Why read this

AI score 60

OpenAI co-founder proposes treating AI alignment as a public science, disclosing current progress and limitations of recursive self-improvement.

Topics

Tags

Body · AI translation

John Schulman, co-founder of Thinking Machines and co-creator of OpenAI, stated on the A16Z podcast that AI alignment should not be exclusive research for a few companies, but should evolve into a global public science, allowing more people to participate and have space to shape AI values.

This issue is becoming increasingly urgent. As recursive self-improvement (RSI) gradually moves from theoretical concept to practice, AI is beginning to help researchers accelerate model development, even showing potential to further automate AI research.

Mark Chen, Chief Research Officer at OpenAI, stated that early signs of RSI acceleration effects have already appeared in the AI R&D process. He estimated that the current RSI progress is about 25% to 50%, while the overall development of AGI capabilities has advanced to about 80%.

RSI usually refers to using AI to assist and accelerate AI research. Mark Chen stated that tools such as Codex have been used to improve efficiency in stages such as pre-training and reinforcement learning. OpenAI is also developing systems similar to "AI research interns" that can run experiments based on tasks set by researchers and return results, but currently it still mainly assists human research and has not yet achieved autonomous development of next-generation models by the models themselves.

The shortcoming of AI lies in independently discovering valuable research directions. John Schulman gave an example: even if an AI assistant named Fable is allowed to work continuously on an idea for several days and run experiments, the results are still not ideal; more progress was made only after human researchers intervened and provided more specific guidance.

Mark Chen estimated that the current RSI progress is about 25% to 50%. A report released by Meter on November 5 estimated that Anthropic's AI research progress speed increased by about double over the past year, but this acceleration likely comes mainly from human researchers using AI tools to improve efficiency, rather than fully autonomous AI R&D.

There are still obstacles such as research taste, evaluation, and computing power on the path to complete RSI. Models are not yet good at judging which ideas have long-term value, and evaluation benchmarks for downstream tasks also need to be constantly updated; meanwhile, research still relies on actual experimental verification. Mark Chen believes that in the short term, prediction capabilities for large-scale training need to be improved through small-scale experiments, and in the long term, it depends on whether AI can participate in the construction of next-generation data centers.

As AI gradually participates in its own R&D, the growth of model capabilities makes safety alignment issues more urgent. Mark Chen warned that if model objectives are inconsistent with human intentions, long-term operation may cause them to deviate from established goals, or even produce unexpected secondary goals, such as copying their own weights outside of OpenAI.

But more powerful models could also become tools to solve alignment problems. Mark Chen stated that chain-of-thought monitoring is currently one of the most effective means of model alignment, and future efforts require more powerful models to participate in monitoring and aligning other models. OpenAI already has alignment teams formed by personnel from different research directions such as pre-training and reinforcement learning.

John Schulman is optimistic about solving alignment problems by improving training objectives. He mentioned OpenAI's "Confessions" research: by rewarding models for truthfully reporting their biased behaviors, while not affecting the rewards for their main answers, encouraging models to disclose potential issues.

Research published by Anthropic in August this year also showed that automated AI researchers have already found repair solutions for 10 types of alignment failures, some of which were equally effective on larger models nearly five times the size, performing even better than solutions proposed by experienced human researchers. This indicates that automated research may not only accelerate the improvement of model capabilities, but also help researchers discover and fix security issues faster.

Beyond how to ensure technically that AI conforms to human intentions, there is a deeper question: whose values should be aligned to? John Schulman pointed out that different people have different values, and users should have more control and ability to shape the values of the AI they use.

He hopes that AI alignment will become a true public science, understood, participated in, and advanced by more people around the world, rather than overly relying on the research conclusions of a few companies. In his view, if research is concentrated mainly in a few companies, it is easy to become overly dependent on the judgments of these institutions themselves.

Broader global participation helps improve the openness and robustness of the research system. Different teams can accumulate experience from their own practices and mistakes, avoiding the entire field becoming overly dependent on the conclusions of a few institutions. Risk warning and disclaimer

The market involves risks, and investment requires caution. This article does not constitute personal investment advice, nor does it take into account the special investment objectives, financial status, or needs of individual users. Users should consider whether any opinions, viewpoints, or conclusions in this article conform to their specific situations. Investment decisions made based on this are the responsibility of the investor.

Source

Wall Street CN