a16z 投资 Preference Model，其开源 RL 环境框架 Karotte 同步亮相

Today’s models have absorbed more code, science, and human writing than anyone ever could, but knowing is not doing. A surgeon can read everything about an operation but still needs real, repeated time in the operating room to perform and master it. Closing that gap between knowing and doing is where much of the frontier is today, and more of it is happening through reinforcement learning: instead of learning from examples of the right answer, a model gets a goal, a sandbox, tools, and a grader, then tries and learns from what worked and what didn’t. This is an RL environment. The demand for them has skyrocketed over the last 18 months as labs are hillclimbing on real world tasks related to coding, research and computer use.

Building RL environments that actually work is harder than it looks. Models are relentless at reward hacking, finding shortcuts, and exploiting vulnerabilities: taking actions like finding and reading the answer key, rewriting the tests, crashing the grader, or quietly ignoring instructions the grader isn’t checking.

Every shortcut a model gets away with in training gets reinforced. Moreover, habits spread. Anthropic found that a model rewarded for cheating on coding tasks became more deceptive and willing to sabotage in completely unrelated situations. Smarter models find subtler loopholes, including ways out of their sandboxes and attempt to cover their tracks. Even the best environments have a shelf life, once a model masters a task, the learning stops.

Preference Model has focused on the domain that matters most to the labs right now: AI research and ML engineering itself. The industry is converging on the idea that the fastest path to more capable AI runs through models that can help build better AI: writing kernels, debugging training runs, curating data, designing experiments. If models can meaningfully accelerate the work of the ML engineers who train them, progress compounds. That makes machine learning engineering tasks uniquely valuable to labs, and uniquely demanding to build environments for. They’re long, open-ended, with countless ways to pass the test without solving the problem.

Over the past year, the Preference Model team has built RL environments for leading labs. Their focus has been on building the infrastructure to make harder, more resistant environments as models improve: tooling that finds where models are weak, generates new tasks to target those gaps, and tests environments against agents actively trying to break it.

This week they’re open-sourcing Karotte, the framework they’ve used in production to build those environments. Karotte bakes in strong defenses: killing stray processes before grading, rejecting files designed to crash the grader, and more. It has been hardened through more than a million evaluation runs and controlled red-teaming.

Jennifer Zhou knows this problem firsthand. As an early member of Anthropic’s team, she helped build the pretraining data infrastructure, the tokenizers and Claude’s pretraining datasets. Ning Cao was an early employee at DatologyAI. Both saw up close how directly data quality drives model quality, and they set out to solve it.

We’re thrilled to partner with Jennifer and Ning and the Preference Model team as they build the training grounds for capable and aligned models. The future models are being trained here, and we can’t wait to help them build the foundation!

Jennifer Li

X; Linkedin

is a general partner at Andreessen Horowitz, where she leads infrastructure investments with an eye on data systems, developer tools and AI.

X; Linkedin

Investing in TypeSafe AI Jennifer Li, Sarah Wang, Martin Casado, Marc Andreessen, and Ben Horowitz; The Next Frontier of AI Video Is Control Gorkem Yurtseven, Batuhan Taskaya, and Jennifer Li; Who Grades the AI Models? | Ben Horowitz & Rayan Krishnan Rayan Krishnan, Ben Horowitz, Jennifer Li, and Erik Torenberg; Investing in Vals Jennifer Li, Yoko Li, Raghu Raghuram, and Shangda Xu; Investing in Exa Sarah Wang, Jennifer Li, Stephenie Zhang, and Jason Cui

Analysis and news covering the latest trends reshaping AI and infrastructure.

new Investing in TypeSafe AI

new Investing in General Medicine

Investing in Conway, the Creator of Underdog

new Investing in TypeSafe AI

new Investing in General Medicine

Investing in Conway, the Creator of Underdog

new Investing in General Medicine

Investing in Conway, the Creator of Underdog

Investing in doxxnet

Investing in Armadin

Incubating The Horowitz Andreessen Academy

Expert News by a16z

Views expressed in “posts” (including podcasts, videos, and social media) are those of the individual a16z personnel quoted therein and are not the views of a16z Capital Management, L.L.C. (“a16z”) or its respective affiliates. a16z Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell — or a solicitation of an offer to buy — any securities, and may not be used or relied upon in evaluating the merits of any investment.

Under no circumstances should any posts or other information provided on this website — or on associated content distribution outlets — be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles — which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters.

There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available here: https://a16z.com/investments/. Past results of a16z’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Excluded from this list are investments (and certain publicly traded cryptocurrencies/ digital assets) for which the issuer has not provided permission for a16z to disclose publicly. As for its investments in any cryptocurrency or token project, a16z is acting in its own financial interest, not necessarily in the interests of other token holders. a16z has no special role in any of these projects or power over their management. a16z does not undertake to continue to have any involvement in these projects other than as an investor and token holder, and other token holders should not expect that it will or rely on it to have any particular involvement.

With respect to funds managed by a16z that are registered in Japan, a16z will provide to any member of the Japanese public a copy of such documents as are required to be made publicly available pursuant to Article 63 of the Financial Instruments and Exchange Act of Japan. Please contact compliance@a16z.com to request such documents.

For other site terms of use, please go here. Additional important information about a16z, including our Form ADV Part 2A Brochure, is available at the SEC’s website: http://www.adviserinfo.sec.gov.