OpenAI releases Pando: a Git backend rebuilt for agent-scale parallel pushes, with median push roughly 5x faster
OpenAI has released Pando, its self-developed Git backend, compatible with the Git protocol and clients. In production, median push latency has dropped to about 1/5 of the original, and it can serve nearly 1 PB of Git data per day. The multi-branch concurrent writes and massive concurrent reads brought by Codex once made the old system slow to push and pull, with reliability at one point below 90%. Pando instead makes individual refs serializable, maintains only causal consistency across the whole repository, and places packs in object storage for reuse across clients. The project went from inception to production in about 10 weeks using Codex, was rewritten in Rust, and rolled out via a dual-write proxy followed by phased traffic migration.
Why read this
AI score 69
The original article breaks down Pando's trade-offs of trading causal consistency for write parallelism and reusable packs for read scaling, offering insight into large-scale repository backend design.
Topics
View original postWe built Pando, a new Git backend for our monorepo, to handle more pushes and fetches without changing how our engineers and agents use Git. Pando cut push latency by ~80% and now serves nearly a petabyte of Git data a day. We can grow Git capacity alongside agent usage. https://t.co/4c0YXOfKqK
View original
We built Pando, a new Git backend for our monorepo, to handle more pushes and fetches without changing how our engineers and agents use Git. Pando cut push latency by ~80% and now serves nearly a petabyte of Git data a day. We can grow Git capacity alongside agent usage. https://t.co/4c0YXOfKqK
Body · AI translation
OpenAI Releases Pando: A Rebuilt Git Backend for Agent-Scale Parallel Push, Median Push About 5x Faster
After Codex pushed OpenAI's monorepo to the limit of "concurrent writes across many branches + massive concurrent reads," they built Pando in-house, a new backend compatible with the Git protocol and clients: branch-level causal consistency, and packfile reuse across clients; in production, median push latency dropped to about 1/5 of the original, and it can serve nearly 1 PB of Git data per day. https://t.co/lj8LXUD8nJ
Codex amplified parallel changes, branches, pushes, and CI fetches all at once; since late last year the old system suffered from slow pushes and pulls and merge failures, with reliability at one point below 90%. They did not replace the Git workflow, but replaced the backend: unrelated branches can be updated in parallel; on the read side, they try to use the same stored pack to saturate bandwidth.
They compared two common approaches: three-phase commit relies on a majority quorum of replicas to lock refs, making whole-repo updates rather serial, with disk and coordination pressure on the push path; a shared WAL/manifest can upload packs in parallel, but unrelated branches still contend for the same compare-and-swap, and if replicas fall behind in building indexes, "pull right after push" suffers. Pando relaxes two assumptions (that the whole repo is consistent at a given moment, and that a dedicated pack is computed on the fly per client) and instead: a single ref is serializable, and the whole repo only maintains causal consistency; packs go into object storage, and metadata records tips, dependencies, and pack pointers. Pushing different branches does not block each other; when reading multiple branches, there is no need to assume the same snapshot.
By default, writes advertise only the main trunk as the base, so the advertisement does not balloon with millions of branches, making packs easier to reuse; the cost is that stacked branches may transfer more. In their repo, the median pack is <20 KiB, and P99 <3 MiB. On the read side, have/want is mapped to stored ref versions, and cacheable packs are delivered via packfile-uri (Express hit rate >99%), so a standard machine can saturate 10 Gbps. In the background, following the LSM approach, L0 is merged into higher levels (up to L4), which both compacts and serves as a skip index, with planning at about O(log Δ).
It took about 10 weeks with Codex from project start to production (taking traffic from around April), rewritten in Rust, first with a dual-write proxy and then phased traffic switching. Results include more stable merges, and at one point >57 TB served in a single hour (>99% local cache). Improvements were fed back upstream (parallel pack downloads, explicit haves, etc.). Simulation and production figures need to be viewed separately; the core trade-off is trading causal consistency for write parallelism and reusable packs for read scaling, while preserving Git client compatibility.