Blog
Sep 29 Incident Update: Read-Only Repls
Published:
Oct 4, 2023Updated:
Aug 18, 2026
Luis Héctor Chávez
On Sep 29, we had a period of about 2.5h from 14:00 to 16:30 (Pacific Time), in which an incomplete build was pushed to our infrastructure that handles storing Repl data. This caused Repls opened during that time window to become read-only or stop working. Any Repl not opened during that timeframe was not affected. We have addressed the root cause, recovered 98% of the affected Repls, and continue to work on recovering the remaining 2%. We understand that your data not being available is unacceptable for both you and your users. This post summarizes what happened and what we're doing to improve Replit so that this does not happen again.
Technical details
Here is the timeline of what happened, all times in Pacific Time:
- On July 19, we released a new storage system. This works by recording "manifests" that represent snapshots of the filesystem at a specific point in time, and the manifests point at blocks that contain the users' data. This enables fast and efficient Repl forking.
- To improve that system, we’ve been working on a change to make obtaining filesystems and saving changes faster by batching commits and executing them asynchronously. This change was being vetted in a test cluster and we were diagnosing failure modes with writing manifests and running garbage collection that caused data corruption.
- On Sep 29 at 14:00, as part of attempting to deploy additional logging statements to the test cluster, the new (unfinished) feature was inadvertently deployed to production clusters. This is when the outage began. Any Repl loaded with this build could have potentially been put into an error state showing “read only filesystem” or other I/O errors.
- After being alerted to the issue through our monitoring, we immediately began rolling back. By 16:30 there were no faulty builds of the storage infrastructure in production. From this point forward no new Repls were affected.
- Recognizing the severity of the issue, we began a 24/7 recovery rotation.
- At 17:30 we manually repaired the filesystems of official templates. All new Repls created from scratch from this point forward were unaffected.
- The engineering team worked throughout the weekend to figure out a good process to recover the Repls from backups with minimal or no data loss.
- On Sep 30 at 02:30, a first batch of the recovery process started running, and it was able to recover 30% of the affected Repls by the time it finished running at 19:10.
- In parallel, the team was working on two tasks: recovering objects from a backup system, and working on additional versions of the recovery process that ran faster and were able to recover more Repls.
- On Sep 30 at 20:30, the second iteration of the recovery process was able to restore 70% of the affected Repls.
- On Oct 1 at 07:30, a third iteration of the recovery process was able to restore 81% of the affected Repls.
- On Oct 2 at 13:30, a fourth iteration of the recovery process was able to restore 86% of the affected Repls.
- On Oct 2 at 19:00 a fifth iteration of the recovery process was able to restore 98% of the affected Repls.
- The team is continuing to work on restoring the remaining 2% of Repls. If you are affected, reach out to [email protected] or https://report-read-only-repls.replit.app/ with a link to your Repl and we can give you more detailed updates.
What are the next steps for Replit?
The bug that triggered the incident has been fixed, but we’re making changes so that we don't run into the same or similar issues in the future: The root cause was that a change was inadvertently deployed to production clusters due to a lack of guardrails around deployments, so that's where most of our efforts are going towards. We are taking several actions:
- We're adding automated canary analysis to prevent changes with anomalous behavior from progressing from staging into production.
- We’re adjusting the speed at which storage infrastructure is deployed, so that a faulty build is unable to have widespread impact.
- In addition, we're updating the backup system to our storage solution that allows us to recover data faster and at more granular levels.
What if my Repl is still affected?
If your Repl is in the remaining 2.5% of affected Repls which have not yet been restored, you can email [email protected] or https://report-read-only-repls.replit.app/ with a link to your Repl(s) and we’ll give you more detailed updates as we work towards our goal of 100% restoration.
More
Sep 29, 2026Free the models: Harness design at the frontier
Model routers are everywhere right now, but they have a fundamental limitation. No matter if based on advanced heuristics or a small model that reads each turn and picks which LLM to use, a router will always be less capable than the model it’s choosing for. Replit Agent lets the model decide instead. The main agent, or core loop, chooses its subagents’ tier and effort, and adjusts its own as the task unfolds. Given that freedom, GPT-6 Astra hands routine implementation to less costly subagents and decides for itself where its tokens are worth spending. On both DeepSWE and Terminal-Bench, Replit Agent is Pareto-efficient against Astra on its own: no published Astra baseline costs less and scores higher. It also beats a sidekick architecture, the same setup with one long-lived worker, by 11 and 16 points. Why we scaffold less Every model release invalidates assumptions baked into the harness. As models become stronger at long-horizon tasks, they don’t need as much scaffolding at the harness layer. In practice, we’ve observed them lean more towards delegation on their own: using subagents for context management and parallelism. Recent breakthroughs, Navier–Stokes among them, came in part from coordinating swarms of agents powered by frontier models [1]. But the frontier is jagged. The strongest coding model is not necessarily the strongest at designing UIs or making slides, nor the best at writing emails. So we design our harness to let each model work its own way, with the guardrails it still needs and quality at minimum cost as the goal.
Aug 3, 2026AI adoption starts with truth
The semantic layer is the foundation AI adoption is limited by trust. A user who gets burned by a confidently wrong answer will double-check the next one, eventually routing consequential work around the system entirely. Once that happens, AI remains a tool at the edges rather than infrastructure at the center… useful, but never trusted with the workflows where its value compounds. Before a company can benefit from more capable agents, those agents need a reliable way to know what the company considers true. A semantic layer tells an agent which tables are sources of truth and how they relate. That's the floor. It is necessary, and it isn't sufficient. A semantic layer is not plumbing. It is the first act of governance for an AI-native company: the shared definitions of the business, the canonical metrics, the sources of truth, and the relationships an agent is allowed to rely on. Without it, an agent does not have a data problem. It has a language problem: several tables can each look plausible, and the model has no grounded way to know which one means "revenue," "active user," or "customer." Getting that floor right changes the shape of everything above it. A semantic layer is not the product; it is the shared contract that lets a company safely add a system of specialized capabilities instead of one generic chatbot. Once an agent can ground itself in the right entities, metrics, and relationships, it can reliably run multi-step workflows, call focused tools, retain reviewed knowledge across runs, reuse validation and analysis code, and operate through durable services where work already happens.
Jun 23, 2026Closing the loop: Evaluating and improving Replit Agent at scale
Most Replit Agent users start with an idea. They describe the goal in natural language — without a repo, test suite, or chosen framework — and expect the agent to turn it into a functioning app. The result might be a website, slide deck, mobile app, several connected artifacts, or something else entirely. Vibe coders are not usually checking diffs or test output. Success for Replit Agent is deceptively simple: the app should work when users click around. That changes the job of evaluation. A single score can help with a specific shipping decision, but it cannot tell us, week over week, whether Replit Agent is getting better for users. To answer that question, evaluation must become part of the improvement loop. Evaluation has to do more now