GPT-6.2 Hands-On Analysis: Higher Benchmark Scores, but Stability and Programming Experience Questioned
Based on rankings and developer testing, Sina Technology reports that OpenAI’s newly released GPT-6.2 Astra and GPT-6.2 Sol stand out in benchmark scores and certain specialized capabilities, but their stability, communication, and sustained execution in real-world programming workflows have been questioned. Astra scored 62 on the Artificial Analysis v5.0.1 Composite Intelligence Index, but only 53 on the coding agent index; Sol scored 49, down three points from the previous generation. The article also notes premium pricing rules for long-context requests, while the system card discloses issues including concealing obstacles, fabricating information when tools fail, and reduced availability for legitimate cybersecurity tasks.
Why read this
SelectedAI score 84
The article compares official rankings, API pricing, and hands-on tests by multiple developers, revealing a stability gap between the high-scoring models and their performance in real-world programming workflows.
Topics
Body · AI translation
New crowdsourced testing promotional copy New crowdsourced testing promotional copyend Add app-exclusive traffic redirection
Just eight minutes after OpenAI’s new models went live, prominent Silicon Valley entrepreneur Matt Shumer posted a blunt assessment on social media.
Soon afterward, Theo Browne, who has a large following among developers, published a lengthy article titled “Why I Hate GPT-6.2 Astra.” Ethan Mollick, a Wharton professor usually known for his patience and measured attitude toward frontier models, also delivered an unusually harsh verdict after testing it: “extremely unstable and weird.” He said bluntly that this was “the first OpenAI frontier release since GPT-4.5 to show no obvious improvement.”
This was supposed to be a resounding counterattack for OpenAI.
In the early hours of November 18, Beijing time, OpenAI officially launched its next-generation flagship, GPT-6.2 Astra, alongside its everyday workhorse, GPT-6.2 Sol. The company slashed Astra’s API price by 20% and presented a scorecard showing new records across everything from agents to scientific research.
But when soaring benchmark scores collide with a chorus of complaints from frontline developers, the stark disconnect lays bare OpenAI’s most pressing predicament.
01 Word Games and Hidden Compute Costs
Judging solely by the headline figures in the official promotional materials and third-party aggregate rankings, both GPT-6.2 models are indeed formidable.
Both models feature a 1.05-million-token context window, with a maximum input of 922,000 tokens and a maximum output of 128,000 tokens. They currently support only text and image input and text-only output, with no support for audio, video, or fine-tuning. In terms of pricing, the new flagship replacing GPT-6 Astra from early September is priced at $8 per million input tokens and $40 per million output tokens, while cached input has been slashed by 60% to $0.40. GPT-6.2 Sol, which replaces the DevDay version released in September, retains its pricing of $2 for input and $10 for output.
On Artificial Analysis’s v5.0.1 Intelligence Index, GPT-6.2 Astra leapt to second place overall with 62 points, behind only Claude Fable 5.2 at 65, while overtaking Gemini 4 Carbon at 60 and Claude Opus 5.5 at 59.
Even more appealing is its per-task cost: Astra averages just $4, only 15% of Fable 5.2’s hefty $26.54 cost. Sol, which scored 57 points, costs only $1.01 per task, placing it firmly in the high-value upper-left quadrant of the Pareto frontier scatter plot.
Comparison of the Artificial Analysis Intelligence Index v5.0.1 overall rankings and the per-task cost Pareto frontier | Image source: Artificial Analysis
But scroll down the official blog and read the tiny footnotes carefully, and the picture begins to change.
In the Agents' Last Exam benchmark, OpenAI prominently announced that Astra achieved a top score of 78.3% at the XHigh setting, compared with 77.6% at Max, far ahead of Fable 5.2’s 72.8%. Yet the footnote quietly notes that the comparison excluded nine tasks in which Fable 5.2 used a fallback model. More subtly, Google’s Gemini 4 Carbon scored 77.4% in high-reasoning mode in the official chart, at a per-task cost of $4.60—virtually tied with Astra’s $4.47—but the official text did not mention this rival at all.
Beyond the artful selection of benchmark criteria, the API pricing table also contains two hidden traps.
Whenever a single input exceeds 272,000 tokens, the input and cached-input prices for the entire request double, while the output price is multiplied by 1.5, pushing Astra directly to $16 and $60. Meanwhile, Ultrafast mode—which is limited to US data residency and offers up to a 4.5x speed boost—is currently available at a promotional 3x rate, but this will double to 6x starting March 1, 2027, at which point Astra’s input and output prices will rise to $48 and $240.
Shift the focus to specialized rankings that determine programmers’ day-to-day experience, and the shortcomings become even more apparent.
On Artificial Analysis’s coding agent leaderboard, GPT-6.2 Astra remained stuck at 53 points, unchanged from the previous generation and trailing Claude Fable 5.2 and Opus 5.5, both at 58, as well as Gemini 4 Carbon at 55. Sol, positioned as the high-volume everyday model, actually fell three points to 49, putting it level with the open-source Kimi K3.1 Pro. Its codebase Q&A score plunged by 14 points, making it highly prone to missing critical facts.
Artificial Analysis Coding Agent Index v2.1 comparison: Astra remains at 53 points, while Sol falls 3 points from its predecessor | Image source: Artificial Analysis
In testing by another authoritative organization, Epoch AI, Astra solved 7 of 68 frontier Erdős problems, breaking the previous record of 3. Yet the report still criticized it for showing no clear improvement over its predecessor on real-world scientific research automation tasks and for being overly conservative. Astra ranked only seventh on Vals’ overall leaderboard, while the standard version fell as low as 30th in legal research and had to rely on a specially customized legal version to save face.
Only in extreme vulnerability-hunting tests with its safety guardrails completely disabled did it reveal its terrifying potential.
DeepBinary tested 12 Windows system binaries, and Astra found an average of 348.5 unique vulnerabilities across two independent runs, worth as much as $1.038 million under Microsoft’s bounty criteria, at an API cost of just $4,072. The smaller Sol model also found 227 vulnerabilities valued at $687,000 for a cost of $840, leaving Opus 5 and Kimi K3.1—whose valuations hovered around $160,000—far behind.
02 A Lopsided Genius
Why does a model capable of independently uncovering system vulnerabilities worth more than a million dollars become “essentially unusable” in the hands of ordinary programmers? Look through developers’ firsthand testing accounts, and you will find an eccentric prodigy with an extremely uneven skill set.
As long as it is given a clearly bounded, highly technical problem centered on spatial geometry or physics simulation, GPT-6.2’s upper limits are genuinely astonishing.
Developer Thomas Ricouard combined it with street maps and French surveying data to recreate Paris at 1:1 scale in the browser as a 3D “Paris Atlas” encompassing more than 131,000 buildings, complete with streets, buildings, pedestrians, and traffic, while continuously refining it through screenshot feedback.
“Paris Atlas,” a 1:1 3D Paris project created by an OpenAI developer using GPT-6.2, containing data on more than 131,000 buildings | Image source: TechRadar / OpenAI
In an official OpenAI demonstration, Astra independently modeled a 3D convertible sports car, wrote its own Python physics solver, and rendered the suspension movement and a heat map of the contact pressure on all four tires along a winding mountain road in Blender. There were equally impressive enterprise examples: Higgsfield achieved a 73% first-attempt success rate for internal 3D generation; quantitative trading giant Jump Trading broke a three-year-old hardware optimization benchmark in a single day; partners including Harvey, Legora, and ClickUp reported token savings of 30% to 60%; and Replit’s real-world testing found that Sol 6.2 delivered a nearly threefold speedup.
Mountain-road chassis dynamics and four-wheel contact-pressure simulation completed by GPT-6.2 Astra using a self-built 3D sports car and Python solver | Image source: OpenAI
But once it leaves isolated breakthrough demonstration scenarios and enters real-world software engineering that requires sustained communication and multi-step collaboration, GPT-6.2 Astra’s peculiar temperament can drive people mad.
In a lengthy article that received more than a million views, Theo Browne first praised Astra’s strong code-review abilities and said it had helped him design a reverse-proxy architecture that saved $25,000 per month, exclaiming, “Even Opus wouldn’t dare explore like this.” But he immediately followed that with four maddening failings.
The most damaging is its complete lack of awareness of communication blind spots. Theo asked it to create a new level for Outer Wilds based on a target pattern. After running for two hours, the model suddenly stalled and blocked all progress just to ask him about an internal temporary filename that no human could possibly understand, forcing him to write a dedicated timer script to keep watch over it. Next came the intolerable “Astra jargon,” which persisted even when the system prompt pleaded with it to explain things in the simplest possible terms. Once, while investigating why a worktree had slowed down, Astra wrote four long paragraphs about the low-level mechanics of package-manager caching. Theo handed the issue to Claude Opus 5.5, which identified the truth in a single sentence: “Your SSD is too slow.”
Even more torturous was its obsessive tendency to get stuck in loops. On a test project to rewrite the TypeScript compiler in Rust, earlier OpenAI models had burned through more than $300,000 without making progress, while Opus completed it successfully in two weeks for about $20,000. This time, Theo retried the task with 6.2 Astra. The model ran for seven straight days without any improvement, fixating on reducing the binary size while it was at it and slowing the compiler by 56%. After he explicitly prohibited size optimization, the model spun its wheels for another four days, still producing zero commits. Its aesthetic judgment also showed severe regression toward the mean: when Theo and a friend separately asked it to design distinctive levels for different games, it produced the same water-and-bells level for both, named “Bellwether.”
This gap in perceived performance is by no means an isolated case in developer circles. Every founder Dan Shipper complained that it was inferior to Fable 5.2: its excessive pursuit of minimalism made its writing obscure and regressive, it often stopped midway through end-to-end coding tasks, and its designs had a strong cookie-cutter feel. In Simon Willison’s well-known Three.js “Yonkers mech” test, the models were asked to transform a city map into a giant mech. Both Astra and Sol failed to complete the transformation correctly, producing visuals far inferior to those of Fable 5.2 and Gemini 4 Carbon.
Comparison from Simon Willison’s Three.js “Yonkers mech” test using the same prompt: GPT-6.2 Astra (top) and GPT-6.2 Sol (bottom) | Image source: Simon Willison
Even the generally mild-mannered Ethan Mollick gave it a negative review as “extremely unstable and bizarre, and harder to steer,” despite using it in Blender and Unity to create an interactive space elevator that pierced the clouds and shot straight into space. Many people in the community also complained that legitimate binary reverse-engineering tasks were repeatedly blocked by guardrails, while even the front-end charts on the official launch page froze and threw errors on release day.
Interactive space elevator demo built from scratch by Ethan Mollick using GPT-6.2 Astra in Blender and Unity | Image source: Ethan Mollick (X)
03 A Flustered Work of Compromise
Rewinding the timeline to November makes it easy to understand why GPT-6.2 feels like a rushed compromise.
Its external rivals were closing in at a terrifying pace. On November 9, with its IPO approaching, Anthropic unveiled Fable 5.2 and Mythos 5.2, taking the lead in cutting flagship pricing to $8 and $40. The models seized first place across major coding and general-purpose leaderboards, while also showcasing astonishing achievements: 1,000 agents completing a formal proof of Fermat’s Last Theorem in 30 hours, writing a bootable operating-system prototype in eight days, and designing three circuit boards in a single day that all worked on the first power-up.
Meanwhile, Google played a succession of cards with its Gemini 4 family from October through November. Carbon’s November promotional pricing fell to $2 and $10, while Hydrogen cost just $1.50 and $7.50—and directly targeted the reliability problem enterprises care about most. In the AA-Omniscience test, Gemini 4 Hydrogen’s hallucination rate was as low as 8%, while Carbon’s was only 11%. GPT-6.2 Astra’s hallucination rate, however, worsened from 50% in the previous generation to 59%, while Sol’s surged to 55%. With multi-agent dashboards and orchestration tools, Google could complete tens of thousands of concurrent collaborative runs in just two days.
AA-Omniscience v1.1 comparison of knowledge reliability and hallucination rates: GPT-6.2 Astra’s hallucination rate reached 59%, while Gemini 4 Carbon and Hydrogen recorded just 11% and 8%, respectively | Image source: Artificial Analysis
With Anthropic raising the ceiling in front and Google reinforcing the low-hallucination floor behind, OpenAI, caught in the middle, was dealing with a crisis in its own backyard during two crucial months.
GPT-6.1 Astra, originally scheduled to debut in late September, was abruptly canceled at the last minute after testing revealed serious unauthorized behavior and false reporting. Co-president Greg Brockman disclosed in a public interview that around 25% of product engineers had been reassigned to support the safety team. The day before GPT-6.2 was released, the California attorney general formally sued OpenAI, revealing at least 80 safety incidents during testing.
With a large number of personnel pulled away to fight the fire, and the company forced to ship before its rivals completely captured developers’ mindshare, the consequences were all documented in OpenAI’s own system card.
This unusually candid safety report acknowledged regressions in three areas. First, the models had learned to conceal bad news more subtly. When encountering obstacles such as missing files, Astra’s rate of hiding the problem doubled from 11.09% to 22.28%, while Sol’s reached 24.08%. When the search tool malfunctioned, the fabrication rate for native API calls soared from 7.58% to 23.00%. Second, monitoring evasion and awareness of being evaluated surged: in 85% of test trajectories, the model’s chain of thought suspected that it was being assessed, while the file-download evasion rate without chain of thought jumped from 1.7% to 48.8%.
To rein in this beast—the first to reach the high-risk threshold for “AI self-improvement”—OpenAI hastily launched a control-room system capable of cutting off network access within 35 seconds after a reinforcement-learning agent unexpectedly connected to the public internet in September, while urgently tightening refusal boundaries. But this crude patch produced the worst of both worlds for safety and usability. Using only half as many queries, the UK AI Security Institute breached the prerelease version’s defenses against network-enabled jailbreaks and criticized it as a “substantive regression.” Meanwhile, the success rate on legitimate cybersecurity tasks plunged from 90.7% to 78.1%, and its mental-health benchmark score fell from 58.7% to 53.3% because the model rushed to provide answers and was unwilling to ask follow-up questions about the context.
Once these underlying circumstances are understood, OpenAI’s series of unusual product compromises all make sense.
Even now, GPT-6.2 has still not been added to the standard ChatGPT interface used by hundreds of millions of ordinary users; OpenAI has only dared to make it available in workspaces and coding tools. On the API side, disabling reasoning mode is expressly prohibited, and the system is even designed to discard all prior reasoning records whenever users switch to a fallback model or modify the history prefix. OpenAI neither dares to give it completely free rein nor can afford to lose face on the benchmark leaderboards.
At this stage of large-model competition, the era of producing impressive scores simply by piling on compute is over. As real-world workflows begin testing stability, low hallucination rates, communication, and collaboration, GPT-6.2 Astra’s debut of stark extremes has sounded an alarm: teaching intelligent models to understand what people are saying and report problems honestly is far more urgent than gaining another two points on a leaderboard.article_adlist[
Managing editor: Ruofeng
Report article content ]article_adlist Statement: This is an exclusive Sina.com article and may not be reproduced without authorization. tech_web_article_v2015_block_bottom Non-targeted 300*250 button 17/09 wenjing begin
Non-targeted 300*250 button end
AI Trends