OpenAI's new internal model exposed: BenchCAD 2.0 engineering test score of 0.875, higher than GPT-6 Astra

Smart Things reported on November 4 that yesterday, according to information disclosed by OpenAI, GPT-6 Astra and Codex have begun participating in physical engineering design and manufacturing-related work. Among them, GPT-6 Astra participated in designing a desktop device that can conduct voice interaction with Codex; two OpenAI team members completed the preliminary plan in less than a week, and a contract manufacturer produced a prototype.

OpenAI said that its hardware team's model usage has grown 23-fold since February of this year, and AI is moving beyond software development further into the design and engineering stages of physical products.

In addition, OpenAI also used the BenchCAD 2.0 test to evaluate the engineering capabilities of its internal models and GPT-6 Astra, focusing on the performance of its models in tasks such as reconstructing three-dimensional parts and assemblies and identifying circuit board connection relationships. In this test, OpenAI's internal model achieved an average score of 0.875, higher than GPT-6 Astra's 0.594 under the highest reasoning setting.

Beyond internal hardware projects, OpenAI also announced new research collaborations with Japanese automaker Mazda and the U.S. professional racing organization CGR (Chip Ganassi Racing) to explore the application of AI in real production scenarios in the automotive industry.

1. GPT-6 Astra participated in the design, and the preliminary plan for the voice device was completed in less than a week

OpenAI first introduced an internal hardware experiment. Justin Lin, a member of the company's research team, wanted to design a desktop device called Codex pet, so that users could directly conduct voice interaction with Codex without opening a laptop.

The device is equipped with a display and physical buttons, and a virtual character is shown on the screen. After the user presses the button, they can ask questions by voice, and the device responds by voice. In the demonstration, Justin Lin asked how many agents were currently running, and the device answered that there were 5 and proactively asked whether it should read them out one by one.

▲Internal chip design of the Codex pet device, source: OpenAI

Justin Lin said that in this way, he hoped to give the software interaction that originally took place on a computer screen a more concrete physical form. Talking to a visible object makes the interaction process feel more alive.

In the design stage, OpenAI used GPT-6 Astra to determine the device's functional requirements and had it participate in the design of the enclosure, printed circuit board (PCB), and software integration.

Specifically, GPT-6 Astra was not only responsible for the device's appearance, but also designed the structures that secure internal components, selected electronic components and completed the circuit board layout and connections, while also generating the user interface, firmware, and the software needed to connect to Codex.

OpenAI said that two team members treated this device as a side project and completed the preliminary design in less than a week. A contract manufacturer then produced a small batch of prototypes in less than a month.

However, going from a design plan to a physical device that can be produced still requires multiple rounds of review and revision. OpenAI said that GPT-6 Astra cannot yet always accurately take into account the details that manufacturers need to pay attention to in actual production, and these issues have also become directions for the company to improve the model in the future.

In the project introduction, Justin Lin also mentioned that this was the first time he had fully experienced the development process from a blank circuit board to an assembled PCB. He explained the requirements to GPT-6 Astra and provided the design plan, and the model completed the circuit board design accordingly, but there were still multiple rounds of revisions with the factory afterward. The team also produced multiple parallel design plans and different versions of the device enclosure.

2. A Tsinghua top student joined its hardware team, and model usage grew 23-fold

In addition to the design experiment for a single device, OpenAI is also using Codex and ChatGPT in its internal hardware and infrastructure teams to handle more engineering tasks.

The company said that its hardware team has used its models to create and modify computer-aided design (CAD) models, produce manufacturing drawings, build simulation environments, and review design plans, involving fields such as robotics and computing infrastructure. Since February of this year, measured by token count, the OpenAI hardware team's model usage has grown 23-fold.

▲Changes in the median daily average usage of Codex by the OpenAI hardware team. Source: OpenAI

Grace Lam, a forward-deployed engineer at OpenAI, said that Codex has accelerated the team's process of turning design ideas into engineering parts. The team can modify CAD models faster, convert three-dimensional designs into two-dimensional drawings that can be used for production, and complete simulation and finite element analysis (FEA) with less manual operation.

OpenAI technical staff member Qiong Wang introduced that hardware system designers are using Codex and ChatGPT to organize design constraints from different teams, explore system architectures, operate electronic computer-aided design (ECAD) software, and conduct design reviews. The team hopes to further promote its models' participation in the generation of circuit board production files.

Qiong Wang graduated from Tsinghua University with a bachelor's degree in electrical engineering, then earned a master's degree and a doctorate at Virginia Tech. After graduation, he stayed at the university's Center for Power Electronics Systems as a research scientist, then joined Google as a power engineer, and currently works at OpenAI. His research and work experience mainly involve power electronics and hardware system design.

▲Qiong Wang's personal résumé, source: IEEE Xplore

In addition, Jackie Lee, a member of the OpenAI infrastructure team, also introduced Codex's applications in the fields of Architecture, Engineering and Construction (AEC) and Building Information Modeling (BIM).

The model can automatically handle repetitive modeling tasks, check information in drawings and technical specifications, and coordinate design changes among different disciplines, then organize the problems found into clear to-do items, with the results finally reviewed by professional technical personnel.

III. Engineering test score of 0.875, in collaboration with Mazda and CGR

To evaluate its model's performance on engineering tasks, OpenAI also introduced the BenchCAD 2.0 test. This test mainly examines whether its model can reconstruct three-dimensional parts and assemblies, as well as identify connection relationships on circuit boards, focusing on testing its model's ability to combine programming capability with engineering knowledge to handle tasks.

The test results show that an updated internal OpenAI model achieved an average score of 0.875, while GPT-6 Astra scored 0.594 under the highest reasoning setting.

▲Comparison of GPT-6 Astra and OpenAI's internal model in the BenchCAD 2.0 test. Source: OpenAI

OpenAI states that the engineering development process is not limited to the design phase but also includes subsequent steps such as verification, manufacturing and production. Therefore, the company believes that its model's progress on engineering design tasks is only an early signal, and it remains to be verified whether it can truly improve iteration efficiency in professional workflows.

On the industry collaboration front, OpenAI is working with a small number of industry partners to test its model's ability to handle real engineering problems. Partner engineers review the results generated by its model, evaluate their accuracy and practical usability, and help OpenAI clarify which tasks can be handed over to its model and which capabilities its model still needs to improve.

Specifically, OpenAI is strengthening its collaboration with the American professional racing organization Chip Ganassi Racing (CGR) to explore the application of its model in engineering design and prototype development.

The two sides' R&D teams are testing a workflow that converts three-dimensional scan data of existing parts into editable CAD models and two-dimensional manufacturing drawings. When original design files are lost or unavailable, this approach can help engineers re-establish digital design documentation for parts.

CGR R&D engineer Matthew Ortman said that Codex can help the team convert raw scan data into solid models. He can collaborate directly with Codex in CAD software, give feedback to the model and apply it directly to the design, reducing manual adjustment work.

In addition, OpenAI also announced a new research collaboration with Japanese automaker Mazda. The company's research team and forward-deployed engineering team will work with Mazda's aerodynamics and crash safety engineers to explore how its model can assist in developing and evaluating new vehicle designs.

In specific applications, GPT-6 Astra can help Mazda aerodynamics engineers explore solutions to reduce vehicle drag. The model can design different front-end shapes, simulate airflow changes, and further optimize designs with better results. The entire process can also take into account vehicle exterior style, performance and cost.

Mazda chief engineer Akihiro Nakata said that designers sometimes question whether body shapes that facilitate airflow meet design requirements. With Astra exploring and simulating dozens of options in Codex, engineers can understand which design features affect aerodynamic performance and also clarify the room designers have for further adjustment.

▲GPT-6 Astra assists in exploring different vehicle front-end shapes and airflow changes. Source: OpenAI

From internal equipment design to participating in vehicle design and parts modeling, OpenAI is trying to bring its AI into more complex physical engineering workflows.

The circuit board design in the Codex pet project, CGR's parts modeling, and Mazda's vehicle aerodynamics research all demonstrate different ways in which AI participates in real engineering tasks.

How to enable its model to more accurately understand engineering constraints and extend design capabilities to verification and manufacturing will become an important topic for OpenAI to further deepen its presence in the physical engineering field.

Source: OpenAI