Hao AI Lab announced that JetSpec++'s models, code, and inference engine are all open-sourced, along with links to the paper, code, and Hugging Face. The method pipelines draft generation and verification, achieving up to 1.54x end-to-end speedup over DSpark and 7.45x over autoregressive decoding on Qwen3-8B, reaching 1,361 TPS in BF16; with NVFP4 and kernel optimizations, it exceeds 4,000 TPS on two GB200s.
X.PIN relays benchmark data for Tianshu Zhixin's TianGai 300, claiming that its throughput in standard inference tests is 1.05 times that of Nvidia's H200. Under MoE workloads, performance improves by 15% and first-token latency drops by 30%, and when fine-tuning Qwen3-32B, its speed is 1.33 times that of the H100.
Benjamin Chislett, an engineer at NVIDIA CentML, used a 12,000-line Megakernel written in CuTe DSL to optimize Moonshot AI's Kimi K3 inference. The vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x increase in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user on GB300 NVL72, a 1.68x improvement. The vLLM blog says agent-optimized inference code is now outpacing human review and maintenance speed, so the team proposed a fast-track model optimization RFC that lets reviewers judge designs from high-level abstractions rather than reviewing code line by line. The article also mentions that NVIDIA is developing optimizations for other models for vLLM, such as DeepSeek V4.1.
Benjamin Chislett, an engineer at NVIDIA CentML, used a 12,000-line Megakernel written in CuTe DSL to optimize Moonshot AI's Kimi K3 inference. The vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x increase in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user on GB300 NVL72, a 1.68x improvement. The vLLM blog says agent-optimized inference code is now outpacing human review and maintenance speed, so the team proposed a fast-track model
OpenRouter has released the biggest improvement to its provider router since its founding. The new routing no longer primarily ranks by price; instead, it rewards the highest-quality reasoning based on the actual cost per request, keeping requests low-cost while also balancing speed and reliability. Providers that deliver higher value automatically receive more traffic, giving them an incentive to keep improving, and as providers improve, users' reasoning experience on OpenRouter improves in tandem.
OpenRouter has released the biggest improvement to its provider router since its founding. The new routing no longer primarily ranks by price; instead, it rewards the highest-quality reasoning based on the actual cost per request, keeping requests low-cost while also balancing speed and reliability. Providers that deliver higher value automatically receive more traffic, giving them an incentive to keep improving, and as providers improve, users' reasoning experience on OpenRouter improves in tan
OpenRouter has released the largest improvement to its provider router since its founding. The new router no longer primarily ranks by price, but instead rewards the highest-quality inference based on the actual cost of a request, keeping requests low-cost while also balancing speed and reliability. Providers that deliver higher value automatically receive more traffic, giving them an incentive to keep improving, and as providers improve, users' inference experience on OpenRouter improves in tandem.
OpenRouter has released the largest improvement to its provider router since its founding. The new router no longer primarily ranks by price, but instead rewards the highest-quality inference based on the actual cost of a request, keeping requests low-cost while also balancing speed and reliability. Providers that deliver higher value automatically receive more traffic, giving them an incentive to keep improving, and as providers improve, users' inference experience on OpenRouter improves in tan
Rohan Paul quotes Sam Altman's view on local inference: from a global resource allocation perspective, having the world's limited DRAM sitting idle in everyone's computers and only 5% utilized is terrible. Rohan Paul adds that on a per-token cost basis, data centers are clearly superior, and a local model with batch size 1 is wasteful by any utilization metric, but it is also the only part of the entire tech stack that no one can take down on you.
Eclypsium demonstrated, with NVIDIA's support, that an attacker using a compromised DPU and lax memory isolation can modify runtime model weights during inference, causing the model's answers to change while the model files on disk and hash verification still appear normal. The experiment was conducted in an NVIDIA Grace Hopper environment and depended on conditions such as IOMMU passthrough and lax PCIe isolation, which the researchers explicitly noted are not Linux default settings. The report brings DPUs, GPUs, management controllers and their firmware into the AI security boundary, and notes that model signing, secure boot, runtime isolation and continuous monitoring each answer different questions.
Eclypsium demonstrated, with NVIDIA's support, that an attacker using a compromised DPU and lax memory isolation can modify runtime model weights during inference, causing the model's answers to change while the model files on disk and hash verification still appear normal. The experiment was conducted in an NVIDIA Grace Hopper environment and depended on conditions such as IOMMU passthrough and lax PCIe isolation, which the researchers explicitly noted are not Linux default settings. The report
On October 29, Akamai released beta versions of two products, Akamai Inference and Akamai Agent Sandbox, allowing users to run AI models on Akamai Cloud and securely execute code generated by AI agents. Akamai Inference supports 12 open-weight models, including OpenAI's gpt-oss-120b, Google's Gemma 4, and Nvidia's Nemotron 3 Ultra, and provides an OpenAI-compatible API. Agent Sandbox is based on WebAssembly isolation technology and restricts code access to computing resources, network connections, and system functions.
On October 29, Akamai released beta versions of two products, Akamai Inference and Akamai Agent Sandbox, allowing users to run AI models on Akamai Cloud and securely execute code generated by AI agents. Akamai Inference supports 12 open-weight models, including OpenAI's gpt-oss-120b, Google's Gemma 4, and Nvidia's Nemotron 3 Ultra, and provides an OpenAI-compatible API. Agent Sandbox is based on WebAssembly isolation technology and restricts code access to computing resources, network connection
Amazon Science and Booking.com combined knowledge distillation with Medusa to compress a 70B-parameter Llama-3.1 teacher model into a 1.1B TinyLlama student model, increasing inference speed by 10 to 20 times and reducing cost to less than one-third while maintaining the teacher model's quality. The approach has been used in three features—AI Trip Planner, Smart Filters, and accommodation description generation—and won the Best Paper Award in the Industry Track at ACL.
Baseten has become Postman's recommended inference partner for enterprise agents, responsible for providing production inference and post-training support for agents built on Postman Agent Stack. Baseten says its seven model APIs released in 2026 all ranked in the top two in Artificial Analysis's throughput and latency rankings, with Beam 1.0, DeepSeek V4.1 Flash and others taking first place. Harvey used Baseten to post-train a 27B open-source model, matching closed-source frontier models on legal agent benchmarks, while the latter cost about $50 per task.
Baseten has become Postman's recommended inference partner for enterprise agents, responsible for providing production inference and post-training support for agents built on Postman Agent Stack. Baseten says its seven model APIs released in 2026 all ranked in the top two in Artificial Analysis's throughput and latency rankings, with Beam 1.0, DeepSeek V4.1 Flash and others taking first place. Harvey used Baseten to post-train a 27B open-source model, matching closed-source frontier models on le
On its fiscal 2027 first-quarter earnings call, SanDisk said AI inference is reshaping the NAND flash supply and demand landscape, that customers have underestimated the NAND capacity needed to run AI infrastructure, and that its new business model order pipeline extends to 2030 and even further out. About 50% of bit shipments in fiscal 2027 have already been locked in under NBM agreements, and that is expected to rise to about two-thirds in fiscal 2028. SanDisk also announced that the first high-bandwidth flash (HBF) memory die has completed tape-out and entered the manufacturing stage, offering read bandwidth approaching HBM with NAND-class capacity for AI inference, with first samples planned for delivery in the next calendar year.
On its fiscal 2027 first-quarter earnings call, SanDisk said AI inference is reshaping the NAND flash supply and demand landscape, that customers have underestimated the NAND capacity needed to run AI infrastructure, and that its new business model order pipeline extends to 2030 and even further out. About 50% of bit shipments in fiscal 2027 have already been locked in under NBM agreements, and that is expected to rise to about two-thirds in fiscal 2028. SanDisk also announced that the first hig
ARC Prize published the test results of Meta's Muse Spark Open on the public ARC-AGI tasks: ARC-AGI-2 improves with reasoning scale, with medium averaging 55k tokens per task for 29.4%, high at 150k tokens for 51.0%, and xhigh at 180k tokens for 57.2%. ARC-AGI-1 already reached 92.8% at high and rose to 94.0% at xhigh. The model scores 57.5% on ARC-AGI-2 at $0.73 per task, and 89.0% on ARC-AGI-1 at $0.29 per task; its scores are close to those of Haiku 5.5 and GPT-6 Luna, but its cost is about 9 times theirs.
ARC Prize published the test results of Meta's Muse Spark Open on the public ARC-AGI tasks: ARC-AGI-2 improves with reasoning scale, with medium averaging 55k tokens per task for 29.4%, high at 150k tokens for 51.0%, and xhigh at 180k tokens for 57.2%. ARC-AGI-1 already reached 92.8% at high and rose to 94.0% at xhigh. The model scores 57.5% on ARC-AGI-2 at $0.73 per task, and 89.0% on ARC-AGI-1 at $0.29 per task; its scores are close to those of Haiku 5.5 and GPT-6 Luna, but its cost is about 9
Yuchen Jin shares a conversation with a friend: a year ago, they built an ultra-fast inference engine for image models, and the entire engine was written by hand. He exclaimed in response, "Holy shit. Just ONE year ago."
Perplexity CEO Aravind Srinivas posted a question: Which reasoning tasks are you still stuck on at the level of intelligence, rather than compute or funding?
The Ant Ling team, in collaboration with the NVIDIA AI team, optimized the inference of the open-weight model Ling-3.0-flash on a single NVIDIA DGX Spark using three techniques. On 1,319 GSM8K problems, decoding throughput increased from 37.81 tok/s to 53.89 tok/s (+42.5%), average latency dropped from 15.21s to 11.49s (-24.5%), and accuracy showed no statistically detectable change.
AI inference is shifting from a one-off computational event to a continuous production workload, whose cost depends on traffic patterns, latency targets, model size, context length, and concurrency. The Stanford 2025 AI Index shows that the query price for GPT-3.5-level performance on MMLU fell from $20 per million tokens in November 2022 to $0.07 per million tokens for Gemini 1.5 Flash-8B in October 2024, a decline of more than 280x in about 18 months. But reasoning models and agents generate more tokens, call models multiple times, and occupy accelerator memory for extended periods, increasingly separating cost per token from cost per completed task; production environments should measure cost per successful task at an agreed service level.
AI inference is shifting from a one-off computational event to a continuous production workload, whose cost depends on traffic patterns, latency targets, model size, context length, and concurrency. The Stanford 2025 AI Index shows that the query price for GPT-3.5-level performance on MMLU fell from $20 per million tokens in November 2022 to $0.07 per million tokens for Gemini 1.5 Flash-8B in October 2024, a decline of more than 280x in about 18 months. But reasoning models and agents generate m
AI Business reported that, in the week ending October 25, Chinese models accounted for more than half of the token statistics on OpenRouter, whereas about a year ago only one of the nine models on the site's weekly usage list came from China. Red Hat is internally piloting self-hosted GLM, used by up to 1,400 engineers, with about 70% of inference traffic going to open source models, and it controls costs with a frontier model quota of $300 per person per month. DataCamp, due to latency and insufficient capacity from open source model providers, is shifting from Anthropic to OpenAI and has not yet moved its AI tutor to open source models.
AI Business reported that, in the week ending October 25, Chinese models accounted for more than half of the token statistics on OpenRouter, whereas about a year ago only one of the nine models on the site's weekly usage list came from China. Red Hat is internally piloting self-hosted GLM, used by up to 1,400 engineers, with about 70% of inference traffic going to open source models, and it controls costs with a frontier model quota of $300 per person per month. DataCamp, due to latency and insu
DeepSeek has open-sourced its inference engine DeepInfer on GitHub—the inference engine behind DeepSeek-V4.1-Flash—which includes technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding, and discloses that a single cluster needs to process about 155 trillion tokens per day. Its Zero-Overhead Batch Scheduler lets the CPU schedule computation one round ahead, overlapping with GPU forward passes, and handles cross-round state misalignment through total_inflight_tokens accounting, Remap mapping, and DeferredRelease deferred reclamation. What is currently available is offline inference capability; components such as online KV Cache storage and cross-replica load balancing have not yet been fully disclosed, and the official team has not separately published end-to-end speedup data for this scheduling mechanism relative to synchronous solutions.
DeepSeek has open-sourced its inference engine DeepInfer on GitHub—the inference engine behind DeepSeek-V4.1-Flash—which includes technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding, and discloses that a single cluster needs to process about 155 trillion tokens per day. Its Zero-Overhead Batch Scheduler lets the CPU schedule computation one round ahead, overlapping with GPU forward passes, and handles cross-round state misalignment
Akamai has released two services, Akamai Inference and Akamai Agent Sandbox: the former provides 12 open-weight models via an OpenAI-compatible API, with parameter counts ranging from 8B to 2.8T, including NVIDIA Nemotron 3 Ultra, OpenAI gpt-oss-120b, and Google Gemma 4; the latter runs code generated by AI agents in isolation based on WebAssembly, and is released as an open-source component of the Spin framework. Akamai has also expanded its NVIDIA Blackwell Ultra GPU clusters, with each eight-GPU node equipped with 2.3 TB of HBM3e memory.
Akamai has released two services, Akamai Inference and Akamai Agent Sandbox: the former provides 12 open-weight models via an OpenAI-compatible API, with parameter counts ranging from 8B to 2.8T, including NVIDIA Nemotron 3 Ultra, OpenAI gpt-oss-120b, and Google Gemma 4; the latter runs code generated by AI agents in isolation based on WebAssembly, and is released as an open-source component of the Spin framework. Akamai has also expanded its NVIDIA Blackwell Ultra GPU clusters, with each eight-
Oracle announced that it will deploy Google TPU on OCI, along with inference chips from OpenAI, Cerebras, SambaNova, and Positron AI, giving customers more computing options beyond Nvidia GPUs. Oracle says it will become the first hyperscale cloud provider other than Google Cloud to offer TPU; TPU and the Cerebras wafer-scale engine are planned to launch in 2027, and Positron Atlas is already available. At the same time, Oracle is expanding its partnership with Nvidia, with the Vera Rubin NVL72 system and Vera CPU now generally available on OCI.
Oracle announced that it will deploy Google TPU on OCI, along with inference chips from OpenAI, Cerebras, SambaNova, and Positron AI, giving customers more computing options beyond Nvidia GPUs. Oracle says it will become the first hyperscale cloud provider other than Google Cloud to offer TPU; TPU and the Cerebras wafer-scale engine are planned to launch in 2027, and Positron Atlas is already available. At the same time, Oracle is expanding its partnership with Nvidia, with the Vera Rubin NVL72
