Search “Reasoning”

OpenRouter Launches New Provider Router, Rewarding High-Quality Reasoning Based on Actual Cost per Request

OpenRouter has released the biggest improvement to its provider router since its founding. The new routing no longer primarily ranks by price; instead, it rewards the highest-quality reasoning based on the actual cost per request, keeping requests low-cost while also balancing speed and reliability. Providers that deliver higher value automatically receive more traffic, giving them an incentive to keep improving, and as providers improve, users' reasoning experience on OpenRouter improves in tandem.

OpenRouter launches a new version of its provider router, rewarding high-quality inference based on actual cost

OpenRouter has released the largest improvement to its provider router since its founding. The new router no longer primarily ranks by price, but instead rewards the highest-quality inference based on the actual cost of a request, keeping requests low-cost while also balancing speed and reliability. Providers that deliver higher value automatically receive more traffic, giving them an incentive to keep improving, and as providers improve, users' inference experience on OpenRouter improves in tandem.

DeepSeek Open-Sources Inference Engine DeepInfer, Discloses That a Single Cluster Processes About 155 Trillion Tokens per Day

DeepSeek has open-sourced its inference engine DeepInfer on GitHub—the inference engine behind DeepSeek-V4.1-Flash—which includes technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding, and discloses that a single cluster needs to process about 155 trillion tokens per day. Its Zero-Overhead Batch Scheduler lets the CPU schedule computation one round ahead, overlapping with GPU forward passes, and handles cross-round state misalignment through total_inflight_tokens accounting, Remap mapping, and DeferredRelease deferred reclamation. What is currently available is offline inference capability; components such as online KV Cache storage and cross-replica load balancing have not yet been fully disclosed, and the official team has not separately published end-to-end speedup data for this scheduling mechanism relative to synchronous solutions.

Cohere Model Vault Launches Verifiable Confidential AI Inference

Cohere has launched confidential computing capabilities on Model Vault, currently in public beta, running on NVIDIA GPUs on Microsoft Azure. The feature combines confidential CPU and GPU execution, OHTTP end-to-end encryption of request content, and remote attestation, allowing customers to verify the target environment before releasing sensitive data. Cohere plans to open-source the confidential inference stack and advance support for NVIDIA Blackwell and B200 GPUs.

DeepSeek open-sources the Deepinfer inference framework, lowering the cost of self-hosted model deployment

DeepSeek has open-sourced its internal inference framework Deepinfer on GitHub, with the goal of reducing the compute and energy costs of deploying models on your own. During the inference stage, the framework distributes load more evenly across available processor cores, uses speculative decoding to speed up responses, and improves the structure of results before output; it currently supports only DeepSeek-V4.1-Flash, with more of its own models to be opened up later. On the hardware side, it requires Nvidia GPUs with the SM100 architecture or Huawei Ascend 950 NPUs, so it is not aimed at individual users.

English translation is being prepared.

DeepSeek Open-Sources Its Internal Inference Framework DeepInfer

DeepSeek has open-sourced its internal inference framework DeepInfer, which it says powers the high-throughput, low-latency inference of DeepSeek-V4.1-Flash. The framework supports both Nvidia and Huawei Ascend hardware, and offers capabilities such as distributed inference, speculative decoding, SWA Bounded Replay, and structured output. DeepSeek also released deepseek-recipe, a production-grade tokenization component fully aligned with the official API.

English translation is being prepared.

English translation is being prepared.

DeepSeek open-sources the DeepInfer inference framework, processing 155 trillion tokens per day in a single cluster

DeepSeek has open-sourced DeepInfer, its self-developed inference framework that supports DeepSeek-V4.1-Flash, and published production data: a single cluster processes 155 trillion tokens per day, 96% input cache hit rate, 400ms first-token latency, and 230 tokens/s generation speed. The framework is customized based on vLLM and supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, with target platforms NVIDIA SM100 and Huawei Ascend 950. DeepSeek also states that components such as KV cache management and load balancing are not yet fully open-sourced, while the vLLM team says that under a matching GB200 configuration, decoding throughput is within 6% of the results published by DeepInfer.

DeepSeek open-sources the DeepInfer inference framework, processing 155 trillion tokens per day on a single cluster

DeepSeek has open-sourced DeepInfer, the production inference framework supporting V4.1 Flash, and published production data including 155 trillion tokens processed per day on a single cluster, a 96% input cache hit rate, an average 400ms time to first token, and 230 tokens/s generation. The framework is customized based on vLLM and uses the Apache 2.0 license, supports distributed inference, speculative decoding, SWA Bounded Replay and structured output, and targets NVIDIA SM100 and Huawei Ascend 950. DeepSeek says components such as KV cache management and load balancing are not fully open-sourced because they are coupled with internal infrastructure, while the vLLM team reports that under matched GB200 configurations, decoding throughput differs from DeepInfer's published results by less than 6%.

English translation is being prepared.

English translation is being prepared.

English translation is being prepared.

English translation is being prepared.

DeepSeek open-sources inference framework DeepInfer and codec components under Apache 2.0

DeepSeek has open-sourced, under the Apache 2.0 license, the inference framework DeepInfer that powers its online services and the API-identical codec component deepseek-recipe, and the company says inference code for future new models will also continue to be merged into this repository. The framework includes high-performance operators, distributed inference, speculative decoding, SWA Bounded Replay, and structured output, but the README shows that this release only includes the offline inference engine; the KV cache system, prefill-decode disaggregated deployment, and multi-replica load balancing are not included. On hardware, it currently supports only NVIDIA's Blackwell architecture SM100 and Huawei Ascend 950, and the only supported model is V4.1-Flash.

English translation is being prepared.

DeepSeek open-sources inference framework DeepInfer and token codec component deepseek-recipe

DeepSeek has open-sourced its internal large-scale cloud service inference framework DeepInfer and the production-grade token codec component deepseek-recipe. The former supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, and can run on NVIDIA and Huawei Ascend platforms. DeepSeek says the high-throughput, low-latency inference service of DeepSeek V4.1 Flash is based on this framework, with a single cluster serving 155 trillion Tokens/day in throughput, at 38 US dollars per card per hour calculated at off-peak prices. The processing logic of deepseek-recipe is fully aligned with the official DeepSeek API. The team also explained that the KVCache management and load balancing components are deeply coupled with internal infrastructure, so the relevant code has not yet been fully open-sourced.

DeepSeek open-sources DeepInfer, the inference engine behind its API, with support for both Nvidia and Huawei Ascend

DeepSeek has open-sourced DeepInfer, its in-house inference framework used to serve its models in production. Released under the Apache 2.0 license, it lets developers run DeepSeek-V4.1-Flash on their own infrastructure, with support for subsequent models planned. The framework includes optimized compute kernels and communication libraries such as FlashMLA, DeepGEMM, and DeepEP, supports speculative decoding, asynchronous scheduling, and structured output, and also open-sources the deepseek-recipe library (available in Rust and Python versions) for handling request and response transformations. On the hardware side, it supports Nvidia SM100 (Blackwell) architecture and the Huawei Ascend 950 NPU series from day one; older architectures such as Hopper are not listed. Batch-invariant inference and deterministic sampling are available only on Nvidia hardware, and the published performance data is also based on Nvidia GPUs. The current version supports offline inference but lacks complete production serving components such as KV cache management, cross-machine inference stage separation, and replica load balancing.

English translation is being prepared.