←Search results

Search “Reasoning”

By relevance · Total 24 results Clear search

Friday

  1. TMTPost · AGI

    NVIDIA uses a 12,000-line Megakernel to optimize Kimi K3 inference, and vLLM says AI code is now faster than human review

    Benjamin Chislett, an engineer at NVIDIA CentML, used a 12,000-line Megakernel written in CuTe DSL to optimize Moonshot AI's Kimi K3 inference. The vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x increase in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user on GB300 NVL72, a 1.68x improvement. The vLLM blog says agent-optimized inference code is now outpacing human review and maintenance speed, so the team proposed a fast-track model optimization RFC that lets reviewers judge designs from high-level abstractions rather than reviewing code line by line. The article also mentions that NVIDIA is developing optimizations for other models for vLLM, such as DeepSeek V4.1.

    Benjamin Chislett, an engineer at NVIDIA CentML, used a 12,000-line Megakernel written in CuTe DSL to optimize Moonshot AI's Kimi K3 inference. The vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x increase in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user on GB300 NVL72, a 1.68x improvement. The vLLM blog says agent-optimized inference code is now outpacing human review and maintenance speed, so the team proposed a fast-track model

    Matched fieldsTitle

    #hot-signal#Industry news#Reasoning#English text is being prepared.
  2. Story1 reports · 0 facts

    DeepSeek open-sources the inference engine DeepInfer on GitHub

    On October 29, 2026, DeepSeek open-sourced the inference engine DeepInfer on GitHub; this engine is the inference engine behind DeepSeek-V4.1-Flash. Leiphone reported the same day that DeepInfer includes technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding, and disclosed that a single cluster needs to process approximately 155 trillion tokens per day. Its Zero-Overhead Batch Scheduler lets the CPU schedule computation one round in advance, overlapping with GPU forward passes, and handles cross-round state misalignment through total_inflight_tokens accounting, Remap mapping, and DeferredRelease deferred reclamation. Currently, only offline inference capabilities are open; components such as online KV Cache storage and cross-replica load balancing have not yet been fully disclosed, and the official team has not separately published end-to-end speedup data for this scheduling mechanism relative to synchronous solutions.

    On October 29, 2026, DeepSeek open-sourced the inference engine DeepInfer on GitHub; this engine is the inference engine behind DeepSeek-V4.1-Flash. Leiphone reported the same day that DeepInfer includes technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding, and disclosed that a single cluster needs to process approximately 155 trillion tokens per day. Its Zero-Overhead Batch Scheduler lets the CPU schedule computation one round in a

    Matched fieldsTitle

  3. Story1 reports · 1 facts

    vLLM Blog Publishes Technical Retrospective on Kimi K3 Inference Optimization

    On October 16, 2026, the vLLM blog published a technical retrospective titled "Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly." According to a TMTPost·AGI report, NVIDIA CentML engineer Benjamin Chislett used a Megakernel written in 12,000 lines of CuTe DSL to optimize inference for Moonshot AI's Kimi K3; the vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x improvement in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user on GB300 NVL72, a 1.68x improvement. The vLLM blog states that agent-optimized inference code is now outpacing human review and maintenance capacity, and the team therefore proposed a fast-track model optimization RFC so that reviewers judge designs from high-level abstractions rather than reviewing code line by line. The report also mentions that NVIDIA is developing optimizations for other models in vLLM, such as DeepSeek V4.1.

    On October 16, 2026, the vLLM blog published a technical retrospective titled "Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly." According to a TMTPost·AGI report, NVIDIA CentML engineer Benjamin Chislett used a Megakernel written in 12,000 lines of CuTe DSL to optimize inference for Moonshot AI's Kimi K3; the vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x improvement in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user

    Matched fieldsTitle

Thursday

  1. AI Business

    Chinese Open Source Models Enter the Enterprise: Red Hat Uses GLM to Handle 70% of Inference Traffic

    AI Business reported that, in the week ending October 25, Chinese models accounted for more than half of the token statistics on OpenRouter, whereas about a year ago only one of the nine models on the site's weekly usage list came from China. Red Hat is internally piloting self-hosted GLM, used by up to 1,400 engineers, with about 70% of inference traffic going to open source models, and it controls costs with a frontier model quota of $300 per person per month. DataCamp, due to latency and insufficient capacity from open source model providers, is shifting from Anthropic to OpenAI and has not yet moved its AI tutor to open source models.

    AI Business reported that, in the week ending October 25, Chinese models accounted for more than half of the token statistics on OpenRouter, whereas about a year ago only one of the nine models on the site's weekly usage list came from China. Red Hat is internally piloting self-hosted GLM, used by up to 1,400 engineers, with about 70% of inference traffic going to open source models, and it controls costs with a frontier model quota of $300 per person per month. DataCamp, due to latency and insu

    Matched fieldsTitle

    #hot-signal#English text is being prepared.#Open-source ecosystem#English text is being prepared.

Saturday

  1. Huxiu

    Amazon CEO Jassy on $220 Billion in Capital Expenditure: Compute Is Still in Short Supply, Inference Costs Need to Fall by Another Order of Magnitude

    Amazon CEO Andy Jassy said in an interview on the a16z podcast that the company's capital expenditure is expected to reach $220 billion in 2026, but compute capacity is already sold out for 2026, essentially fully booked for 2027, and a large share of 2028 quotas has been locked in. He described enterprise AI demand as dumbbell-shaped: a large number of production systems in the middle have yet to be deeply integrated because inference costs are too high, and costs need to fall by another order of magnitude before an explosion in adoption occurs. Jassy also noted that 99% of AWS's top 1,000 customers use Graviton, which offers about 40% better price-performance than x86, while Trainium has a 30% to 40% advantage over comparable hardware.

    Amazon CEO Andy Jassy said in an interview on the a16z podcast that the company's capital expenditure is expected to reach $220 billion in 2026, but compute capacity is already sold out for 2026, essentially fully booked for 2027, and a large share of 2028 quotas has been locked in. He described enterprise AI demand as dumbbell-shaped: a large number of production systems in the middle have yet to be deeply integrated because inference costs are too high, and costs need to fall by another order

    Matched fieldsTitle

    #hot-signal#Industry news#English text is being prepared.#English text is being prepared.
  2. IT Home · Artificial Intelligence

    Amazon CEO Jassy: AI inference costs need to fall to about one-tenth of current levels

    In an a16z interview, Amazon CEO Andy Jassy said AI inference costs are still an order of magnitude too high, and that for AI to be widely used in enterprise software, costs need to fall to about one-tenth of current levels. He said Amazon plans to spend $220 billion in capital expenditure this year, that its main compute supply for 2026 is essentially fully booked, and that most of 2027 has also been reserved; the in-house Trainium chip offers about 30% to 40% better price-performance than comparable processors, and has already secured more than $225 billion in future revenue commitments.

    In an a16z interview, Amazon CEO Andy Jassy said AI inference costs are still an order of magnitude too high, and that for AI to be widely used in enterprise software, costs need to fall to about one-tenth of current levels. He said Amazon plans to spend $220 billion in capital expenditure this year, that its main compute supply for 2026 is essentially fully booked, and that most of 2027 has also been reserved; the in-house Trainium chip offers about 30% to 40% better price-performance than comp

    Matched fieldsTitle

    #analysis#English text is being prepared.#Reasoning#English text is being prepared.
  3. Nathan Lambert

    Nathan Lambert: DeepSeek processes 150T tokens per day on a single cluster, more than the combined total of several open-source inference platforms

    Nathan Lambert says DeepSeek processes 150T tokens per day on a single cluster, more than the combined open-source model inference volume on OpenRouter, Fireworks, Together, and BaseTen. He also posted a reconstructed comparison chart of his own based on these companies' inference volume reports, and noted that this figure corresponds to only a single cluster; it remains unclear how many clusters DeepSeek actually has.

    Matched fieldsTitle

    #editorial#Industry news#DeepSeek#English text is being prepared.
  4. Story3 reports · 3 facts

    DeepSeek open-sources inference framework DeepInfer

    On October 20, 2026, DeepSeek open-sourced its internal inference framework DeepInfer. According to Yuchen Jin, citing data disclosed by DeepSeek, a single cluster processes 155 trillion tokens per day and generates $38 in revenue per GPU-hour (off-peak pricing); DeepInfer supports both NVIDIA Blackwell and Huawei Ascend. A subsequent 36Kr report stated that its accompanying open-source deepseek-recipe supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. On October 21, swyx, citing Latent.Space data, said a single cluster processes 155T tokens per day with a 96% input cache hit rate, and argued that the lifetime value of each sold GPU has risen by about 30%. On October 24, Nathan Lambert said DeepSeek's single cluster processes 150T tokens per day, exceeding the combined open-source model inference volume of OpenRouter, Fireworks, Together, and BaseTen, and noted that the actual number of clusters is unknown; on the same day, LMSYS published a same-hardware benchmark comparison, stating that SGLang meets or exceeds DeepInfer on DeepInfer's own benchmark, with prefill throughput up to 1.04x and decode step length shortened by 1.1%–7.4% on 64x GB300.

    On October 20, 2026, DeepSeek open-sourced its internal inference framework DeepInfer. According to Yuchen Jin, citing data disclosed by DeepSeek, a single cluster processes 155 trillion tokens per day and generates $38 in revenue per GPU-hour (off-peak pricing); DeepInfer supports both NVIDIA Blackwell and Huawei Ascend. A subsequent 36Kr report stated that its accompanying open-source deepseek-recipe supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. On October 21, swyx, citing Latent.Spac

    Matched fieldsTitle

Wednesday

Thursday

  1. Latent Space

    Leash Bio's Ian Quigley on DEL data, model cheating, and an agentic chemistry reasoning sandbox

    In a Latent Space podcast interview, Leash Bio co-founder Ian Quigley discusses how the company tamed DNA-encoded libraries (DEL) and extracted clean, reproducible protein-small molecule interaction data from them. The interview mentions that in its Kaggle competition, none of the roughly 2,000 teams surpassed random performance on an unseen chemical library, and that a model can guess a chemist's identity from molecular structure, while Hermes, using only its own data, matched Boltz on public benchmarks. Ian also previewed that the team is turning years of accumulated data into tens of thousands of lab-in-the-loop simulations for agents to perform verifiable scientific reasoning; that work later developed into Minos, with a preprint, blog post, and open dataset released.

    In a Latent Space podcast interview, Leash Bio co-founder Ian Quigley discusses how the company tamed DNA-encoded libraries (DEL) and extracted clean, reproducible protein-small molecule interaction data from them. The interview mentions that in its Kaggle competition, none of the roughly 2,000 teams surpassed random performance on an unseen chemical library, and that a model can guess a chemist's identity from molecular structure, while Hermes, using only its own data, matched Boltz on public b

    Matched fieldsTitle

    #newsletter#Papers/Research#Agents#English text is being prepared.

Tuesday

Tuesday

  1. Cailian Press

    Inference spending to surpass training for the first time in 2026, rise of Token factories to change AI infrastructure investment logic

    Gartner predicts that global inference spending will reach $23.3 billion in 2026, surpassing training spending of $19 billion for the first time and accounting for 55% of total AI IaaS spending; a CAICT report predicts that in 2026 inference computing power will account for 70.5% of total AI computing load. Multiple industry insiders point out that Hugging Face weight downloads and OpenRouter Token call volumes are often conflated as overseas expansion results, but Token call volume does not equal revenue; Token going global is essentially inference services going global, which is more directly related to actual revenue. Public cloud giants, independent MaaS inference service providers, and GPU vendors are all laying out Token factories, with compliance and global low-latency distribution being the main current bottlenecks.

    Gartner predicts that global inference spending will reach $23.3 billion in 2026, surpassing training spending of $19 billion for the first time and accounting for 55% of total AI IaaS spending; a CAICT report predicts that in 2026 inference computing power will account for 70.5% of total AI computing load. Multiple industry insiders point out that Hugging Face weight downloads and OpenRouter Token call volumes are often conflated as overseas expansion results, but Token call volume does not equ

    Matched fieldsTitle

    #chinese#English text is being prepared.#English text is being prepared.#English text is being prepared.

Thursday

Thursday

Wednesday

Friday

Wednesday

Friday

  1. FactRelated story

    vLLM Blog Publishes Technical Retrospective on Kimi K3 Inference Optimization

    On October 16, 2026, the vLLM blog published a technical retrospective titled "Hyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly." According to a TMTPost·AGI report, NVIDIA CentML engineer Benjamin Chislett used a Megakernel written in 12,000 lines of CuTe DSL to optimize inference for Moonshot AI's Kimi K3; the vLLM blog reports over 840 tokens/s at batch 1 and a 1.5x to 2x improvement in decoding throughput across multiple configurations, while SemiAnalysis estimates 423 tok/s/user on GB300 NVL72, a 1.68x improvement. The vLLM blog states that agent-optimized inference code is now outpacing human review and maintenance capacity, and the team therefore proposed a fast-track model optimization RFC so that reviewers judge designs from high-level abstractions rather than reviewing code line by line. The report also mentions that NVIDIA is developing optimizations for other models in vLLM, such as DeepSeek V4.1.

    Matched fieldsTitle

    View story:vLLM Blog Publishes Technical Retrospective on Kimi K3 Inference Optimization

Tuesday

  1. Story1 reports · 0 facts

    DeepSeek open-sources inference framework DeepInfer

    On October 20, 2026, DeepSeek announced the open-sourcing of its internal production-grade inference framework, DeepInfer. According to Latent Space, the framework powers DeepSeek-V4.1-Flash, is customized based on vLLM, supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, and targets the NVIDIA SM100 and Huawei Ascend 950 platforms; the published single-cluster production figures are 155 trillion tokens processed per day, a 96% input cache hit rate, 400 ms first-token latency, and a generation speed of 230 tokens/s. The report also noted that components such as KV cache management and load balancing have not yet been fully open-sourced. The next day, DeepSeek published the DeepInfer technical report on Zhihu, disclosing internal details including the hardware selection for Prefill and Decode and the KVCache targets, and providing data showing a single cluster serving 155 trillion Tokens/day at $38 per card per hour at off-peak prices. The framework is currently open-sourced, but the KV cache system, Prefill/Decode disaggregation, and load balancing required for a complete service are not fully included in this open-source project.

    On October 20, 2026, DeepSeek announced the open-sourcing of its internal production-grade inference framework, DeepInfer. According to Latent Space, the framework powers DeepSeek-V4.1-Flash, is customized based on vLLM, supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, and targets the NVIDIA SM100 and Huawei Ascend 950 platforms; the published single-cluster production figures are 155 trillion tokens processed per day, a 96% input cache hit rate, 4

    Matched fieldsTitle

  2. FactRelated story

    DeepSeek open-sources inference framework DeepInfer

    On October 20, 2026, DeepSeek open-sourced its internal inference framework DeepInfer. According to Yuchen Jin, citing data disclosed by DeepSeek, a single cluster processes 155 trillion tokens per day and generates $38 in revenue per GPU-hour (off-peak pricing); DeepInfer supports both NVIDIA Blackwell and Huawei Ascend. A subsequent 36Kr report stated that its accompanying open-source deepseek-recipe supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. On October 21, swyx, citing Latent.Space data, said a single cluster processes 155T tokens per day with a 96% input cache hit rate, and argued that the lifetime value of each sold GPU has risen by about 30%. On October 24, Nathan Lambert said DeepSeek's single cluster processes 150T tokens per day, exceeding the combined open-source model inference volume of OpenRouter, Fireworks, Together, and BaseTen, and noted that the actual number of clusters is unknown; on the same day, LMSYS published a same-hardware benchmark comparison, stating that SGLang meets or exceeds DeepInfer on DeepInfer's own benchmark, with prefill throughput up to 1.04x and decode step length shortened by 1.1%–7.4% on 64x GB300.

    Matched fieldsTitle

    View story:DeepSeek open-sources inference framework DeepInfer