AI Trends

Trending story

Hao AI Lab used vstack to discover and submit fixes for more than 20 issues in SGLang

6 reports6 story updatesUpdated Nov 18 02:27

Understand this story

AI overview

Hao AI Lab collaborated with the SGLang community to use vstack to identify more than 20 concurrency and multi-step state defects in SGLang, and submitted fix PRs. Among them, the weight update race condition, before the fix on GB200, caused in-flight steps to compute tokens, logprobs, and KV with the new weights in 8 out of 8 trials; after the fix, this dropped to 0 out of 8. During a 364.69-second decode server hang, /health still returned 200 in 363 out of 363 cases. Another class of issues involved streaming stop-string truncation: when a byte-level tokenizer splits a multi-byte character across multiple tokens, the trailing decoded U+FFFD corrupts the prefix check, causing the beginning of the stop string to be sent to the client prematurely; tested on GB200 with Qwen3-4B and default parameters, in 8 streaming comparisons between the chat and completions endpoints, truncation did not occur in 8 out of 8 cases, and after removing the trailing U+FFFD, this dropped to 0. The fixes include adding a GPU stream wait before parameter writes and restricting only newly preallocated memory, among others.

AI-generated from public reports

Latest development · Report

Hao AI Lab expands SGLang validation coverage with vstack

Developments

1 confirmed developments

  1. Hao AI Lab used vstack to discover and submit fixes for more than 20 issues in SGLang

    Source:UCSD Hao AI Lab

Reports and discussion timeline

Browse reports and discussion by time. Their evidence levels remain distinct.

Nov 18

  1. UCSD Hao AI LabFirst-party

    Hao AI Lab expands SGLang validation coverage with vstack

    Hao AI Lab stated that, beyond the 5 cases already shared, the team is continuing to expand validation coverage of SGLang with vstack, and after validating on GB200, will submit confirmed fixes upstream.

  2. UCSD Hao AI LabFirst-party

    Hao AI Lab fixes SGLang multi-GPU communication misalignment with vstack

    Hao AI Lab used vstack to locate and fix a multi-GPU rank misalignment in SGLang caused by the prefill delayer and interval min batch size under DP attention: busy ranks skipped prefill and went straight into the per-round synchronization, while idle ranks entered the delayer negotiation, and the communication library then aborted. After making ranks with delayed prefill still participate in the negotiation with zero prefill, all 30 cross-threshold runs on GB200 completed with no aborts.

  3. UCSD Hao AI LabFirst-party

    Hao AI Lab uses vstack to discover SGLang request cancellation loss issue

    Hao AI Lab used vstack to discover a cancellation loss issue under DP attention in SGLang: a fast abort may arrive before its generation request and be dropped, causing the request to continue running until max_new_tokens. On GB200, 33 of 204 aborts issued within 20 ms were lost and ran to max_new_tokens; after changing to transmitting aborts in order on each rank's work socket, this dropped to 0/204. A fix PR has been submitted for this issue.

  4. UCSD Hao AI LabFirst-party

    Hao AI Lab used vstack to find 20+ concurrency bugs in SGLang

    Hao AI Lab and the SGLang community used vstack to identify 20+ concurrency and multi-step state bugs in SGLang and submitted fix PRs. Among them, the weight update race condition, before the fix on GB200, caused in-flight steps to compute tokens, logprobs, and KV with the new weights in 8/8 trials; after the fix, this was 0/8. During the 364.69 seconds the decode server was stuck, /health still returned 200 in 363/363 cases. Fixes include adding a GPU stream wait before parameter writes and restricting only new preallocations, among others.

  5. UCSD Hao AI LabFirst-party

    Hao AI Lab uses vstack to discover SGLang decode server hang of 364 seconds

    Hao AI Lab used vstack to discover a PD-disaggregated decode server hang issue in SGLang: a single aborted request freezes the transfer queue, while /health still returns 200. On GB200, the decode server stalled for 364.69 seconds, and all 363 health checks passed; gating only new preallocations allows already-arrived transfers to complete and resolves the hang. This issue is one of 20+ issues and fix PRs proposed in collaboration with the SGLang community.

  6. UCSD Hao AI LabFirst-party

    Hao AI Lab uses vstack to discover SGLang streaming stop-string truncation defect

    Hao AI Lab uses vstack to discover a streaming stop-string truncation defect in SGLang: when a byte-level tokenizer splits a multi-byte character across multiple tokens, a trailing decoded U+FFFD breaks the prefix check, causing the beginning of the stop string to be sent to the client prematurely. Tested on GB200 with Qwen3-4B and default parameters, 8 out of 8 streaming comparisons across the chat and completions endpoints were not truncated, and after additionally checking the text with the trailing U+FFFD removed, this dropped to 0. This issue is one of 20+ issues raised in collaboration with the SGLang community.

Related stories

  1. Related storiesHao AI Lab uses vstack to conduct agentic formal verification on SGLang
  2. Related storiesMicrosoft Research and others release open-source plugin TokenEconomizer
  3. Related storiesMicrosoft and NVIDIA Launch New Windows Collaboration
  4. Related storiesGoogle open-sources ABFS on GitHub