量化感知修复（QAH）：4-bit 压缩模型性能超越其全精度原版

Why the usual healing methods fall short here

Most efficiency pipelines follow the same three steps: compress the architecture, quantize the compressed weights, then heal the damage. The difference between methods is entirely in that last step.

The dominant healing recipe is quantization-aware training (QAT). It inserts fake-quantization operators into the forward pass and keeps fine-tuning the model on a task loss, so the weights learn to tolerate the low-precision representation. In practice this means re-running an already expensive multi-stage post-training process, supervised fine-tuning, RLHF, agentic tuning, through a noisier, lower-precision forward pass. It is costly, and as our results show, it can also become unstable if training continues too long past its best point.

An alternative, quantization-aware distillation (QAD), avoids re-running that history. Instead of a task loss, it distills a frozen full-precision teacher directly into the quantized student through a KL-divergence loss on the output logits. This works well when the only change is quantization, because a genuine full-precision version of the exact same model exists to act as teacher. But once a model has gone through structural compression, fewer layers, heads, or neurons, and not just fewer bits, that assumption breaks. There is no independently trained full-precision version of the smaller architecture. The only candidate teacher is the recovered bfloat16 checkpoint, which is itself a distilled approximation of the original model. Distilling from it anchors the quantized student to a degraded target and caps its accuracy at that recovered checkpoint's own ceiling.

So the question of how to heal a model that has been both structurally compressed and quantized was, until now, genuinely open.

Our approach

QAH removes that ceiling with one change: it distills directly from the original, pre-compression model rather than from the recovered one. Teacher and student do not even share an architecture. The teacher is full-size and full-precision, the student is half the size and running in MXFP4. Because a teacher's output distribution is architecture-agnostic, nothing about the size or shape mismatch prevents the transfer. The student never sees hard labels, only the teacher's output distribution, matched through KL divergence on the logits.

This reframes what the quantization stage is doing. Under QAH it is no longer a lossy postprocessing step applied after healing is finished. It is a second, full pass of distillation against the original teacher, supervision that the bfloat16 checkpoint never received. The 4-bit student is not compensating for information lost to quantization; it is picking up information the earlier recovery stage did not have the time or data to transfer.

There is also a stability benefit that falls out of the loss itself. Because KL distillation ties the student to a fixed teacher distribution, once the student catches up there is no further pressure for it to drift. A cross-entropy task loss, by contrast, keeps pushing the student toward hard labels indefinitely. That difference turns out to matter for both accuracy and training stability, as the comparison below shows.

To make QAH work at long context, where the healing corpus includes documents up to 32k tokens, we reuse the memory-efficient chunked KL-divergence loss from our companion paper on efficient distillation. That loss computes the KL one slice of the sequence at a time and never materializes the full vocabulary-by-sequence grid, which is what makes 32k-token healing fit inside a fixed GPU memory budget. We covered the mechanics of that loss in a previous post .

QAH overview. After structural compression and quantization, capabilities drop sharply. QAH distills from the original model, a frozen teacher whose logits are precomputed offline, rather than from the recovered checkpoint. Source: paper Figure 1.

Results

We applied QAH to a GPT-OSS 120B model, compressed to 60B parameters and recovered in bfloat16, then re-quantized to MXFP4 under QAH. The natural comparison is against that same 60B model's bfloat16 checkpoint, the best full-precision version of this architecture that exists. The QAH model wins on 7 of the 9 benchmarks.

The two benchmarks where QAH trails, MMLU-Pro and SciCode, lose by less than a point and a half. Everywhere else the 4-bit model is ahead of its own 16-bit source, and the largest gains land on exactly the capabilities compression usually damages most: long-context reasoning (+7.4 on AA-LCR) and math (+5.6 on AIME 2025).

The comparison against the original 120B teacher is just as telling. Despite running at half the teacher's parameter count and roughly a quarter of its weight memory, the QAH model surpasses the full-size teacher on LiveCodeBench (66.5 vs. 66.0) and comes within 1.6 points on GPQA Diamond (67.4 vs. 69.0). The largest remaining gap against the teacher is on AA-LCR, an extreme long-context benchmark where the capacity lost to compression is intrinsically the hardest to recover.

The 4-bit QAH model matches or beats its bfloat16 source on 7 of 9 benchmarks, and beats the full-size teacher on LiveCodeBench. Source: paper Figure 2.

QAH against QAT, head to head

To isolate the effect of the loss function from everything else, we also compared QAH directly against QAT under matched conditions, quantizing a GPT-OSS 9B model to MXFP4 and tracking average performance across MMLU-Pro, LiveCodeBench, and GPQA Diamond as training progresses.

Both methods reach a similar peak, 54.9 for QAH against 54.6 for QAT, so on best-case accuracy they are effectively tied. The difference is in how they get there and what happens afterwards. QAH reaches its peak in about 100 steps, roughly 7 times faster than QAT's 700, and then stays within about two points of that peak for the rest of training. QAT collapses sharply once past its peak, shedding nearly 19 points by step 1,200.

The practical consequence is a real deployment risk difference. A QAT checkpoint needs careful early stopping against a held-out signal to avoid shipping a model that has already started to degrade, whereas a sufficiently trained QAH checkpoint can be served safely because it simply does not drift. This is consistent with the mechanism: KL distillation against a frozen teacher gives the student no incentive to move once it matches the teacher, while a cross-entropy objective keeps pushing on hard labels and eventually erodes capabilities the model inherited from the original.

QAH peaks at 54.9 in roughly 100 steps and holds; QAT reaches 54.6 only around step 700, then loses nearly 19 points by step 1,200. Source: paper Figure 3.

What this changes in practice

The accuracy story comes paired with the efficiency story that motivated compression in the first place. At 4-bit precision the QAH model uses roughly 4 times less weight memory than the bfloat16 student, and at half the parameter count of the 120B teacher it roughly halves compute per token, which is what lets it run on substantially smaller hardware. For model families that ship in bfloat16 rather than 4-bit, the combined parameter and precision reduction would be closer to 8 times less compute per token.

The takeaway is that a compressed, 4-bit model does not have to be a lower-accuracy version of its full-precision counterpart. With this healing recipe it can be smaller, cheaper to serve, and more accurate at the same time, and it reaches that point in a fraction of the training a QAT recipe would need. Quantization stops being a tax you pay for efficiency and becomes an extra opportunity to teach the model.

This work is part of Multiverse Computing's ongoing research into making large models smaller and cheaper to run without giving up the capabilities that make them useful. It sits alongside our companion work on efficient distillation, which supplies the long-context training machinery QAH depends on.

Models mentioned in this article 2

MultiverseComputingCAI/Hypernova-60B-2605

Text Generation • 59B • Updated Aug 7 • 1.71k • 21 openai/gpt-oss-120b Text Generation • 117B • Updated Aug 26, 2025 • 4.48M • 5.34k Papers mentioned in this article 1 Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs Paper • 2608.20953 • Published Aug 21 • 14 More from this author

15 September 29, 2026 Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem 33 September 21, 2026 Community Making a large language model smaller almost always comes with a cost. The now-standard recipe for efficient deployment is to compress the architecture first, cutting the parameter count by removing layers, heads, or neurons, and then quantize the remaining weights down to 4 bits to shrink memory and compute further. Both steps save a lot, but together they systematically degrade the capabilities people actually care about: reasoning, mathematical problem-solving, and code generation. Because of this, serious deployment pipelines add a recovery step, usually called healing, before the model goes into production. Recent open-weight releases such as rbtv77 gpt-oss, NVIDIA's Nemotron family, and our own Hypernova 60B all rely on some version of this compress-then-heal approach.\n\nThis is a really fascinating take on model compression 🤯🔥. The idea that a 4-bit model can actually outperform the full-precision version turns the usual assumption about quantization completely upside down.... Instead of simply accepting the accuracy loss that comes with compression, QAH seems to show how targeted recovery can bring back—and even improve—important capabilities like reasoning, math, and coding. 🧠⚡\n\nWhat really caught my attention is the result with the compressed GPT-OSS 120B → 60B model.... Getting a smaller MXFP4 model to beat its own bfloat16 checkpoint on 7 of 9 benchmarks is pretty impressive. 📊🚀 It makes the whole compress-then-heal pipeline feel much more practical for real-world deployment, especially when memory usage and inference costs matter.\n\nI also like the broader implication here.... Quantization doesn't necessarily have to mean sacrificing model quality if the recovery process is designed with the quantized model in mind. 💡🔧 A smaller model that costs less to operate while still delivering stronger benchmark performance could be a huge advantage for teams trying to deploy capable LLMs efficiently.\n\nDefinitely an interesting direction for efficient AI deployment.... Turning a heavily compressed 4-bit model into something that can outperform its larger full-precision counterpart is the kind of result that makes you rethink what \"compression\" actually means. 👏🔥 Really curious to see how far Quantization-Aware Healing can go across other architectures and workloads.... 🚀🧠","html":" \n Making a large language model smaller almost always comes with a cost. The now-standard recipe for efficient deployment is to compress the architecture first, cutting the parameter count by removing layers, heads, or neurons, and then quantize the remaining weights down to 4 bits to shrink memory and compute further. Both steps save a lot, but together they systematically degrade the capabilities people actually care about: reasoning, mathematical problem-solving, and code generation. Because of this, serious deployment pipelines add a recovery step, usually called healing, before the model goes into production. Recent open-weight releases such as rbtv77 gpt-oss, NVIDIA's Nemotron family, and our own Hypernova 60B all rely on some version of this compress-then-heal approach. \n \n This is a really fascinating take on model compression 🤯🔥. The idea that a 4-bit model can actually outperform the full-precision version turns the usual assumption about quantization completely upside down.... Instead of simply accepting the accuracy loss that comes with compression, QAH seems to show how targeted recovery can bring back—and even improve—important capabilities like reasoning, math, and coding. 🧠⚡ \n What really caught my attention is the result with the compressed GPT-OSS 120B → 60B model.... Getting a smaller MXFP4 model to beat its own bfloat16 checkpoint on 7 of 9 benchmarks is pretty impressive. 📊🚀 It makes the whole compress-then-heal pipeline feel much more practical for real-world deployment, especially when memory usage and inference costs matter. \n I also like the broader implication here.... Quantization doesn't necessarily have to mean sacrificing model quality if the recovery process is designed with the quantized model in mind. 💡🔧 A smaller model that costs less to operate while still delivering stronger benchmark performance could be a huge advantage for teams trying to deploy capable LLMs efficiently. \n Definitely an interesting direction for efficient AI deployment.... Turning a heavily compressed 4-bit model into something that can outperform its larger full-precision counterpart is the kind of result that makes you rethink what \"compression\" actually means. 👏🔥 Really curious to see how far Quantization-Aware Healing can go across other architectures and workloads.... 🚀🧠 \n","updatedAt":"2026-08-25T12:20:27.487Z","author":{"_id":"6a3cd0fb3af2b0b0921e329a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1Yp81w2J8OBsuzrHcQoZu.jpeg","fullname":"Elton Williams","name":"eltonwilliams","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9268456697463989},"editors":["eltonwilliams"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1Yp81w2J8OBsuzrHcQoZu.jpeg"],"reactions":[],"isReport":false}},{"id":"6a92b9c57d68a90b4fa5e6da","author":{"_id":"69d408afb011ec4ae15a4bfe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png","fullname":"Saga","name":"peculiar-ragdoll","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":976,"isUserFollowing":false},"createdAt":"2026-08-29T10:51:49.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Hi there! From how I'm reading this, I think you might have missed a method error in your experiment that invalidates your headline.\n\nThe headline comparison looks like it has no control. The bf16 60B got one round of distillation and the MXFP4 60B got that round plus a second round from a better teacher. The obvious missing comparison is to give the bf16 60B the identical second distillation pass in bf16, same data, same steps, same teacher. Without that comparison you cannot separate \"QAH is a good recipe\" from \"we trained one checkpoint longer against a teacher than the other.\"\n\nResults table labels the teacher \"120B teacher (MXFP4)\", which is right since gpt-oss-120b ships with MXFP4 MoE weights natively. But the approach section says the teacher is full-size and full-precision, and the title claims the student beats its full-precision original. Those contradict.\n\nUnless I'm wrong, I'd retract this and consdier republishing when you can show the significant difference on the comparable arms without extending one further than the other with more compute.","html":" Hi there! From how I'm reading this, I think you might have missed a method error in your experiment that invalidates your headline. \n The headline comparison looks like it has no control. The bf16 60B got one round of distillation and the MXFP4 60B got that round plus a second round from a better teacher. The obvious missing comparison is to give the bf16 60B the identical second distillation pass in bf16, same data, same steps, same teacher. Without that comparison you cannot separate \"QAH is a good recipe\" from \"we trained one checkpoint longer against a teacher than the other.\" \n Results table labels the teacher \"120B teacher (MXFP4)\", which is right since gpt-oss-120b ships with MXFP4 MoE weights natively. But the approach section says the teacher is full-size and full-precision, and the title claims the student beats its full-precision original. Those contradict. \n Unless I'm wrong, I'd retract this and consdier republishing when you can show the significant difference on the comparable arms without extending one further than the other with more compute. \n","updatedAt":"2026-08-29T10:51:49.474Z","author":{"_id":"69d408afb011ec4ae15a4bfe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png","fullname":"Saga","name":"peculiar-ragdoll","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":976,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9457246661186218},"editors":["peculiar-ragdoll"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png"],"reactions":[],"isReport":false},"replies":[{"id":"6a9469b77d68a90b4fa5e7a5","author":{"_id":"668e37fd9c9aa124a3c867e8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/668e37fd9c9aa124a3c867e8/4ivwrPQnZGMDF6ovxnAdA.jpeg","fullname":"Antonio Tiene","name":"AntonioTN","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":14,"isUserFollowing":false,"primaryOrg":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6835694d56d5a69517655698/C3QUBARPGF4kbzJzZaT29.png","fullname":"Multiverse Computing","name":"MultiverseComputingCAI","type":"org","isHf":false,"plan":"team"}},"createdAt":"2026-08-30T17:34:47.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"You're right that the headline table compares checkpoints with unequal training. The 60B bf16 checkpoint got one distillation pass (the recovery); the 60B MXFP4 checkpoint got that pass plus a second one against the 120B teacher. We describe that second pass in the approach section, but we should have been explicit that the table therefore doesn't, on its own, separate \"distilling under quantization\" from \"distilling for longer.\" We'll add that caveat to the text.\n\nA bit of context on what the paper is: it's an applied case study, not a theoretical claim. The question we set out to answer is the one a deployment team faces: Given a compressed model that has to ship in 4 bits, what's the best way to run the quantization step? The result we stand behind is narrow and reproducible: the shipped 4-bit checkpoint scores above the bf16 checkpoint it was quantized from, rather than below it, which is the usual outcome of direct post-training quantization. What we shouldn't have leaned into is the stronger causal reading, that quantization itself is \"free training\", and you're right that the current table doesn't isolate that.\n\nThe comparison that would settle it is the one you describe: the bf16 60B given the identical second pass. same data, steps, teacher and, separately, that continued bf16 checkpoint post-training-quantized to MXFP4. The second is really the decision anyone deploying faces: heal under quantization, or quantize afterwards. If we extend this for a revised version, those are the arms we'd add; either way we're flagging them as a limitation of the current version rather than claiming the table resolves the question.\n\nWe don't think retraction is the right call. The reported numbers are real and reproducible, and the practical result stands. But we agree the causal framing needs that control, and we'd rather correct it in the open than argue it. Appreciate the push.","html":" You're right that the headline table compares checkpoints with unequal training. The 60B bf16 checkpoint got one distillation pass (the recovery); the 60B MXFP4 checkpoint got that pass plus a second one against the 120B teacher. We describe that second pass in the approach section, but we should have been explicit that the table therefore doesn't, on its own, separate \"distilling under quantization\" from \"distilling for longer.\" We'll add that caveat to the text. \n A bit of context on what the paper is: it's an applied case study, not a theoretical claim. The question we set out to answer is the one a deployment team faces: Given a compressed model that has to ship in 4 bits, what's the best way to run the quantization step? The result we stand behind is narrow and reproducible: the shipped 4-bit checkpoint scores above the bf16 checkpoint it was quantized from, rather than below it, which is the usual outcome of direct post-training quantization. What we shouldn't have leaned into is the stronger causal reading, that quantization itself is \"free training\", and you're right that the current table doesn't isolate that. \n The comparison that would settle it is the one you describe: the bf16 60B given the identical second pass. same data, steps, teacher and, separately, that continued bf16 checkpoint post-training-quantized to MXFP4. The second is really the decision anyone deploying faces: heal under quantization, or quantize afterwards. If we extend this for a revised version, those are the arms we'd add; either way we're flagging them as a limitation of the current version rather than claiming the table resolves the question. \n We don't think retraction is the right call. The reported numbers are real and reproducible, and the practical result stands. But we agree the causal framing needs that control, and we'd rather correct it in the open than argue it. Appreciate the push. \n","updatedAt":"2026-08-30T17:34:47.320Z","author":{"_id":"668e37fd9c9aa124a3c867e8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/668e37fd9c9aa124a3c867e8/4ivwrPQnZGMDF6ovxnAdA.jpeg","fullname":"Antonio Tiene","name":"AntonioTN","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":14,"isUserFollowing":false,"primaryOrg":{"avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6835694d56d5a69517655698/C3QUBARPGF4kbzJzZaT29.png","fullname":"Multiverse Computing","name":"MultiverseComputingCAI","type":"org","isHf":false,"plan":"team"}}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.961466908454895},"editors":["AntonioTN"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/668e37fd9c9aa124a3c867e8/4ivwrPQnZGMDF6ovxnAdA.jpeg"],"reactions":[{"reaction":"🔥","users":["Jonrods"],"count":1}],"isReport":false,"parentCommentId":"6a92b9c57d68a90b4fa5e6da"}},{"id":"6a946fbe103ce7f24f1febf7","author":{"_id":"69d408afb011ec4ae15a4bfe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png","fullname":"Saga","name":"peculiar-ragdoll","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":976,"isUserFollowing":false},"createdAt":"2026-08-30T18:00:30.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Right! The observation that the recovered checkpoint is a worse teacher, and that you should skip it and go to the original, is correct and useful. But fundamentally, your QAH is QAT: LLM-QAT was doing data-free distillation from the full-precision model years ago.","html":" Right! The observation that the recovered checkpoint is a worse teacher, and that you should skip it and go to the original, is correct and useful. But fundamentally, your QAH is QAT: LLM-QAT was doing data-free distillation from the full-precision model years ago. \n","updatedAt":"2026-08-30T18:00:30.020Z","author":{"_id":"69d408afb011ec4ae15a4bfe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png","fullname":"Saga","name":"peculiar-ragdoll","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":976,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9607752561569214},"editors":["peculiar-ragdoll"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png"],"reactions":[],"isReport":false,"parentCommentId":"6a92b9c57d68a90b4fa5e6da"}},{"id":"6a96c5063b88398f333a44ef","author":{"_id":"614a1ebb8f82f1df64d55126","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1632247447995-noauth.jpeg","fullname":"Iker García-Ferrero","name":"Iker","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":46,"isUserFollowing":false},"createdAt":"2026-09-01T12:28:54.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"The LLM-QAT paper uses cross-entropy as the loss function to train the model. QAH uses KL divergence (a custom more efficient implementation described in this other blog post: https://huggingface.co/blog/MultiverseComputingCAI/efficient-knowledge-distillation). One of our main contributions is demonstrating that KL divergence is a better loss function than cross-entropy loss for Quantization Aware Training. \n\nWe aim to provide a recipe of what works to efficiently compress and perform quantization of a pre-trained LLM. This blog post, focuses on the second part. ","html":" The LLM-QAT paper uses cross-entropy as the loss function to train the model. QAH uses KL divergence (a custom more efficient implementation described in this other blog post: https://huggingface.co/blog/MultiverseComputingCAI/efficient-knowledge-distillation ). One of our main contributions is demonstrating that KL divergence is a better loss function than cross-entropy loss for Quantization Aware Training. \n We aim to provide a recipe of what works to efficiently compress and perform quantization of a pre-trained LLM. This blog post, focuses on the second part. \n","updatedAt":"2026-09-01T12:31:00.892Z","author":{"_id":"614a1ebb8f82f1df64d55126","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1632247447995-noauth.jpeg","fullname":"Iker García-Ferrero","name":"Iker","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":46,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.8928592801094055},"editors":["Iker"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1632247447995-noauth.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"6a92b9c57d68a90b4fa5e6da"}},{"id":"6a96d829e66a691275080285","author":{"_id":"69d408afb011ec4ae15a4bfe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png","fullname":"Saga","name":"peculiar-ragdoll","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":976,"isUserFollowing":false},"createdAt":"2026-09-01T13:50:33.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Alright! Thanks for the clarification. As you state yourself in the blog, the observation that KL divergence is a good loss function for QAT already belongs to QAD, and we agree that data-free distillation from the full-precision model belongs to LLM-CAT. LLM-QAT, QAD and QAH are all QAT, where the loss function and teacher are the moving variables, and where QAH is (as I understand you) a combination of LLM-CAT and CAD. Again, we agree that it's correct to use KL div against the original model as the most optimal QAT-approach for people with the hardware to measure KL against the original model. But what you are claiming as your contribution, which is the advantage of combining QAD and LLM-CAT, has not been proven until you add the comparison arm that gets the same amount of compute with the hypothesized less optimal parametrization. Maybe the difference is within noise, which would make it a waste of money to rent a GPU to fit the larger original model, right? So what it looks like you're left with is a result that is neither complete, novel, or useful to me the reader, because its incompleteness makes it insufficient for proving the value of the combination of two existing QAT approaches which is your grain of novelty. The result would be useful if you completed the comparison arm so we could see the gains or lack thereof, though :) Even a hypothesis rejection would be a very valuable contribution. Anyways, I want to say that I'm not trying to be combative here as much as I'm trying to clarify the core of my feedback, because I really want to see your results either way it lands! So thanks for your work and for engaging with the discussion, and I hope it lands as good faith.","html":" Alright! Thanks for the clarification. As you state yourself in the blog, the observation that KL divergence is a good loss function for QAT already belongs to QAD, and we agree that data-free distillation from the full-precision model belongs to LLM-CAT. LLM-QAT, QAD and QAH are all QAT, where the loss function and teacher are the moving variables, and where QAH is (as I understand you) a combination of LLM-CAT and CAD. Again, we agree that it's correct to use KL div against the original model as the most optimal QAT-approach for people with the hardware to measure KL against the original model. But what you are claiming as your contribution, which is the advantage of combining QAD and LLM-CAT, has not been proven until you add the comparison arm that gets the same amount of compute with the hypothesized less optimal parametrization. Maybe the difference is within noise, which would make it a waste of money to rent a GPU to fit the larger original model, right? So what it looks like you're left with is a result that is neither complete, novel, or useful to me the reader, because its incompleteness makes it insufficient for proving the value of the combination of two existing QAT approaches which is your grain of novelty. The result would be useful if you completed the comparison arm so we could see the gains or lack thereof, though :) Even a hypothesis rejection would be a very valuable contribution. Anyways, I want to say that I'm not trying to be combative here as much as I'm trying to clarify the core of my feedback, because I really want to see your results either way it lands! So thanks for your work and for engaging with the discussion, and I hope it lands as good faith. \n","updatedAt":"2026-09-01T13:50:33.073Z","author":{"_id":"69d408afb011ec4ae15a4bfe","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png","fullname":"Saga","name":"peculiar-ragdoll","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":976,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9668367505073547},"editors":["peculiar-ragdoll"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/69d408afb011ec4ae15a4bfe/IfD8yxNRBjZqmU6RNUCkq.png"],"reactions":[{"reaction":"➕","users":["gbuzhf"],"count":1}],"isReport":false,"parentCommentId":"6a92b9c57d68a90b4fa5e6da"}},{"id":"6a96e02bd04cda54d1296c9e","author":{"_id":"614a1ebb8f82f1df64d55126","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1632247447995-noauth.jpeg","fullname":"Iker García-Ferrero","name":"Iker","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":46,"isUserFollowing":false},"createdAt":"2026-09-01T14:24:43.000Z","type":"comment","data":{"edited":true,"hidden":false,"latest":{"raw":"The problem for this comparison is that QAT always requires doing some extra training iterations. So even if we train the bf16 checkpoint for longer, the QAT checkpoint one will again need extra compute. You can always do QAT on an earlier checkpoint, to try to simulate the same amount of compute. However, this is not a real-world scenario, because we cannot chose between deploying a bf16 model trained for longer or a 4-bit model trained less time, as the bf16 doesn't fit in the target deployment hardware. \n\nIn any case, take into account, that while the bf16 compressed model was trained in millions of samples, QAT only requires an amount of training samples in the thousands. So it is not like we are doing a huge training for QAT. The data used for QAT is also a subset of the data used for compression, so no new knowledge is intruded in this phase. \n\nAs I understand, your main criticism is around the claim that the 60B 4 bit model outperforms the 60B BF16 checkpoint, and that this wouldn't happen if we had trained the BF16 checkpoint for longer. This might be true, also, GPT-OSS is an special case, as the original 120B model was already a 4-bit model. Theoretically, if you would plateau the 60B model until it cannot learn anything because all the parameters of the model are exhusted, it would never be posible to do a lossless quantization. So I think that you are right here, in our experimental setup with our data and hardware constrains we got a 4-bit model that outperformed the bf16 one, but this is not always expected. But what we wanted to share is not \"a quantization method that makes a 4-bit model better than a bf16 one\", but the recipe that worked for us to minimize the loss of precision during a compression and quantization process. In this particular case, this loss was so minimal, that in fact, we outperformed the bf16 model, but this is not what you should expect from every quantization. But you should totally expect that:\n\n- If you use the original model as teacher, instead of the compressed one, you will get better performance. \n- If you use KL divergence instead of cross-entropy loss, you will get better performance. \n\nThere are some ablations left on the table that we would have loved to perform, for me the most important one, is to perform QAT with the 60B bf16 model as teacher instead of the 120B one. But the fact that the 60B 4-bit model outperforms the 60B bf16 model, is a very strong indication, that using the 60B bf16 model as teacher would perform worse, as the performance ceiling would be lower. I am the first one that would have loved to run more experiments, unfortunately, the GPUs were needed elsewhere and we had to move on. We wrote the report with the data we had, and we hope that some people will find it interesting.","html":" The problem for this comparison is that QAT always requires doing some extra training iterations. So even if we train the bf16 checkpoint for longer, the QAT checkpoint one will again need extra compute. You can always do QAT on an earlier checkpoint, to try to simulate the same amount of compute. However, this is not a real-world scenario, because we cannot chose between deploying a bf16 model trained for longer or a 4-bit model trained less time, as the bf16 doesn't fit in the target deployment hardware. \n In any case, take into account, that while the bf16 compressed model was trained in millions of samples, QAT only requires an amount of training samples in the thousands. So it is not like we are doing a huge training for QAT. The data used for QAT is also a subset of the data used for compression, so no new knowledge is intruded in this phase. \n As I understand, your main criticism is around the claim that the 60B 4 bit model outperforms the 60B BF16 checkpoint, and that this wouldn't happen if we had trained the BF16 checkpoint for longer. This might be true, also, GPT-OSS is an special case, as the original 120B model was already a 4-bit model. Theoretically, if you would plateau the 60B model until it cannot learn anything because all the parameters of the model are exhusted, it would never be posible to do a lossless quantization. So I think that you are right here, in our experimental setup with our data and hardware constrains we got a 4-bit model that outperformed the bf16 one, but this is not always expected. But what we wanted to share is not \"a quantization method that makes a 4-bit model better than a bf16 one\", but the recipe that worked for us to minimize the loss of precision during a compression and quantization process. In this particular case, this loss was so minimal, that in fact, we outperformed the bf16 model, but this is not what you should expect from every quantization. But you should totally expect that: \n \n If you use the original model as teacher, instead of the compressed one, you will get better performance. \n If you use KL divergence instead of cross-entropy loss, you will get better performance. \n \n There are some ablations left on the table that we would have loved to perform, for me the most important one, is to perform QAT with the 60B bf16 model as teacher instead of the 120B one. But the fact that the 60B 4-bit model outperforms the 60B bf16 model, is a very strong indication, that using the 60B bf16 model as teacher would perform worse, as the performance ceiling would be lower. I am the first one that would have loved to run more experiments, unfortunately, the GPUs were needed elsewhere and we had to move on. We wrote the report with the data we had, and we hope that some people will find it interesting. \n","updatedAt":"2026-09-01T14:26:46.798Z","author":{"_id":"614a1ebb8f82f1df64d55126","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1632247447995-noauth.jpeg","fullname":"Iker García-Ferrero","name":"Iker","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":46,"isUserFollowing":false}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.9725795388221741},"editors":["Iker"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1632247447995-noauth.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"6a92b9c57d68a90b4fa5e6da"}},{"id":"6a96e765b151968ec9a6a8a8","author":{"_id":"64ad4367b2f83dec961cf8a3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64ad4367b2f83dec961cf8a3/40tSr2BOT_aJzkcmXIUK4.jpeg","fullname":"gbhf","name":"gbuzhf","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":42,"isUserFollowing":false},"createdAt":"2026-09-01T14:55:33.000Z","type":"comment","data":{"edited":true,"hidden":true,"hiddenBy":"","hiddenReason":"Off-Topic","latest":{"raw":"This comment has been hidden","html":"This comment has been hidden","updatedAt":"2026-09-01T21:25:52.379Z","author":{"_id":"64ad4367b2f83dec961cf8a3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64ad4367b2f83dec961cf8a3/40tSr2BOT_aJzkcmXIUK4.jpeg","fullname":"gbhf","name":"gbuzhf","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":42,"isUserFollowing":false}},"numEdits":1,"editors":[],"editorAvatarUrls":[],"reactions":[],"parentCommentId":"6a92b9c57d68a90b4fa5e6da"}}]},{"id":"6a9ed0f244f827143a88d92b","author":{"_id":"6a79323836f65eeb0e290b24","avatarUrl":"/avatars/77fee78df81627ffa5d1e77b22a02545.svg","fullname":"Jozsef Hegedus","name":"jhegedus42","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-09-07T14:57:54.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Finally, now make the network complex, and use classical versions of 7 1 3 code to build a carnot cycle, details: jhegedus42/Szima on github, the codes will give the hamiltonian/legendre... if u want look at the 7x7x7 steano tower and E8 - more details : https://chatgpt.com/share/6a9ed0ab-0578-83ed-adb6-5cd5f1fac11e - u go on the right path as an engineer, now u need the theory to know where u r going","html":" Finally, now make the network complex, and use classical versions of 7 1 3 code to build a carnot cycle, details: jhegedus42/Szima on github, the codes will give the hamiltonian/legendre... if u want look at the 7x7x7 steano tower and E8 - more details : https://chatgpt.com/share/6a9ed0ab-0578-83ed-adb6-5cd5f1fac11e - u go on the right path as an engineer, now u need the theory to know where u r going \n","updatedAt":"2026-09-07T14:57:54.230Z","author":{"_id":"6a79323836f65eeb0e290b24","avatarUrl":"/avatars/77fee78df81627ffa5d1e77b22a02545.svg","fullname":"Jozsef Hegedus","name":"jhegedus42","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8241384625434875},"editors":["jhegedus42"],"editorAvatarUrls":["/avatars/77fee78df81627ffa5d1e77b22a02545.svg"],"reactions":[],"isReport":false}},{"id":"6a9f0dba2dd7f4a3f35c1f87","author":{"_id":"6a79323836f65eeb0e290b24","avatarUrl":"/avatars/77fee78df81627ffa5d1e77b22a02545.svg","fullname":"Jozsef Hegedus","name":"jhegedus42","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-09-07T19:17:14.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"\n\n","html":" \n","updatedAt":"2026-09-07T19:17:14.518Z","author":{"_id":"6a79323836f65eeb0e290b24","avatarUrl":"/avatars/77fee78df81627ffa5d1e77b22a02545.svg","fullname":"Jozsef Hegedus","name":"jhegedus42","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.452982097864151},"editors":["jhegedus42"],"editorAvatarUrls":["/avatars/77fee78df81627ffa5d1e77b22a02545.svg"],"reactions":[],"isReport":false}}],"status":"open","isReport":false,"pinned":false,"locked":false,"collection":"community_blogs"},"contextAuthors":["AntonioTN","Iker","ali-hashemi","bryskulov-mc"],"primaryEmailConfirmed":false,"discussionRole":0,"acceptLanguages":["en","zh"],"withThread":true,"cardDisplay":false,"repoDiscussionsLocked":false,"hideComments":true}"> eltonwilliams Aug 25 Making a large language model smaller almost always comes with a cost. The now-standard recipe for efficient deployment is to compress the architecture first, cutting the parameter count by removing layers, heads, or neurons, and then quantize the remaining weights down to 4 bits to shrink memory and compute further. Both steps save a lot, but together they systematically degrade the capabilities people actually care about: reasoning, mathematical problem-solving, and code generation. Because of this, serious deployment pipelines add a recovery step, usually called healing, before the model goes into production. Recent open-weight releases such as rbtv77 gpt-oss, NVIDIA's Nemotron family, and our own Hypernova 60B all rely on some version of this compress-then-heal approach.

This is a really fascinating take on model compression 🤯🔥. The idea that a 4-bit model can actually outperform the full-precision version turns the usual assumption about quantization completely upside down.... Instead of simply accepting the accuracy loss that comes with compression, QAH seems to show how targeted recovery can bring back—and even improve—important capabilities like reasoning, math, and coding. 🧠⚡

What really caught my attention is the result with the compressed GPT-OSS 120B → 60B model.... Getting a smaller MXFP4 model to beat its own bfloat16 checkpoint on 7 of 9 benchmarks is pretty impressive. 📊🚀 It makes the whole compress-then-heal pipeline feel much more practical for real-world deployment, especially when memory usage and inference costs matter.

I also like the broader implication here.... Quantization doesn't necessarily have to mean sacrificing model quality if the recovery process is designed with the quantized model in mind. 💡🔧 A smaller model that costs less to operate while still delivering stronger benchmark performance could be a huge advantage for teams trying to deploy capable LLMs efficiently.

Definitely an interesting direction for efficient AI deployment.... Turning a heavily compressed 4-bit model into something that can outperform its larger full-precision counterpart is the kind of result that makes you rethink what "compression" actually means. 👏🔥 Really curious to see how far Quantization-Aware Healing can go across other architectures and workloads.... 🚀🧠

Reply peculiar-ragdoll Aug 29 Hi there! From how I'm reading this, I think you might have missed a method error in your experiment that invalidates your headline.

The headline comparison looks like it has no control. The bf16 60B got one round of distillation and the MXFP4 60B got that round plus a second round from a better teacher. The obvious missing comparison is to give the bf16 60B the identical second distillation pass in bf16, same data, same steps, same teacher. Without that comparison you cannot separate "QAH is a good recipe" from "we trained one checkpoint longer against a teacher than the other."

Results table labels the teacher "120B teacher (MXFP4)", which is right since gpt-oss-120b ships with MXFP4 MoE weights natively. But the approach section says the teacher is full-size and full-precision, and the title claims the student beats its full-precision original. Those contradict.

Unless I'm wrong, I'd retract this and consdier republishing when you can show the significant difference on the comparable arms without extending one further than the other with more compute.

6 replies

AntonioTN Article author Aug 30 You're right that the headline table compares checkpoints with unequal training. The 60B bf16 checkpoint got one distillation pass (the recovery); the 60B MXFP4 checkpoint got that pass plus a second one against the 120B teacher. We describe that second pass in the approach section, but we should have been explicit that the table therefore doesn't, on its own, separate "distilling under quantization" from "distilling for longer." We'll add that caveat to the text.

A bit of context on what the paper is: it's an applied case study, not a theoretical claim. The question we set out to answer is the one a deployment team faces: Given a compressed model that has to ship in 4 bits, what's the best way to run the quantization step? The result we stand behind is narrow and reproducible: the shipped 4-bit checkpoint scores above the bf16 checkpoint it was quantized from, rather than below it, which is the usual outcome of direct post-training quantization. What we shouldn't have leaned into is the stronger causal reading, that quantization itself is "free training", and you're right that the current table doesn't isolate that.

The comparison that would settle it is the one you describe: the bf16 60B given the identical second pass. same data, steps, teacher and, separately, that continued bf16 checkpoint post-training-quantized to MXFP4. The second is really the decision anyone deploying faces: heal under quantization, or quantize afterwards. If we extend this for a revised version, those are the arms we'd add; either way we're flagging them as a limitation of the current version rather than claiming the table resolves the question.

We don't think retraction is the right call. The reported numbers are real and reproducible, and the practical result stands. But we agree the causal framing needs that control, and we'd rather correct it in the open than argue it. Appreciate the push.

Expand 5 replies jhegedus42 24 days ago Finally, now make the network complex, and use classical versions of 7 1 3 code to build a carnot cycle, details: jhegedus42/Szima on github, the codes will give the hamiltonian/legendre... if u want look at the 7x7x7 steano tower and E8 - more details : https://chatgpt.com/share/6a9ed0ab-0578-83ed-adb6-5cd5f1fac11e - u go on the right path as an engineer, now u need the theory to know where u r going

Reply jhegedus42 24 days ago

Reply Edit Preview Upload images, audio, and videos by dragging in the text input, pasting, or clicking here . Tap or paste here to upload images Comment · Sign up or log in to comment

+58

Models mentioned in this article 2

MultiverseComputingCAI/Hypernova-60B-2605

openai/gpt-oss-120b

Papers mentioned in this article 1