用 TRL 对 LFM2.5-350M 做 GRPO 微调，提升结构化输出能力

Prerequisites

This guide has two halves that run in different places:

Fine-tuning runs on a GPU. The accompanying notebook is sized for a free-tier Colab or Kaggle GPU.

Evaluation can run locally on a MacBook (here, a MacBook Pro with an Apple M5 Max and 36 GB of unified memory) through llama.cpp , which exposes an OpenAI-compatible server that the IFStruct evaluator talks to.

We will need uv for the Python tooling and llama.cpp for serving. Following the Liquid AI llama.cpp deployment docs , install llama.cpp with Homebrew and verify that llama-server is available:

IFStruct Evaluation on LFM2.5-350M (Base model)

Before we begin, let's evaluate LFM2.5-350M on the IFStruct benchmark and see whether we can reproduce the reported score of 21.1% .

IFStruct is a benchmark for testing the validity of LLM outputs and schema adherence. The benchmark is open-source in Liquid4All/ifstruct , with the public benchmark dataset available on Hugging Face at LiquidAI/ifstruct-v1.0 .

For the eval comparison, we serve the model locally on the MacBook with llama.cpp . We will use the BF16 GGUF ( LiquidAI/LFM2.5-350M-GGUF ).

Then we start the base-model server with the following command:

--alias : model name IFStruct sends to the OpenAI-compatible endpoint

-ngl 99 : asks llama.cpp to offload all layers to the GPU when available

-np 4 : serves four requests in parallel

-c 32768 : size of the prompt context

Once the server is running, we can run the full benchmark with 2000 samples:

The IFStruct release blog reports 21.1% for LFM2.5-350M . Our local llama.cpp/BF16 setup measures 22.6%, close to the 21.1% reported in the IFStruct blog. We use this local result as the baseline for the same serving stack comparison.

GRPO Fine-tuning with TRL on Structured Outputs

The full, runnable pipeline lives in the accompanying notebook . We will cover only the relevant pieces in this section.

Training data

We use nvidia/Nemotron-RL-instruction_following-structured_outputs , which pairs each prompt with a target JSON Schema and an expected field count. We use about 500 samples for training.

Because the Nemotron data distribution differs from the IFStruct evaluation, we augment the prompts to close two gaps between them:

40% get a "return the output inside a fenced code block" instruction appended, so the model learns to follow the format instruction rather than always emitting raw JSON.

A disjoint 20% are converted into top-level-array tasks (the schema is wrapped in an array with a required item count), which trains bare-list output and item-count compliance.

Model and LoRA

We load LiquidAI/LFM2.5-350M and attach a LoRA adapter. Because LFM2.5 uses a hybrid attention/convolution architecture, we target the LFM-specific module names:

This trains ~6M parameters, about 1.66% of the model.

Reward functions

Then we define three reward functions, each on a [0, 1] scale, which score every completion on whether the extracted structure is correct:

json_format_reward : Is the output parseable, and in the requested form? Full credit ( 1.0 ) for the requested form (fenced vs. raw), 0.2 for the wrong-but-parseable form, 0.0 for unparseable output.

field_count_reward : Does the object have the expected number of top-level fields? An exact match earns 1.0 , and the score decays linearly with the miss.

schema_validation_reward : Does the output validate against the row's JSON Schema? It counts every constraint violation and gates partial credit on required-key coverage.

We combine the three as a weighted sum with reward_weights=[1.0, 0.5, 2.0] .

Training

We train for 100 steps with 8 generations per prompt group, sized for a free-tier 16 GB GPU:

As you can see in the notebook, over the run, all three reward components climb, the KL from the reference model lifts off zero after warmup, and the truncated-completion fraction stays near zero.

Merging and saving the model

Finally, we merge the LoRA adapter back into the base weights and save it as a single self-contained checkpoint, ready to convert to GGUF for serving:

IFStruct Evaluation on GRPO Tuned LFM2.5-350M

After GRPO fine-tuning, we rerun the IFStruct evaluation. For this, we need to convert the merged model checkpoint into a BF16 GGUF. The converter script ships with the llama.cpp source, so we clone the repo once and install the converter's gguf package.

Then we serve the merged model with the following command:

Then, we will run the full IFStruct evaluation again with the fine-tuned model:

Comparing the two runs on the identical serving stack:

The gains land exactly where the training aimed: the JSON pass rate rises by nearly 14 points (18.0% → 31.9%), while YAML stays mostly the same. While this is still below the Qwen3.5-2B score of 33.15% , it shows that even light task-specific fine-tuning can bring a small model close to a larger one.

Conclusion

A short GRPO run with about 500 samples and 100 steps can lift a small 350M parameter model from 22.6% to 29.7% on IFStruct. The takeaway is that a cheap, task-specific reward signal can make a small model substantially more reliable about form , closing much of the gap to models several times its size.

To reproduce or extend this work, see the original IFStruct v1.0 blog post , the Liquid4All/ifstruct benchmark repo, and the LiquidAI/ifstruct-v1.0 dataset.

Models mentioned in this article 2

LiquidAI/LFM2.5-350M

LiquidAI/LFM2.5-350M-GGUF

Datasets mentioned in this article 2

LiquidAI/ifstruct-v1.0

nvidia/Nemotron-RL-instruction_following-structured_outputs

用 TRL 对 LFM2.5-350M 做 GRPO 微调，提升结构化输出能力

主题

标签

正文