Fireworks 教你用一杯咖啡的成本把 LLM 微调成嵌入模型

Join us for our inaugural conference, Forge 2026

PUBLISHED 7/29/2026

Table of Contents

Why Fine-Tune an LLM into an Embedding Model?; How Embedding Fine-Tuning Works: In-Batch Negatives and InfoNCE; Experimental Results; Experiment 1: Legal Citation Retrieval (LegalBench); Experiment 2: Clinical Trial Matching (TREC Clinical Trials); Experiment 3: EU Case-Law Citation Retrieval (LegalPincite); Code Example: Train on Your Private Data; What Else We Found; When Fine-tuning Didn't Move The Needle; Getting Started; Three Fireworks Training Modes: Flexibility vs. Efficiency; Interested in Fireworks Training?

Table of Contents

TL;DR

Retrieval-augmented generation (RAG), semantic search, reranking, recommendation, intention classification, deduplication, clustering — an enormous share of production AI boils down to one primitive: turning text into a vector so that related things land close together and unrelated things land far apart.

The quality of that vector space is the ceiling on every system built on top of it. A great reranker cannot rescue a retriever that never surfaced the right document; a RAG pipeline cannot cite a case it failed to retrieve. So the question becomes: how do you get the best possible embeddings for your domain?

In this post we show a practical, low cost recipe ( ", "positive": " "}. You don't need to supply negatives — in-batch negatives are generated automatically.

Then run the contrastive fine-tune through the Fireworks Training SDK. The trainer is provisioned for you; the loop streams batches, computes the bidirectional InfoNCE loss, and promotes the final checkpoint to a model you can deploy:

Once training finishes, your model serves through the standard embeddings endpoint — same API as the base model, now specialized to your domain:

Or call the /v1/embeddings endpoint directly (the response is OpenAI-compatible):

Index your corpus once with the same model, and retrieval is a nearest-neighbor lookup over these vectors.

What Else We Found

Beyond the two headline results, we ran a series of sweeps. The details are out of scope for this post, but the takeaways are worth stating:

•Fine-tuning's gain tracks the base model's headroom. Where the base is already strong, gains are small; where it struggles, they are large. We saw the same pattern on three more domains — code search (CoSQA), developer Q&A (StackOverflow-QA), and e-commerce product search (Amazon ESCI): a clear lift on ESCI, a small one on the already-near-ceiling StackOverflow-QA, and essentially flat on CoSQA; •~150 steps is the sweet spot. Training to 300 steps gave no improvement and sometimes slightly regressed top-rank metrics — a short adaptation specializes the geometry without overfitting; •LoRA nearly matches full fine-tuning. A rank-32 LoRA adapter landed within ~1 point of full-parameter fine-tuning on every metric, at a fraction of the cost — a great default for a cheap, swappable domain adapter; •Bigger batches didn't help here. Pushing in-batch negatives from 63 (B=64) to 255 (B=256) gave no gain on these tasks and ran far slower — 64 was already plenty; •Context length can matter more than fine-tuning. On a full-judgment (case-level) variant, a model fine-tuned at a 512-token window barely beat the base under long-context evaluation; training at an 8k–16k window instead nearly doubled nDCG and MRR. When documents exceed the encoder's window, truncation — not model quality — is the bottleneck, so train and evaluate at a length that fits your data

Not every dataset improved. The clearest non-results came from tasks where the base model was already strong, or where "relevance" is close to plain semantic/lexical similarity:

natural-language query → code snippet

0.967 (already near-ceiling)

These non-results fall into two scenarios:

The task isn't actually domain-specialized. When "relevance" reduces to general semantic or lexical similarity, the base model already handles it and there is no specialized signal to add. CoSQA — matching a natural-language query to the code snippet that answers it — is close to generic semantic search; StackOverflow-QA answer posts are lexically almost identical to their questions (base nDCG@10 0.94, near ceiling). Fine-tuning has nothing to learn; The benchmark is popular and already saturated by foundation models. FiQA (finance) and NFCorpus (medical) look domain-specific, but they are widely-used public IR benchmarks whose documents and relevance patterns were almost certainly in the base model's pretraining data. The model has effectively already seen the task, so fine-tuning on a small slice of it adds nothing (FiQA even dipped slightly)

Together these refine the hypothesis: fine-tuning delivers large gains only when the task carries a specialized relevance signal the base model has not already internalized — not merely domain-specific vocabulary, and not on public benchmarks that foundation models have already absorbed. The big wins — LegalBench citation retrieval (+36% nDCG@10) and full-judgment case matching (~2×) — came precisely where "relevant" means something the domain defines (which precedent governs the question, which trial a patient qualifies for) and that general pretraining never taught.

Getting Started

If your product relies on search, RAG, reranking, or recommendation over a specialized corpus, the base embedding model is leaving quality on the table — and training from scratch would cost you generalization. Contrastive fine-tuning of an embedding LLM is the cheap middle path that captures your domain's relevance while keeping the model's broad competence.

•Collect a few thousand (query, positive) pairs from your domain; •Fine-tune Qwen3-Embedding-8B with the recipe above (~150 steps, minutes of trainer time); •Evaluate on held-out queries with nDCG / Recall / MRR before and after; •Deploy through the same embeddings API and re-index your corpus

Three Fireworks Training Modes: Flexibility vs. Efficiency

Now available in preview, Fireworks exposes this contrastive fine-tuning loop in three modes that trade off flexibility (how freely you can define the loss) against training efficiency (how much data has to move between the client driving the loop and the trainer holding the model). They optimize the same family of objective; they differ in where the loss is computed and what crosses the client↔trainer boundary.

embedding — most flexible, least efficient. The trainer runs the forward pass and returns the raw embedding vectors, and you compute the loss yourself, client-side, with whatever function you like — InfoNCE, triplet, custom margins, multi-positive schemes, anything. This is the most general option: you are not limited to any built-in objective. The cost is bandwidth. The full embedding matrix (batch × hidden dim — e.g. 4096-d) is sent from the trainer to the client on the forward pass, and the corresponding gradients are sent back on the backward pass. For large batches and the wide hidden states of an 8B model, moving those tensors back and forth dominates step time.

cos_similarity_matrix — a middle ground. The trainer computes the pairwise cosine-similarity matrix and returns its rows; you still define the loss client-side, but over the much smaller N×N similarity matrix instead of the N×D embeddings. When the batch size N is below the embedding dimension D (the usual case), this transfers far less data than embedding while still letting you customize any loss expressible over similarities — InfoNCE variants, temperature schedules, custom masking. The trade-off: the full matrix has to be assembled on one device, so this mode requires a single-data-parallel (DP=1) trainer.

contrastive_loss — least flexible, most efficient. The bidirectional InfoNCE loss is computed entirely on the trainer in a single round trip: no embeddings or gradients cross the boundary — the batch goes in, a scalar loss comes back. This is the fastest option and the right default when InfoNCE is what you want, which for retrieval it almost always is. You give up the ability to plug in an arbitrary loss in exchange for minimal communication overhead.

Mode

Loss computed

Crosses the client↔trainer boundary

embeddings + gradients (N×D)

Rule of thumb: start with contrastive_loss for standard retrieval fine-tuning; reach for cos_similarity_matrix when you want a custom similarity-based loss without paying the full embedding-transfer cost; use embedding when you need complete freedom over the objective. (The experiments above use the embedding mode.)

Interested in Fireworks Training?

Tell us about your training needs so our forward deployed team can best help you on your journey to specialized intelligence.

Get in touch