FireRedTeam 开源 ReMatch：CVPR 2026 多模态检索方案与 ReMatch-3B 权重

FireRedTeam; ReMatch

Repository navigation

NotificationsYou must be signed in to change notification settings; Fork0 (0); Star9 (9)You must be signed in to star a repository

Official implementation of "ReMatch: Boosting Representation through Matching for Multimodal Retrieval" (CVPR 2026)

Repository files navigation

README; Apache-2.0 license

This repository contains the official implementation of ReMatch, accepted to CVPR 2026.

ReMatch turns a multimodal large language model into a stronger multimodal retriever by adding a chat-style generative matching objective during training. The same MLLM learns to judge query-document relevance from both raw multimodal inputs and projected embeddings, complementing standard contrastive learning with instance-wise supervision on hard negatives. ReMatch also augments each input with multiple learnable representation tokens and fuses them into an efficient single-vector embedding for retrieval.

👥 Authors

University of Glasgow, Xiaohongshu Inc., Huazhong University of Science and Technology

* Equal contribution. # Project leader.

🔍 Method

ReMatch is built around two core ideas:

Query-Document Matching: an additional autoregressive matching stage that predicts relevance from the query, document, and their projected embeddings; Learnable Multi-Token Embeddings: multiple learnable tokens capture fine-grained contextual signals; an orthogonality regularizer encourages complementary representations, and the fused output remains a standard dense embedding

2026-05: ReMatch code, the ReMatch-3B checkpoint, and evaluation scripts are released; 2026-02: ReMatch is accepted to CVPR 2026; 2025-11: The ReMatch technical report is available on arXiv

🛠️ Installation

flash-attn can be sensitive to CUDA, PyTorch, and compiler versions. If installation fails, install the wheel matching your environment from the official FlashAttention release instructions, then rerun the remaining dependencies.

🤗 Checkpoints

We release ReMatch-3B, a Qwen2.5-VL-3B based checkpoint trained with the ReMatch recipe:

FireRedTeam/ReMatch-3B

For local checkpoints, pass the base model through --model_name and the adapter/full checkpoint through --checkpoint_path when evaluating.

🚀 Training

The public ReMatch-3B training entry point is:

Before training, download the mmE5 hard-negative MMEB training data from Hugging Face:

intfloat/mmE5-MMEB-hardneg

In addition to mmE5, please follow the original VLM2Vec data preparation instructions to download the corresponding MMEB training and evaluation data used by the public configs in this repository.

Then edit experiments/public/rematch/train_image_mme5_hardneg.yaml and replace every DATASET_BASE_PATH with the directory that contains your mmE5/ folder. The expected layout is:

The default script trains a Qwen2.5-VL-3B based ReMatch model with LoRA, 16 learnable query tokens, residual average fusion, orthogonal regularization, and the matching objective enabled. You can override common paths without editing the script:

📊 Evaluation

Evaluation configs are under experiments/public/eval:

image.yaml

Please prepare the MMEB evaluation data following the original VLM2Vec instructions, then set DATA_BASEDIR to the directory containing the downloaded evaluation files.

Note: Evaluation scores may vary slightly across environments, as different PyTorch, CUDA, and flash-attn versions can introduce small numerical differences.

For checkpoints produced by this repository, we recommend using eval_all.py. It reads the experiment name and automatically matches the evaluation configuration used by ReMatch, including backbone type, target-side instruction prefix, chat template, learnable query tokens, and residual embedding fusion. For example, an experiment name containing Qwen2.5vl, TgtInstruction, Queries16, ResidualAvg, and ChatTemplate will be evaluated with the corresponding qwen2_5_vl, target instruction, 16 learnable tokens, average residual fusion, and chat-template settings.

Evaluate one experiment checkpoint:

If no arguments are provided, eval_all.py scans outputs/ / /, evaluates every checkpoint directory, and writes summaries to:

For the released ReMatch-3B checkpoint, use eval.py directly and pass the matching ReMatch configuration explicitly:

🙏 Acknowledgements

This codebase is built on top of VLM2Vec. We sincerely thank the VLM2Vec authors for releasing their training and evaluation infrastructure for massive multimodal embedding tasks.

We also thank the authors of Qwen-VL, MMEB, and mmE5 for their open models, benchmarks, and data resources.

📚 Citation

Official implementation of "ReMatch: Boosting Representation through Matching for Multimodal Retrieval" (CVPR 2026)