FireRedTeam 开源 IVC-Prune：面向 LVLM 的免训练视觉 token 剪枝
GitHub - FireRedTeam/IVC-Prune: IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning · GitHub
小红书（Xiaohongshu Inc.）与武汉大学团队开源 IVC-Prune，一种免训练、提示词感知的视觉 token 剪枝策略，通过 RoPE 的数学性质识别 IVC token 并两阶段筛选前景 token。该方法采用单次选择剪枝策略，在中间层选定 token 后应用于所有层，以最大化 KV-cache 压缩并保留原始位置 ID。目前已支持 Qwen-VL、LLaVA-v1.5、InternVL2.5 和 DeepSeek-VL2。
正文
当前显示原文，中文全文译文尚未就绪。
FireRedTeam; IVC-Prune
Repository navigation
NotificationsYou must be signed in to change notification settings; Fork1 (1); Star18 (18)You must be signed in to star a repository
Repository files navigation
README
Xiaohongshu Inc. Wuhan university
💡 We reveal a fundamental mechanism of how LVLMs process spatial information：
LVLMs implicitly establish visual coordinate systems through Rotary Position Embeddings (RoPE).
Identity matrix (real-axis references); 90° rotation matrix (imaginary-axis references)
This provides the first theoretical characterization of spatial reasoning mechanisms in LVLMs.
🚀 Method: IVC-Prune
A training-free, prompt-aware pruning strategy that preserves two crucial token types:
IVC Tokens: Identified by analyzing RoPE's mathematical properties (cosine/sine components across dimensions); Foreground Tokens: Selected via a robust two-stage process:
Stage 1: Semantic seed identification using value-vector similarity (avoiding positional bias); Stage 2: Contextual refinement to capture complete objects
Key Innovation: Single-selection pruning strategy—tokens are selected once at an intermediate layer and applied across all layers, maximizing KV-cache reduction while preserving original position IDs.
📝 TODO List:
Supported LVLMs
✅ Qwen-VL Support (transformers code); ✅ LLaVA-v1.5 Support (transformers code); ✅ InternVL2.5 Support (transformers code); ✅ DeepSeek-VL2 Support (transformers code)
Table of Contents
Installation; Dataset; Usage; Citation
Installation
Based on VLMEvalKit and transformers. We have supported grounding data testing code in VLMEvalKit.
Dataset
For RefCOCO grounding dataset, We provide MDETR format annotations on Hugging Face.
Setup Instructions
Step 1: Download Required Files
MDETR JSON annotations from Hugging Face; COCO train2014 images from COCO official site
Step 2: Generate TSV Files
Standard models:
Qwen2.5-VL (resized coordinates to multiples of 28):
Step 3: Configure Paths
Modify DATASET_URL in vlmeval/dataset/image_grounding.py:
Note: Always use --qwen25 flag when evaluating Qwen2.5-VL to ensure GT boxes match the model's patch size (28)."
Usage
To evaluate the model on grounding tasks, run the following script:
To evaluate general VQA tasks, run:
Note:
Before running the scripts, please make sure to modify the dataset paths in test_ivcp_qwen_grounding.sh (specifically lines 2-3) to point to your actual dataset location, e.g.: export LMUData="/your/dataset/path"
Some VQA Benchmark tests require the use of an API. Please configure your API according to the VLMEvalKit instructions. By default, we use GPT-4o.
Citation