Central China Normal University proposes CoReFuse-Med: purify first, then fuse, to address low-quality modalities dragging down multimodal MRI segmentation

Source: Multimodal Information Fusion

More modalities is not always better. When MRI is already spatially aligned but the sampling quality of modalities such as T1, T2, and FLAIR differs greatly, low-quality modalities may bring resampling artifacts generated in high-quality modalities into the fusion network. CoReFuse-Med addresses the problem from two aspects: shallow-level contamination and deep-level modality bias.

Institutions: Central China Normal University, Huazhong University of Science and Technology, Guangzhou University of Chinese Medicine and its First Affiliated Hospital, University of North Carolina at Chapel Hill

Paper link:

Code link:

FUSION FAILURE · FIG. 1 The paper observed the high-resolution T1 single-modality DSC on the IH EPVS dataset and found an unexpected result: the high-resolution T1 single-modality DSC was 0.703, but after fusing T1, T2, and FLAIR it instead dropped to about 0.695. Spatial alignment does not equal consistency of information quality. After resampling, low-resolution sequences have images the same size as T1, but may contain unreliable information. Three pieces of evidence: contamination and bias. The authors limited the problem to quality degradation caused by differing resolutions, and did not involve misalignment or complete absence of modalities. First they compared the F-CNR of shallow features, then jointly trained gradients and occlusion tests, and finally tracked noise propagation through skip connections: this can explain why "weak training contribution" may also be "strong inference interference." Evidence one: low resolution produces noise. The feature contrast-to-noise ratio F-CNR proposed in the paper: separability is measured by dividing the difference between the means of the lesion region and the background region by the standard deviation of the background region. The feature visualization on page three shows that the F-CNR value of T1 is 1.0283, that of T2 is 0.4301, and that of FLAIR is only 0.2980; T1 is about 3.4 times that of FLAIR. Even if they overlap geometrically, the reliability of shallow representations differs. Equation (1) | F-CNR: lesion and background features can be distinguished. Evidence two: optimization and inference are inconsistent. In the 3D U-Net for EPVS, T1 accounts for 77.4% of the gradient contribution, while FLAIR accounts for only 13.1%; however, occluding FLAIR reduces DSC by 70.2%, as large as the effect of occluding T1. Fewer gradient updates do not mean there is no effect at prediction time. The paper regards the contrast as the main evidence of optimization–inference inconsistency. Evidence three | Skip connections are contagious. Skip connections in U-shaped networks were originally used to send spatial details to the decoder; in anisotropic MRI, they also carry resampled structural artifacts into the new image. The network also tends to rely on the "cleaner," more easily optimized high-quality modality, thus giving rise to feature contamination and greedy modality selection.

Shallow features F-CNR of T1, T2, FLAIR, etc.

Table 1 Gradient contribution, DSC drop after modality occlusion. 02 First purify, then balance the contribution of each modality

COREFUSE-MED·FIG.3 CoReFuse-Med is not simply stacking another stronger cross-attention; instead it divides fusion into two parts: LACE first builds an efficient anisotropic encoding for each MRI sequence; IMSH is responsible for reliable transfer between the shallow structural basis and the residual; MBCF recalibrates modality contributions in a deep shared semantic space. It remains a three-stream, symmetric U-shaped segmentation network. 01 LACE+IMSH: shallow layers "decontaminate" first. LACE uses local DoubleConv and strip convolution branches along the three directions D, H, and W to preserve details and capture axial context respectively, then uses optional channel attention to fuse the two branches' features. IMSH uses parameter-free 3×3×3 average pooling to separate the structural basis, and replaces the low-frequency residual with a high-frequency residual: the basis part uses competitive Softmax weights to select reliable anatomical structures, the residual part consists of shared Sigmoid gating to filter resampling artifacts, and finally lightweight DSConv3D is used for integration. LACE dual-branch selective fusion: local boundaries and axial context are extracted separately, then synthesized with adaptive weights. Equation (2) LACE selective feature fusion. IMSH spatial scale decomposition: the smooth component carries the shared anatomical layout, and the residual contains valid details and resampling noise. Equation (3) | Average pooling constructs the structural basis and residual. Equation (4) | Competitive weighting then writes back shallow features. 02 MBCF | Correcting modality bias at deep layers. MBCF targets the L3-L4 deep skip features. First, the global descriptions of T1, T2, and FLAIR are concatenated and passed through a shared MLP to produce channel weights; then spatial gating is used to exclude unreliable responses caused by unstable positions; finally, a common query Q that does not belong to any single modality is obtained, and symmetric cross-attention is performed on K and V obtained by concatenating and downsampling the three modalities. The goal is not to force all modalities to speak with equal weight, but to give trustworthy information a reasonable contribution during fusion. Global channel calibration does not use Softmax to force the weights of the three modalities to sum to 1; instead, it produces conditioned channel weights between a shared MLP and Sigmoid; spatial gating and the shared query further constrain modality competition. Equation (5) is the global channel reliability calibration. Equation (6) shares a query that does not depend on a single modality. Equation (7) | K and V after three-modality concatenation and downsampling. Equation (8) symmetric cross-modal attention fusion. How to read the experimental results and boundaries? On EPVS, Ours(Med) achieves DSC=0.7326 and HD95=7.35, better than MedNeXt's 0.7085 and 8.17 in the comparison; on BraTS, DSC=0.8528 and HD95=4.43, somewhat better than MedNeXt's 0.8314 and 7.59. On WMH, DSC=0.80, HD95=6.22, IAVD=0.171, with only 2.665M parameters. The paper also examines the retention of 20% and 30% axial slices in BraTS, retraining each model separately, and achieves the best DSC/HD95 under the two different settings; however, this is not testing the same weights directly across degradation levels. Additional Gaussian noise is used as a supplement for validation. Ablation experiment: on EPVS, Baseline DSC is 0.7146; after combining LACE+IMSH+MBCF, Ours(Base) is 0.7322, indicating that the three modules complement each other. In severely degraded regions on EPVS, the model mainly prevents contamination from damaging T1, rather than "restoring out of thin air" information already lost in T2/FLAIR. Research boundary: the main problem is resolution degradation after spatial alignment, not complete absence of all modalities, nor does it solve all strong-noise scenarios. Future directions are dynamic modality missing and more acquisition artifacts.