Tianjin University's Zhang Changqing introduces generalized multimodal fusion at CNCC 2026: from trustworthy evidence fusion to the Fusion Anything foundation model

The real challenge is enabling models to retain useful information, identify reliable evidence, and adapt to new tasks.

English text is being prepared.

English text is being prepared.

01

Beyond Simple Alignment:

How to Balance Modality Commonality and Individuality? The value of multimodal fusion lies in the ability of information from different sources to complement one another. A single modality can only present partial features of an object or event, whereas combining different modalities may help a model form a more comprehensive judgment. However, an increase in the number of modalities does not necessarily mean prediction accuracy will improve. Newly added information may contain noise and may also mislead the current task. Therefore, multimodal fusion must answer not only how to integrate information, but also how to retain useful information and suppress irrelevant interference. Zhang Changqing first introduced two approaches in multimodal representation: joint representation and coordinated representation. Joint representation attempts to integrate information from different modalities into a common representation; coordinated representation emphasizes the correlations among different modalities. Taking canonical correlation analysis (CCA) and its deep learning variants as an example, related methods map different modalities into a space with strong correlations in order to exploit the common information among them. The problem is that if shared information is overemphasized, modality-specific information may be difficult to fully preserve. For example, in medical diagnosis, pathogen testing and CT imaging can provide information from different perspectives. The former may help identify the infectious pathogen, while the latter can show the location and extent of lung lesions. The two types of information complement each other but cannot simply replace one another. Therefore, multimodal representation needs to consider both commonality and individuality, rather than merely pursuing consistency among different modalities. Zhang Changqing's team attempted to change the approach to representation learning: rather than only extracting common features from observed data, they also learn latent representations capable of accommodating multiple kinds of information by reconstructing observations of different modalities. Under this approach, the latent representation behind each sample can be treated as a variable to be learned, and the observed data of each modality can then be reconstructed through different mapping mechanisms. Compared with simply mapping different modalities into a common space, this approach attempts to start from the reconstruction objective and preserve the different information contained in the observed data. Zhang Changqing connected this idea with the stereoscopic structure recovery of neural radiance fields (NeRF). NeRF learns a scene representation from observations at different viewpoints and generates images for corresponding viewpoints accordingly. The two share a similar idea of explaining multiple observations through latent representations, but their specific tasks and technical mechanisms are not the same. This research direction is also related to the problem of missing modalities in medical and industrial scenarios. In hospitals, different patients may undergo different examinations: some patients have CT scans, while others do not; some patients have complete laboratory data, while others lack some test results. If an algorithm must first fill in all missing data before making predictions, it may face problems such as the difficulty of completing high-dimensional data and the unreliability of completion results. To address this problem, the team explored learning methods based on latent representations, hoping to use currently observable data for modeling without having to manually complete every missing case. The significance of such research lies in the fact that real-world multimodal data are not always complete and well-structured. How to effectively use existing information under conditions of missing modalities remains an important problem that multimodal learning needs to solve.

02

From Static Fusion to Dynamic Trustworthy Decision-Making:

How does a model determine whether information is reliable? Even if different modalities can all provide information, their value in a specific task may change with the environment and the sample. In his talk, Zhang Changqing used an everyday scenario to explain this dynamism: if a person is bitten by a mosquito outdoors and it is too dark to see the mosquito clearly, they may swat at the spot where their skin itches; if there is enough light, they can first observe and then act. The task has not changed, but the person adjusts the basis for their judgment according to the information available at the time. For machine learning models, the problem is similar. In scenarios such as autonomous driving, visible-light images and thermal imaging may provide complementary information, but different modalities are not equally effective in different environments. If a model always relies excessively on one modality, it may fail to make full use of other information. Some earlier studies attempted to adjust modality weights based on external indicators such as the lighting quality of an image. However, lighting conditions cannot fully represent an image's discriminative value for a specific task. Therefore, adjusting weights solely on the basis of preset quality indicators does not necessarily solve the problem of dynamic fusion. Another line of research introduced by Zhang Changqing is to bring uncertainty estimation into multimodal fusion. In the related work Trusted Multi-View Classification with Dynamic Evidential Fusion, the team proposed the trusted multi-view classification method TMC. This method uses variational Dirichlet distributions to characterize class probability distributions and, combined with Dempster–Shafer evidence theory, fuses the evidence provided by different views to estimate the uncertainty of predictions. Compared with approaches that output only a single classification result, this type of method also attempts to answer: How much evidential support does the current judgment receive? Is the information from different sources consistent? Is the available evidence sufficient to support a relatively reliable judgment? This idea is especially applicable to scenarios where information quality is unstable. The model does not have to treat every input as equally reliable; instead, it can adjust the fusion result according to the evidence provided by different sources. The talk also introduced the application of dynamic evidential fusion in crystal structure identification. When electron microscopy is used to obtain crystal structure information, the amount of information provided by different observation angles may differ considerably. If all observations are treated indiscriminately and the results are integrated through fixed weights or simple voting, effective evidence may be disturbed by a large number of low-value observations. Dynamic fusion, by contrast, attempts to update the judgment according to the evidence provided by each observation. When the corresponding conditions are met, the system can also decide whether to continue collecting data based on whether the uncertainty reaches a set threshold. The experimental results shown in the talk indicate that the classification accuracy of this crystal structure identification task increased from about 58%–59% to around 93.9%. The specific results should still be understood in light of the experimental setup of the corresponding paper, and it cannot be inferred from them that all multimodal tasks can achieve improvements of the same magnitude. What this type of research focuses on is not only how to put information from different modalities together, but also how to form judgments based on evidence quality and stop collecting further information under appropriate conditions.

03

Fusion Anything:

Exploring a General-Purpose Multimodal Prediction Foundation The first two parts discussed specific problems in multimodal fusion: how to preserve modality-specific information, and how to make dynamic, trustworthy decisions among different information sources. On this basis, Zhang Changqing further introduced the team's general-purpose multimodal prediction foundation—the Fusion Anything Model (FAM). The related paper Fusion Anything: A Generalized Multimodal Foundation Model poses a more general research question: Can a model be trained so that it is no longer limited to pre-specified modality combinations and a single task, but can instead use in-context examples to adapt to different multimodal prediction tasks? Traditional multimodal models are usually designed for specific data types and tasks. When facing a new application scenario, researchers may need to reselect algorithms, organize data, train models, and repeatedly tune parameters. This approach can optimize for a specific task, but it also increases the cost of transferring to new scenarios. FAM hopes to explore another path: learning transferable multimodal association patterns through pretraining, and, when facing a new task, completing predictions using given in-context examples without having to re-update model parameters for each task. The main difficulty facing this goal is how to obtain sufficiently diverse training data. Real-world multimodal data often struggle to cover all possible modality combinations and association patterns. To this end, the team used Structural Multimodal Causal Models (SMCMs) to generate synthetic multimodal prediction tasks, constructing data with different correlation patterns by varying latent information, shared information, modality-specific information, and their association structures. In terms of model architecture, FAM uses three types of attention mechanisms:

Intra-modality Attention: modeling the relationships among different features within the same modality.

Cross-modality Attention: modeling the relationships among different modalities within the same sample.

Sample-level Attention: leverages information among contextual examples to help the model predict new samples based on existing examples. These three types of attention mechanisms focus respectively on associations within a modality, between modalities, and across different samples, jointly supporting the model's handling of multimodal prediction tasks. Here, two concepts need to be distinguished: not requiring model parameter updates for a new task does not mean the model does not need task examples at all. FAM adopts in-context learning at the inference stage: researchers provide some labeled examples, and the model predicts new query samples based on the inputs and target values in these examples. The model does not need additional parameter fine-tuning for the task, but it still needs contextual examples to provide task information. In the evaluation reported in the paper, FAM was tested on 18 real-world datasets, 12 modalities, and 11 types of prediction tasks. The results show that it achieved competitive performance on a variety of tasks, with the paper reporting an average accuracy about 6.9 percentage points higher than the specialized fusion baselines compared. However, performance across different datasets was not entirely consistent, and the model was not superior to other methods in all tests. In addition to the public evaluation in the paper, Zhang Changqing also presented the team's preliminary tests on several practical tasks. In the direction of embodied intelligence, the team applied FAM to visual-tactile grasp success rate prediction. In the results demonstrated on site, FAM's prediction performance was close to that of a specialized method. This suggests that a general-purpose prediction foundation model may be usable for handling information from different sources such as vision and touch, but predicting whether a grasp will succeed and the robot actually executing the grasp action remain different problems. In another embodied intelligence-related test, the team attempted to use the model for scoring tasks in reinforcement learning, combining text instructions and video observations to evaluate different action trajectories. In the results shown on the on-site PPT, the scheme with FAM added outperformed the baseline schemes compared. The specific magnitude of improvement still needs to be understood in light of the corresponding experimental settings. In the medical direction, the team conducted multimodal skin disease diagnosis tests with the Department of Dermatology at Xiangya Hospital, involving information such as clinical images, dermoscopic images, and medical history. The report presented comparison results of FAM and methods such as PanDerm in the corresponding tests. These results can serve as preliminary evidence for exploring the potential of general multimodal prediction models in medical tasks, but they cannot directly lead to the conclusion that it is superior to specialized models in all skin disease diagnosis scenarios. The team also presented a historical manuscript recognition test. Facing handwritten text that is difficult to recognize directly, researchers used visual information, context, and other features for prediction. This case shows that a general prediction framework can be used not only for common image and text tasks, but may also integrate different types of auxiliary information. However, this work still needs to be further understood in light of its specific data, evaluation metrics, and experimental settings. From these explorations, it can be seen that the goal of Fusion Anything is not to immediately replace all specialized models with a single model, but to try to reduce the adaptation cost for new tasks, enabling multimodal fusion to be more flexibly transferred to different application scenarios. At present, many issues in this direction still require continued research, including post-training of the model, domain-specific adaptation, the choice of modality encoders, and how to reliably apply general prediction capabilities to real robot systems. Looking back at the entire report, multimodal fusion research involves three interrelated levels: at the representation stage, how to balance the commonality and individuality of different modalities; at the decision stage, how to use evidence to assess the uncertainty of predictions; and at the more general model level, how to make learned association patterns adapt to new modality combinations and prediction tasks. These issues have not all been resolved, but they jointly point to a clear research direction: enabling multimodal models not only to integrate more information, but also to use information more effectively, and to reduce the cost of repeated design and training when facing new tasks. Leiphone

CNCC is about to be held. Researchers, students, and readers interested in the frontiers of the computer field are welcome to join our CNCC exchange community.

Here, you can meet peers, exchange research, share admissions information, scientific research cooperation, dinner invitations, submissions, team formation for attending the conference, etc., and find potential partners.

During the conference, we will also continue to bring on-site report updates, interpretations of guests' speeches, summaries of frontier directions, and forum highlights.

See you at the venue, and see you in the group.

? Group entry portal: Scan the QR code to join the group or add WeChat Qvv0909777, with the note: CNCC + unit/school + name + direction.