LG AI Research 提出 cluster-aware upcycling：在 MoE 中打造真正的专家

Pyunghwan Ahn

2026.06.26

The most intuitive way to improve the performance of an AI model is to make it larger. However, in a dense model where all parameters process every input, the costs of training and inference increase in proportion to the model’s size. Mixture-of-Experts, or MoE, is a structure developed to mitigate this problem. MoE consists of multiple subnetworks (Experts) within a single network, and by activating only a subset of these experts for each input token, it allows the overall model capacity to be increased while keeping the actual computational load relatively low.

One method for efficiently training such MoE structures is upcycling. This approach initializes the MoE model using the parameters of a pre-trained dense model and then performs additional MoE training for a relatively short period, thereby achieving high performance with relatively low training costs. The paper “Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling”[1], presented at CVPR 2026, analyzes how the initialization method used when upcycling a pre-trained dense model into an MoE model affects the final training results and proposes an initialization method for effective training.

Image 1. Sparse upcycling[2] concept

“We switched to MoE, but the expert doesn’t 'act like an expert'”

The appeal of the MoE structure lies in its ability to achieve high performance with minimal inference costs. Looking at real-world industrial applications, cost is just as important as performance when operating large-scale AI models. While performance may improve as models continue to grow, dense structures that use all parameters every time come with the burden of increasing both training and inference costs. In contrast, MoE structures retain the majority of the parameters but call upon only a subset of experts for each input, thereby ensuring inference efficiency.

However, training a MoE from scratch remains costly. To address this, a Google Research group proposed “sparse upcycling”[2] in 2022, a method for repurposing existing dense checkpoints as MoEs. The basic approach involves copying the FFN of a dense model to multiple experts and reinitializing the router. The problem is that all experts start with the same parameters. While it may appear that multiple experts have been created, they are actually closer to “duplicates” that perform similar functions, which can weaken early specialization. The paper defines this as the “expert symmetry” problem.

Previous studies have attempted to break this symmetry through methods such as noise injection, partial parameter reinitialization, and fine-tuning based on passive domain partitioning. However, noise may have limited effectiveness, methods that significantly alter parameters risk damaging the pretrained representation, and domain-specific fine-tuning becomes less scalable as the number of experts increases.

LG AI Research analyzed how the final training results and performance can vary depending on the initialization method used when upcycling pre-trained dense models into MoE models, and proposed an initialization technique for more effective training.

This paper is based on the core premise that “the activations of an already well-trained dense model contain the semantic structure of the data.” Therefore, rather than copying all experts with the same weights, it is possible to first examine the activation space of the dense model, identify regions where semantically similar inputs cluster, and initialize each expert to handle a different region. The paper refers to this strategy as “cluster-aware upcycling.”

To use an analogy, traditional sparse upcycling is like placing several employees who have received the same training in a room and telling them, “Now, each of you should find your own area of expertise.” In contrast, cluster-aware upcycling is more akin to first analyzing work data to group semantically related tasks such as “teams strong in image-text search” or “teams sensitive to specific visual patterns,” then assigning each employee to the appropriate task from the start.

Step 1. Clustering FFN activations with spherical k-means In the first step, we extract the activations input to each FFN block of the pretrained dense model using a small calibration dataset. We then divide the extracted activations into a specified number of clusters using cosine-similarity-based spherical k-means clustering. This choice is important because the logit calculation in the MoE router is also based on directional alignment, that is, the similarity between the activations and the router weights. As a result of this process, a number of activation clusters and centroids equal to the number of experts are generated. The paper views these clusters and centroids as a semantic partition of the activations generated by the dense model and uses them as the basis for initializing the experts and the router.

Step 2. Initializing each expert according to the cluster subspace

The core objective of this paper was to preserve the capabilities of the pretrained dense model as much as possible while ensuring that the unique capabilities of each expert were differentiated from one another. In other words, each expert must be initialized to effectively preserve the input distribution of its corresponding cluster, and data-aware truncated SVD was used to achieve this. Rather than performing SVD based solely on the weight matrix, it preserves the principal singular directions by reflecting the input statistics of the respective cluster. This approach allows each expert to be responsible for a different principal subspace while retaining pretrained knowledge. As a result, it is possible to reduce expert redundancy while maintaining the benefits of upcycling.

Image 2. Cluster-aware initialization concept diagram. [1]

After dividing the FFN input activations of the dense model into clusters, the subspaces for each cluster are used for expert initialization, and the cluster centroids are used for router initialization.

EESD: Refer to the ensemble teacher when routing is uncertain

The paper proposes a new loss function to prevent potential losses arising from uncertain expert assignments during the training process. In MoE, when it is unclear which expert a token should be assigned to, resulting in nearly uniform router probabilities, that token may hinder expert specialization. To prevent this, EESD uses a dense ensemble teacher updated via the EMA method. While standard sparse MoE activates only the top-k experts, the ensemble teacher activates all experts simultaneously to provide more stable predictions; this teacher is designed to provide strong guidance for tokens with high routing uncertainty. Conversely, for tokens where the router is already confident, the loss is minimized so as not to interfere with existing specialization.

Verification through experiment: CLIP ViT-B/32 and ViT-B/16

The experimental setup for this paper used CLIP ViT-B/32 and ViT-B/16 as dense baselines. Dense pre-training followed the LAION-400M-based openCLIP setup, and for MoE upcycling, a dense checkpoint from the 15th epoch was used, alternating between the existing dense layer and the MoE layer.

Image 3. Cluster-aware upcycling has been shown to reduce similarity among experts and stabilize routing entropy. This demonstrates that the internal structure of MoE has become more specialized, going beyond a simple performance improvement. [1]

[1] Chu, Sanghyeok, et al. "Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[2] Komatsuzaki, Aran, et al. "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints." The Eleventh International Conference on Learning Representations.