蚂蚁 inclusionAI 开源 dFactory:面向 dLLM 的高效微调工具
蚂蚁 inclusionAI 在 GitHub 开源 dFactory,用于 dLLM 的高效微调,支持 LLaDA2.0-mini(16B)和 LLaDA2.0-flash(100B)两款模型,集成(连续)监督微调(block-diffusion、full attention)等方法。项目采用 Apache-2.0 许可,提供 MoE 专家权重合并与拆分脚本,训练需先将权重转为 merged-expert 格式。
into a single, higher-dimensional tensor Merging Experts Convert a model with separate expert weights into the consolidated "merged" format. By merging expert weights, we can leverage highly efficient batched matrix multiplication on GPUs, significantly speeding up computation. How it Works: The script iterates through each MoE layer and stacks the weights of all experts (e.g., gate proj, up proj, down proj) into a single tensor. Usage: Splitting Experts This process performs the reverse operat
匹配字段正文

