---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
---

# dMoE-16B: dLLMs with Learnable Block Experts

[dMoE](https://fscdc.github.io/dMoE/) is a block-level Mixture-of-Experts (MoE) framework designed for Diffusion Large Language Models (dLLMs). By aggregating token-level expert distributions within each block into a unified block-level distribution, dMoE substantially reduces the number of uniquely activated experts during inference, mitigating memory-bound bottlenecks without sacrificing performance.

- **Paper:** [dMoE: dLLMs with Learnable Block Experts](https://huggingface.co/papers/2605.30876)
- **Project Page:** [https://fscdc.github.io/dMoE/](https://fscdc.github.io/dMoE/)
- **Repository:** [https://github.com/fscdc/dMoE](https://github.com/fscdc/dMoE)

## Highlights

- **Learnable Block Experts**: Introduces block-level MoE routing into dLLMs, drastically compressing the number of activated unique experts across diffusion steps.
- **Reduced MoE Bandwidth**: Significantly reduces memory bandwidth consumed by expert weight loading during the block diffusion process.
- **Improved Efficiency-Accuracy Trade-off**: Achieves 1.14x to 1.66x end-to-end latency speedup while maintaining competitive performance on benchmarks.
- **Plug-and-play on LLaDA-2.0**: Built directly on top of LLaDA-2.0-mini without architectural changes.

## Sample Usage

Please refer to the GitHub: https://github.com/fscdc/dMoE

## Citation

```bibtex
@article{feng2026dmoe,
  title={dMoE: dLLMs with Learnable Block Experts},
  author={Feng, Sicheng and Chen, Zigeng and Fang, Gongfan and Ma, Xinyin and Wang, Xinchao},
  journal={arXiv preprint arXiv:2605.30876},
  year={2026}
}
```