--- license: apache-2.0 tags: - mxfp4_hybrid - gguf - text-generation - quantized - cpu - gpu - mxfp4 - mxfp4_moe - magicquant - magic_quant - IQ4_NL base_model: - unsloth/Qwen3-4B-Instruct-2507 --- # MagicQuant GGUF Hybrids - Qwen3 4B Instruct 2507 > **MagicQuant is an automated quantization, benchmarking, and evolutionary hybrid-GGUF search system for LLMs.** Each release includes models optimized to outperform standard baseline quants (Q8, Q6, Q5, Q4). If a baseline GGUF exists in this repo, the evolutionary engine couldn’t beat it. If a baseline is missing, it’s because a hybrid configuration outperformed it so completely that including the baseline would've been pointless. These hybrid GGUFs are built to be as small, fast, and low-drift as possible while preserving model capability. To dive deeper into how MagicQuant works, see the main repo: [MagicQuant on GitHub (by MagicCodingMan)](https://github.com/magiccodingman/MagicQuant-Wiki) **Notes:** * The HuggingFace hardware compatibility where it shows the bits is usually wrong. It doesn't understand hybrid mixes, so don't trust it. * Naming scheme can be found on the MagicQuant Wiki. * (tips) Less precision loss means less brain damage. More TPS means faster! Smaller is always better right? **Precision Loss Guide** * **0–0.1%** → God-tier, scientifically exact * **0.1–1%** → True near-lossless, agent-ready * **1–3%** → Minimal loss, great for personal use * **3–5%** → Borderline, but still functional * **5%+** → Toys, not tools, outside MagicQuant’s scope [Learn more about precision loss here](https://github.com/magiccodingman/MagicQuant-Wiki/blob/main/docs/precision-loss-guide.md). ### Table - File Size + TPS + Avg Precision Loss | model_name | file_size_gb | bench_tps | avg_prec_loss | | ---------- | ------------ | --------- | ------------- | | [mxfp4_moe-K-B16-QO-Q6K-EUD-Q8_0](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-K-B16-QO-Q6K-EUD-Q8_0.gguf?download=true) | 3.98 | 373.48 | 0.0533% | | [mxfp4_moe-O-Q5K-EQKUD-Q6K](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-O-Q5K-EQKUD-Q6K.gguf?download=true) | 3.03 | 428.37 | 0.1631% | | [mxfp4_moe-QUD-IQ4NL-KO-MXFP4-E-Q8_0](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-QUD-IQ4NL-KO-MXFP4-E-Q8_0.gguf?download=true) | 2.28 | 411.49 | 0.7356% | | [mxfp4_moe-K-B16-QU-IQ4NL-O-MXFP4-E-Q5K-D-Q6K](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-K-B16-QU-IQ4NL-O-MXFP4-E-Q5K-D-Q6K.gguf?download=true) | 2.62 | 467.79 | 0.8322% | | [IQ4_NL](./../../resolve/main/Qwen3-4B-Instruct-2507-IQ4_NL.gguf?download=true) | 2.23 | 426.86 | 0.8996% | | [mxfp4_moe-EQUD-IQ4NL-KO-MXFP4](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-EQUD-IQ4NL-KO-MXFP4.gguf?download=true) | 2.10 | 518.15 | 2.0904% | ### Table - PPL Columns | model_name | gen | gen_er | code | code_er | math | math_er | | ---------- | --- | ------ | ---- | ------- | ---- | ------- | | [mxfp4_moe-K-B16-QO-Q6K-EUD-Q8_0](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-K-B16-QO-Q6K-EUD-Q8_0.gguf?download=true) | 8.8766 | 0.2053 | 1.5463 | 0.0122 | 6.7119 | 0.1368 | | [mxfp4_moe-O-Q5K-EQKUD-Q6K](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-O-Q5K-EQKUD-Q6K.gguf?download=true) | 8.8564 | 0.2036 | 1.5473 | 0.0122 | 6.6976 | 0.1358 | | [mxfp4_moe-QUD-IQ4NL-KO-MXFP4-E-Q8_0](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-QUD-IQ4NL-KO-MXFP4-E-Q8_0.gguf?download=true) | 9.0127 | 0.2057 | 1.5546 | 0.0119 | 6.6919 | 0.1331 | | [mxfp4_moe-K-B16-QU-IQ4NL-O-MXFP4-E-Q5K-D-Q6K](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-K-B16-QU-IQ4NL-O-MXFP4-E-Q5K-D-Q6K.gguf?download=true) | 9.0490 | 0.2096 | 1.5535 | 0.0121 | 6.7221 | 0.1358 | | [IQ4_NL](./../../resolve/main/Qwen3-4B-Instruct-2507-IQ4_NL.gguf?download=true) | 8.9948 | 0.2072 | 1.5600 | 0.0123 | 6.7484 | 0.1362 | | [mxfp4_moe-EQUD-IQ4NL-KO-MXFP4](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-EQUD-IQ4NL-KO-MXFP4.gguf?download=true) | 9.2104 | 0.2106 | 1.5598 | 0.0119 | 6.8261 | 0.1363 | ### Table - Precision Loss Columns | model_name | loss_general | loss_code | loss_math | | ---------- | ------------ | --------- | --------- | | [mxfp4_moe-K-B16-QO-Q6K-EUD-Q8_0](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-K-B16-QO-Q6K-EUD-Q8_0.gguf?download=true) | 0.0720 | 0.0388 | 0.0492 | | [mxfp4_moe-O-Q5K-EQKUD-Q6K](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-O-Q5K-EQKUD-Q6K.gguf?download=true) | 0.2994 | 0.0259 | 0.1640 | | [mxfp4_moe-QUD-IQ4NL-KO-MXFP4-E-Q8_0](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-QUD-IQ4NL-KO-MXFP4-E-Q8_0.gguf?download=true) | 1.4601 | 0.4978 | 0.2489 | | [mxfp4_moe-K-B16-QU-IQ4NL-O-MXFP4-E-Q5K-D-Q6K](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-K-B16-QU-IQ4NL-O-MXFP4-E-Q5K-D-Q6K.gguf?download=true) | 1.8687 | 0.4267 | 0.2012 | | [IQ4_NL](./../../resolve/main/Qwen3-4B-Instruct-2507-IQ4_NL.gguf?download=true) | 1.2586 | 0.8469 | 0.5933 | | [mxfp4_moe-EQUD-IQ4NL-KO-MXFP4](./../../resolve/main/Qwen3-4B-Instruct-2507-mxfp4_moe-EQUD-IQ4NL-KO-MXFP4.gguf?download=true) | 3.6857 | 0.8339 | 1.7515 | --- ### Baseline Models (Reference) ### Table - File Size + TPS + Avg Precision Loss | model_name | file_size_gb | bench_tps | avg_prec_loss | | ---------- | ------------ | --------- | ------------- | | BF16 | 7.50 | 254.70 | 0.0000% | | Q8_0 | 3.99 | 362.48 | 0.0724% | | Q6_K | 3.08 | 397.92 | 0.2492% | | Q5_K | 2.69 | 385.17 | 0.7920% | | IQ4_NL | 2.23 | 426.86 | 0.8996% | | Q4_K_M | 2.33 | 377.19 | 0.9376% | | MXFP4_MOE | 2.00 | 467.13 | 8.2231% | ### Table - PPL Columns | model_name | gen | gen_er | code | code_er | math | math_er | | ---------- | --- | ------ | ---- | ------- | ---- | ------- | | BF16 | 8.8830 | 0.2056 | 1.5469 | 0.0122 | 6.7086 | 0.1369 | | Q8_0 | 8.8754 | 0.2053 | 1.5488 | 0.0123 | 6.7080 | 0.1367 | | Q6_K | 8.8441 | 0.2034 | 1.5452 | 0.0121 | 6.6952 | 0.1357 | | Q5_K | 8.9707 | 0.2079 | 1.5542 | 0.0123 | 6.7701 | 0.1384 | | IQ4_NL | 8.9948 | 0.2072 | 1.5600 | 0.0123 | 6.7484 | 0.1362 | | Q4_K_M | 8.9446 | 0.2051 | 1.5694 | 0.0125 | 6.7532 | 0.1371 | | MXFP4_MOE | 9.8799 | 0.2282 | 1.6122 | 0.0130 | 7.3275 | 0.1494 | ### Table - Precision Loss Columns | model_name | loss_general | loss_code | loss_math | | ---------- | ------------ | --------- | --------- | | BF16 | 0.0000 | 0.0000 | 0.0000 | | Q8_0 | 0.0856 | 0.1228 | 0.0089 | | Q6_K | 0.4379 | 0.1099 | 0.1997 | | Q5_K | 0.9873 | 0.4719 | 0.9167 | | IQ4_NL | 1.2586 | 0.8469 | 0.5933 | | Q4_K_M | 0.6935 | 1.4545 | 0.6648 | | MXFP4_MOE | 11.2226 | 4.2213 | 9.2255 | --- ## Support I’m a solo developer working full time for myself to achieve my dream, pouring nights and weekends into open protocols and tools that I hope make the world a little better. If you chip in, you're helping me keep the lights on while I keep shipping. [Click here to see ways to support](https://sayou.biz/support) - BTC, Paypal, GitHub sponsors. Or, just drop a like on the repo :)