steampunque/Qwen3.5-4B-MP-GGUF

Mixed Precision GGUF layer quantization of Qwen3.5-4B by Qwen

Original model: https://huggingface.co/Qwen/Qwen3.5-4B

The mixed precision quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 8/27/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOSD"],
   [0 ,"Q6_K_8868"], [1 ,"Q6_K_8866"], [2 ,"Q5_K_8866"], [3 ,"Q5_K_8666"],
   [4 ,"Q5_K_6666"], [5 ,"Q5_K_6656"], [6 ,"Q5_K_6656"], [7 ,"Q5_K_8666"],
   [8 ,"Q5_K_6555"], [9 ,"Q5_K_6555"], [10,"Q5_K_6555"], [11,"Q5_K_8665"],
   [12,"Q5_K_6555"], [13,"Q5_K_6555"], [14,"Q5_K_6555"], [15,"Q5_K_8665"],
   [16,"Q5_K_6665"], [17,"Q5_K_6665"], [18,"Q5_K_6665"], [19,"Q5_K_8666"],
   [20,"Q5_K_6666"], [21,"Q5_K_6666"], [22,"Q5_K_6666"], [23,"Q5_K_8666"],
   [24,"Q5_K_6685"], [25,"Q5_K_6685"], [26,"Q5_K_6685"], [27,"Q6_K_8886"],
   [28,"Q6_K_8888"], [29,"Q6_K_8888"], [30,"Q6_K_8888"], [31,"Q8_0_8666"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized for strong performance across a set of curated reasoning prompts with a minimum quant of Q5_K used across layers.

Comparison:

Quant	size	PPL	Comment
Q6_K	3.5e9	9.7	Q6_K with default embedding and output
Q6_E_H	3.3e9	9.7	Mixed precision quant with Q6_K embedding Q6_K output

Usage:

Qwen3.5-4B is a vision capable dense RL edge model. It can be used together with its multimedia projector layers to process images and text inputs and generate text outputs while being sized for applications on small/low resource edge platforms. The mmproj file is made available in this repository.

Straightforward speculation does not work with the model due to the attention scheme it uses.

On a 4070 with all layers and context in VRAM with no vision tower approx performance is:

Q	QKV	NKV	gen tps
Q6_E_H	F16	250k+	104
Q6_E_H + vision tower	F16	220k+	104
Q6_E_H	Q8_0	380k+	103

High context yarn config is as follows: set base context for yarn rope scale compute to 262144 (256k), then with a context of 380k tokens the rope scale = 380 / 256 = 1.48.

Then on model start pass --rope-scaling yarn --yarn-orig-ctx 262144 --rope_scale 1.48 (must be ajusted if kv other than 380k)

Later versions of llama.cpp have a bug which soft caps context length to the training context, effectively disabling yarn context extension. Patch server-context.cpp according to https://github.com/ggml-org/llama.cpp/issues/22140 to fix it.

The model appears to be trained to decide itself whether to do a think block or not. When it does a think block it can fall into very heavy overthinking and is prone to getting stuck in rep loops. To avoid the overthinking/rep loops inject think start and think stop tokens first thing after assistant prompt:

THINK_START="<think>\n"
THINK_STOP="\n</think>\n\n"

If the model doesnt feel like doing thinking on a given prompt it will automatically do this. To force the model into a think block inject a bootstrap think start following the assistant prompt:

"<think>\n"

The model was found to be capable on reasoning tasks when skipping think block, with little to no overthinking, just direct deductions to final solutions. When doing thinking with greedy sampling the model will go into infinite rep loops from time to time. This is similar behaviour to other qwen3 thinkers which have trouble with infinite repeat when using greedy sampling particularly at smaller quant sizes (<10B params). The problem seems to get more prevalent with RL training on smaller models. Setting a sampling TEMP 0.7 can reduce the rep problem at the expense of losing deterministic/ highest probability results.

VISION:

The model was tested in vision mode on a couple pretty tough bird ID image and did OK, iding 1 of 2 tough images correctly.

CODE:

The model was tested across a small set of code gen prompts and was unable to generate working code on all of the test prompts.

LONG CONTEXT:

The quant was tested against https://huggingface.co/datasets/steampunque/benchlm/blob/main/Qwen3_Runescape_Massive_Prompt.txt and solved the prompt correctly using greedy deterministic sampling:

lm Qwen3_Runescape_Massive_Prompt.txt 
<think>

</think>

Based on the **Skills/Experience Table** provided in the text, you reach **half the experience needed for the maximum level (Level 99)** at **Level 49**.

Here is the breakdown:
*   **Total experience needed for Level 99:** 13,034,431
*   **Half of that total:** 6,517,215.5
*   **Experience needed to reach Level 49:** 9,172,100 (Wait, let me re-calculate based on the table rows).

Let's look at the cumulative totals in the table:
*   **Level 48:** 83,014
*   **Level 49:** 91,721
*   **Level 50:** 101,333
*   ...
*   **Level 90:** 5,346,332
*   **Level 91:** 5,902,831
*   **Level 92:** 6,517,253

The experience needed to reach **Level 92** is **6,517,253**.
The experience needed to reach **Level 99** is **13,034,431**.
Half of 13,034,431 is **6,517,215.5**.

Since the experience required to reach **Level 92** (6,517,253) is extremely close to half the total (6,517,215.5), you are at **Level 92** when you have reached approximately half the experience required for the maximum level.

*(Note: The table shows the cumulative experience needed to reach that specific level. Level 92 is the first level where the cumulative total exceeds the halfway mark of the total experience for Level 99.)*

**Answer:** You are at **Level 92**.

The response suggests the correct answer is pulled out of trained latent space.

The quant also passes a 86k token needle in haystack test.

Benchmarks:

A full set of both math and vision benchmarks for the model will eventually be given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
Qwen3.5-4B.Q6_E_H.gguf	Q6_E_H	3.3e9 B	0.2B smaller than Q6_K
Qwen3.5-4B.mmproj.gguf	F16	0.67e9 B	multimedia projector

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040