Qwen3.8-27B-DFlash2 — EXL3 4.00 bpw (draft)
EXL3 (ExLlamaV3) quantization of the incoai/Qwen3.8-27B-DFlash2 block-diffusion draft for use with r0b0tlab/Qwen3.8-27B-EXL3-4.00bpw.
Backbone Linears quantized at 4.00 bpw; the dynamic-convolution base_kernel tensors and the
candidate-selector codebooks stay fp16 (uncalibrated). Quantization is acceptance-neutral on
this stack (overlay measurement: mean acceptance 5.474 EXL3 vs 5.463 BF16 draft). Native
engine GSM8K: 5.657 AL / 162.9 tok/s.
Requires r0b0tlab/exllamav3 branch community
@ 355c6ee — native DFlash2DraftModel, selected automatically via -dm <dir>. Do not
use the old dflash2-pathway overlay.
Resource requirements
- Weights on disk: 1.2 GiB (this repo). Target repo: 15.4 GiB.
- This draft adds ~1.22 GiB of fp32 GDN verify states (per sequence slot) to the target's footprint; its own KV is windowed (5 sliding-window layers, tens of MB). Target at full 262,144-token context with cq3: ~21.7 GB loaded, peak ~22.7 GiB.
Use
python examples/chat.py -m <target-dir> -mode chatml \
-dm <this-dir> -cs 262144 -cq 3
Container (click-run, auto-downloads both repos on first run):
docker run --gpus all -v qwen38-models:/models \
ghcr.io/r0b0tlab/qwen38-exl3-dflash2:1.5.0-native
Metrics and harnesses: https://github.com/r0b0tlab/qwen38-exl3-dflash2
- Downloads last month
- 776