Instructions to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="brandonmusic/GLM-5.3-Flash-tr3-4bpw")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("brandonmusic/GLM-5.3-Flash-tr3-4bpw")
model = AutoModelForMultimodalLM.from_pretrained("brandonmusic/GLM-5.3-Flash-tr3-4bpw", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "brandonmusic/GLM-5.3-Flash-tr3-4bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.3-Flash-tr3-4bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw

SGLang

How to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "brandonmusic/GLM-5.3-Flash-tr3-4bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.3-Flash-tr3-4bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "brandonmusic/GLM-5.3-Flash-tr3-4bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.3-Flash-tr3-4bpw",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use brandonmusic/GLM-5.3-Flash-tr3-4bpw with Docker Model Runner:
```
docker model run hf.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw
```

GLM-5.3-Flash-tr3-4bpw / runtime-results /v71 /benchmarks /nvfp4-dcp2-mtp3-ws13-oc6000-c1-c16-through128k-tui.txt

brandonmusic

Publish v71 Docker image and long-decode TUI

2eb81e4 verified about 1 month ago

Raw History Blame Contribute Delete

8.74 kB



	llm-decode-bench v0.4.29
	╭────────────────────────────────── Phase 2 ───────────────────────────────────╮
	│ Sustained Decode │
	│ Steady-state decode throughput after the engine has admitted the requested │
	│ concurrency and passed warmup. Use this as the main tuning/regression signal │
	│ for kernels, NCCL, DCP, MTP, and scheduler changes. │
	╰──────────────────────────────────────────────────────────────────────────────╯
	Aggregate tok/s + TTFT/ITL
	╭────────────┬──────────┬──────────┬──────────┬──────────────┬───────────────╮
	│ ctx \ conc │ 1 │ 2 │ 4 │ 8 │ 16 │
	├────────────┼──────────┼──────────┼──────────┼──────────────┼───────────────┤
	│ 0 │ 146.7 │ 258.2 │ 393.9 │ 564.8 (7/8)* │ 563.1 (7/16)* │
	│ │ 76/7 │ 119/8 │ 7k/11 │ 481/13 │ 27k/13 │
	│ 16k │ 143.1 │ 260.3 │ 395.0 │ 315.3 (6/8)* │ 360.9 (6/16)* │
	│ │ 3k/7 │ 4k/8 │ 7k/13 │ 11k/15 │ 1e+01k/15 │
	│ 32k │ 144.6 │ 242.4 │ 385.6 │ 481.7 (6/8)* │ 445.1 (6/16)* │
	│ │ 5k/7 │ 8k/10 │ 14k/15 │ 19k/17 │ 20k/18 │
	│ 64k │ 143.2 │ 238.5 │ 379.2 │ 17.1 (5/8)* │ 26.5 (5/16)* │
	│ │ 11k/7 │ 17k/11 │ 28k/27 │ 40k/20 │ 39k/22 │
	│ 128k │ 0.0 (1e) │ 0.0 (2e) │ 0.0 (4e) │ 0.0 (8e) │ ∅ │
	╰────────────┴──────────┴──────────┴──────────┴──────────────┴───────────────╯
	Sustained Decode: aggregate tok/s uses OpenAI stream usage by default
	(continuous completion_tokens when the server supports it). Prometheus is kept
	as validation/scheduler data.
	Aggregate source(s): none, openai_continuous_usage
	∅ = skipped/hidden because the cell does not fit in KV cache; exact deficit is
	kept in JSON timeout_reason
	(X/Y) = avg running / requested concurrency from Prometheus; * =
	capacity-limited or warmup timed out
	Per-Request tok/s
	╭────────────┬───────┬───────┬──────┬─────────────┬──────────────╮
	│ ctx \ conc │ 1 │ 2 │ 4 │ 8 │ 16 │
	├────────────┼───────┼───────┼──────┼─────────────┼──────────────┤
	│ 0 │ 146.7 │ 129.1 │ 98.5 │ 70.6 (7/8)* │ 35.2 (7/16)* │
	│ 16k │ 143.1 │ 130.2 │ 98.8 │ 39.4 (6/8)* │ 22.6 (6/16)* │
	│ 32k │ 144.6 │ 121.2 │ 96.4 │ 60.2 (6/8)* │ 27.8 (6/16)* │
	│ 64k │ 143.2 │ 119.3 │ 94.8 │ 2.1 (5/8)* │ 1.7 (5/16)* │
	│ 128k │ - │ - │ - │ - │ ∅ │
	╰────────────┴───────┴───────┴──────┴─────────────┴──────────────╯
	Client request latency: p50 / p90 ms
	╭────────────┬─────┬─────┬─────┬───────────────┬───────────────╮
	│ ctx \ conc │ 1 │ 2 │ 4 │ 8 │ 16 │
	├────────────┼─────┼─────┼─────┼───────────────┼───────────────┤
	│ 0 │ —/— │ —/— │ —/— │ 56.5k/57.6k │ 55.7k/57.5k │
	│ 16k │ —/— │ —/— │ —/— │ 64.9k/66.7k │ 65.5k/65.5k │
	│ 32k │ —/— │ —/— │ —/— │ —/— │ —/— │
	│ 64k │ —/— │ —/— │ —/— │ 106.1k/110.9k │ 106.5k/111.3k │
	│ 128k │ - │ - │ - │ - │ ∅ │
	╰────────────┴─────┴─────┴─────┴───────────────┴───────────────╯
	Aggregate cells show dim detail as TTFT ms / ITL ms for the same ctx/conc
	coordinate. ITL is computed from observed generated tokens, including streams
	stopped at the measurement boundary; a missing ITL means no stream produced at
	least two measured output tokens. Per-request tok/s and request latency are
	shown in separate per-cell matrices. Completion/sample counts and full
	request-level distributions remain in JSON under request_samples.
	Sustained mode: client latency metrics explain request UX variance; aggregate
	tok/s remains the primary throughput signal.
	ITL=(last_token_time-first_token_time)/(output_tokens-1), user tok/s=1/ITL.

	╭────────────────────────────────── Phase 3 ───────────────────────────────────╮
	│ Burst / E2E Decode │
	│ Not run. Re-run with --run-burst to append a finite client-facing request │
	│ burst after Sustained Decode. This is intentionally disabled by default │
	│ because it adds another full decode matrix. │
	╰──────────────────────────────────────────────────────────────────────────────╯

	╭────────────────────────────── Primary Summary ───────────────────────────────╮
	│ Primary matrices repeated last so the important numbers are visible without │
	│ scrolling back through diagnostics. │
	╰──────────────────────────────────────────────────────────────────────────────╯
	Aggregate decode tok/s
	╭────────────┬──────────┬──────────┬──────────┬──────────────┬───────────────╮
	│ ctx \ conc │ 1 │ 2 │ 4 │ 8 │ 16 │
	├────────────┼──────────┼──────────┼──────────┼──────────────┼───────────────┤
	│ 0 │ 146.7 │ 258.2 │ 393.9 │ 564.8 (7/8)* │ 563.1 (7/16)* │
	│ 16k │ 143.1 │ 260.3 │ 395.0 │ 315.3 (6/8)* │ 360.9 (6/16)* │
	│ 32k │ 144.6 │ 242.4 │ 385.6 │ 481.7 (6/8)* │ 445.1 (6/16)* │
	│ 64k │ 143.2 │ 238.5 │ 379.2 │ 17.1 (5/8)* │ 26.5 (5/16)* │
	│ 128k │ 0.0 (1e) │ 0.0 (2e) │ 0.0 (4e) │ 0.0 (8e) │ ∅ │
	╰────────────┴──────────┴──────────┴──────────┴──────────────┴───────────────╯