Instructions to use QuixiAI/DeepSeek-R1-0528-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuixiAI/DeepSeek-R1-0528-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuixiAI/DeepSeek-R1-0528-AWQ", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("QuixiAI/DeepSeek-R1-0528-AWQ", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("QuixiAI/DeepSeek-R1-0528-AWQ", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use QuixiAI/DeepSeek-R1-0528-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuixiAI/DeepSeek-R1-0528-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/DeepSeek-R1-0528-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuixiAI/DeepSeek-R1-0528-AWQ

SGLang

How to use QuixiAI/DeepSeek-R1-0528-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuixiAI/DeepSeek-R1-0528-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/DeepSeek-R1-0528-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuixiAI/DeepSeek-R1-0528-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/DeepSeek-R1-0528-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use QuixiAI/DeepSeek-R1-0528-AWQ with Docker Model Runner:
```
docker model run hf.co/QuixiAI/DeepSeek-R1-0528-AWQ
```

DeepSeek-R1-0528-AWQ

AWQ of DeepSeek R1 0528.

Quantized by Eric Hartford and v2ray.

Compute for this quant was generously sponsored by Hot Aisle. Thank you for supporting the community!

This quant modified some of the model code to fix an overflow issue when using float16.

To serve using vLLM with 8x 80GB GPUs, use the following command:

VLLM_USE_V1=0 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_MARLIN_USE_ATOMIC_ADD=1 python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 12345 --max-model-len 65536 --max-seq-len-to-capture 65536 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --tensor-parallel-size 8 --gpu-memory-utilization 0.95 --served-model-name deepseek-chat --model cognitivecomputations/DeepSeek-V3-0324-AWQ

You can download the wheel I built for PyTorch 2.6, Python 3.12 by clicking here, the benchmark below was done with this wheel, it contains 2 PR merges and an unoptimized FlashMLA (still faster than Triton) for A100 which boosted performance a lot. The vLLM repo which contained A100 FlashMLA can be found at LagPixelLOL/vllm@sm80_flashmla, which is a fork of vllm-project/vllm. The A100 FlashMLA it used is based on LagPixelLOL/FlashMLA@vllm, which is a fork of pzhao-eng/FlashMLA.

TPS Per Request

GPU \ Batch Input Output	B: 1 I: 2 O: 2K	B: 32 I: 4K O: 256	B: 1 I: 63K O: 2K	Prefill
8x H100/H200	61.5	30.1	54.3	4732.2
4x H200	58.4	19.8	53.7	2653.1
8x A100 80GB	46.8	12.8	30.4	2442.4
8x L40S	46.3	OOM	OOM	688.5

Note:

The A100 config uses an unoptimized FlashMLA implementation, which is only superior than Triton during high context inference, it would be faster if it's optimized.
The L40S config doesn't support FlashMLA, thus the Triton implementation is used, this makes it extremely slow with high context. But the L40S doesn't have much VRAM, so it can't really have that much context anyway, and it also doesn't have the fast GPU to GPU interconnection bandwidth, making it even slower. It is not recommended to serve with this config, as you must limit the context to <= 4096, --gpu-memory-utilization to 0.98, and --max-num-seqs to 4.
All types of GPU used during benchmark are SXM form factor except L40S.
Inference speed will be better than FP8 at low batch size but worse than FP8 at high batch size, this is the nature of low bit quantization.
vLLM supports MLA for AWQ now, you can run this model with full context length on just 8x 80GB GPUs.

Downloads last month: 1,238

Safetensors

Model size

671B params

Tensor type

I32

BF16

Model tree for QuixiAI/DeepSeek-R1-0528-AWQ

Base model

deepseek-ai/DeepSeek-R1-0528

Quantized

(48)

this model