Instructions to use QuixiAI/DeepSeek-R1-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuixiAI/DeepSeek-R1-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuixiAI/DeepSeek-R1-AWQ", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("QuixiAI/DeepSeek-R1-AWQ", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("QuixiAI/DeepSeek-R1-AWQ", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use QuixiAI/DeepSeek-R1-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuixiAI/DeepSeek-R1-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/DeepSeek-R1-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuixiAI/DeepSeek-R1-AWQ

SGLang

How to use QuixiAI/DeepSeek-R1-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuixiAI/DeepSeek-R1-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/DeepSeek-R1-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuixiAI/DeepSeek-R1-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuixiAI/DeepSeek-R1-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use QuixiAI/DeepSeek-R1-AWQ with Docker Model Runner:
```
docker model run hf.co/QuixiAI/DeepSeek-R1-AWQ
```

v2ray commited on Mar 26, 2025

Commit

ba89d43

1 Parent(s): 007e788

Updated README.md to include better benchmark.

Browse files

Files changed (2) hide show

README.md +13 -6
config.json +1 -1

README.md CHANGED Viewed

@@ -11,18 +11,25 @@ library_name: transformers
 # DeepSeek R1 AWQ
 AWQ of DeepSeek R1.
 This quant modified some of the model code to fix an overflow issue when using float16.
 To serve using vLLM with 8x 80GB GPUs, use the following command:
 ```sh
-VLLM_WORKER_MULTIPROC_METHOD=spawn python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 12345 --max-model-len 65536 --max-num-batched-tokens 65536 --trust-remote-code --tensor-parallel-size 8 --gpu-memory-utilization 0.97 --dtype float16 --served-model-name deepseek-reasoner --model cognitivecomputations/DeepSeek-R1-AWQ
 ```
-You can download the wheel I built for PyTorch 2.6, Python 3.12 by clicking [here](https://huggingface.co/x2ray/wheels/resolve/main/vllm-0.7.3.dev187%2Bg0ff1a4df.d20220101.cu126-cp312-cp312-linux_x86_64.whl).
-Inference speed with batch size 1 and short prompt:
-- 8x H100: 48 TPS
-- 8x A100: 38 TPS
 Note:
 - Inference speed will be better than FP8 at low batch size but worse than FP8 at high batch size, this is the nature of low bit quantization.
-- vLLM supports MLA for AWQ now, you can run this model with full context length on just 8x 80GB GPUs.

 # DeepSeek R1 AWQ
 AWQ of DeepSeek R1.
+Quantized by [Eric Hartford](https://huggingface.co/ehartford) and [v2ray](https://huggingface.co/v2ray).
 This quant modified some of the model code to fix an overflow issue when using float16.
 To serve using vLLM with 8x 80GB GPUs, use the following command:
 ```sh
+VLLM_USE_V1=0 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_MARLIN_USE_ATOMIC_ADD=1 python -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 12345 --max-model-len 65536 --max-seq-len-to-capture 65536 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --tensor-parallel-size 8 --gpu-memory-utilization 0.95 --served-model-name deepseek-reasoner --model cognitivecomputations/DeepSeek-R1-AWQ
 ```
+You can download the wheel I built for PyTorch 2.6, Python 3.12 by clicking [here](https://huggingface.co/x2ray/wheels/resolve/main/vllm-0.8.3.dev166%2Bg29930428e.cu128-cp312-cp312-linux_x86_64.whl), the benchmark below was done with this wheel, it contains 2 PR merges which boosted performance a lot.
+## TPS Per Request
+| GPU \ Batch Input Output | B: 1 I: 2 O: 2K | B: 32 I: 4K O: 256 | B: 1 I: 63K O: 2K | Prefill |
+|:-:|:-:|:-:|:-:|:-:|
+| **8x H100/H200** | 61.5 | 30.1 | 54.3 | 4732.2 |
+| **4x H200** | 58.4 | 19.8 | 53.7 | 2653.1 |
+| **8x A100 80GB** | 45.5 | 11.9 | 7.3 | 2435.5 |
 Note:
+- The A100 config is extremely slow on high context is caused by FlashMLA not supporting anything below Hopper GPUs (H200, H100, H800, H20), before it's supported, vLLM will use the Triton implementation which is extremely slow on high context. Thus it's best to serve this model with either 8x H100 or 4x H200.
+- All 3 types of GPU are SXM form factor.
 - Inference speed will be better than FP8 at low batch size but worse than FP8 at high batch size, this is the nature of low bit quantization.
+- vLLM supports MLA for AWQ now, you can run this model with full context length on just 8x 80GB GPUs.

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "/home/ubuntu/models/DeepSeek-R1-bf16",
   "architectures": [
     "DeepseekV3ForCausalLM"
   ],

 {
+  "_name_or_path": "cognitivecomputations/DeepSeek-R1-AWQ",
   "architectures": [
     "DeepseekV3ForCausalLM"
   ],