Instructions to use CohereLabs/c4ai-command-r-plus with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CohereLabs/c4ai-command-r-plus with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="CohereLabs/c4ai-command-r-plus")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("CohereLabs/c4ai-command-r-plus")
model = AutoModelForCausalLM.from_pretrained("CohereLabs/c4ai-command-r-plus")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use CohereLabs/c4ai-command-r-plus with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CohereLabs/c4ai-command-r-plus"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/c4ai-command-r-plus",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CohereLabs/c4ai-command-r-plus

SGLang

How to use CohereLabs/c4ai-command-r-plus with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CohereLabs/c4ai-command-r-plus" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/c4ai-command-r-plus",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CohereLabs/c4ai-command-r-plus" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/c4ai-command-r-plus",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use CohereLabs/c4ai-command-r-plus with Docker Model Runner:
```
docker model run hf.co/CohereLabs/c4ai-command-r-plus
```

MMLU is only 25.64, anything wrong?

by cloudyu - opened Apr 5, 2024

Discussion

cloudyu

Apr 5, 2024

I just check https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard and found the metric about this model is really bad, is there anything wrong about the score?

sarahooker

Apr 5, 2024

Hey @cloudyu -- I don't see this score or any model with this score average on the leaderboard. Can you specify the model name you are seeing?

Command R plus is not yet on the leaderboard -- it should be on the leaderboard shortly. We submitted it jointly with hugging face yesterday and my understanding is that it will be made public shortly.

sarahooker

Apr 5, 2024

I'm going to close this for now -- but feel free to re-open with additional details.

sarahooker changed discussion status to closed Apr 5, 2024

clefourrier

Apr 5, 2024

Hi @cloudyu , a random results file was accidentally pushed on our side under the wrong namespace - you can find the c4ai-command-r-plus details here while the leaderboard is rebuilding.

cloudyu

Apr 5, 2024

Now MMLU is 75.73 on the leaderboard; that's great.

sarahooker

Apr 6, 2024

Thanks @cloudyu ! Our full results on the Open LLM leaderboard is now public on https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard -- here is a quick comparison with a subset of other relevant models whose scores are publicly available on the leaderboard.

Hope this is helpful!

sarahooker changed discussion status to open Apr 6, 2024

sarahooker changed discussion status to closed Apr 7, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment