How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_M# Run inference directly in the terminal:
llama cli -hf shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_MUse pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_M# Run inference directly in the terminal:
./llama-cli -hf shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_MBuild from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_M# Run inference directly in the terminal:
./build/bin/llama-cli -hf shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_MUse Docker
docker model run hf.co/shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_MQuick Links
llama3.1-8b-instruct 中文DPO版
模型介绍
像原版instruct一样，喜欢用有趣中文和表情符号回答问题。
Github：https://github.com/CrazyBoyM/llama3-Chinese-chat
特点：偏好中文和emoji表情，且不损伤原instruct版模型能力。实测中文DPO版问答性能体验超过现在市面上任何llama3.1中文微调版 （微调会大面积破坏llama3.1原版能力，导致遗忘）
训练细节
DPO(beta 0.5) + lora rank128, alpha256 + 打开"lm_head", "input_layernorm", "post_attention_layernorm", "norm"层训练.
模型部署
网页脚本文件：https://github.com/CrazyBoyM/llama3-Chinese-chat/blob/main/deploy/web_streamlit_for_instruct_v2.py
已经在模型文件夹里内置了一份web.py，可以直接使用。
pip install streamlit
pip install transformers==4.40.1
streamlit run web.py ./llama3.1-8b-instruct-dpo-zh
模型下载
SDK下载
#安装ModelScope
pip install modelscope
#SDK模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('shareAI/llama3.1-8b-instruct-dpo-zh')
Git下载
#Git模型下载
git clone https://www.modelscope.cn/shareAI/llama3.1-8b-instruct-dpo-zh.git
目前已经上传 Q4 K_M 的GGUF量化模型，后续将进一步进行手机端或边缘算力端展示
- Downloads last month
- 42
Hardware compatibility
Log In to add your hardware
4-bit
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_M# Run inference directly in the terminal: llama cli -hf shareAI/llama3.1-8b-instruct-dpo-zh:Q4_K_M