Add files using upload-large-folder tool

README.md

@@ -0,0 +1,157 @@
+# KAME Finetuned Moshi Checkpoint
+
+This checkpoint contains a cleaned KAME finetuned Moshi model.
+
+Files:
+
+- `model.safetensors`: model weights
+- `moshi_lm_kwargs.json`: model architecture/config for this checkpoint
+
+Notes:
+
+- The weights are stored in fp32.
+- Inference in the server scripts used in this repo typically runs in bfloat16 by default unless a different dtype is explicitly requested.
+- This release does not include a `config.json`.
+- When loading this checkpoint, pass `moshi_lm_kwargs.json` explicitly as `--config-path`.
+- Tokenizer and Mimi assets are expected to come from `kyutai/moshiko-pytorch-bf16`.
+
+## Quick Start
+
+Example with `server_oracle.py`:
+
+```bash
+cd /path/to/moshi
+
+CHECKPOINT_DIR=/path/to/checkpoint
+MODEL_WEIGHT=$CHECKPOINT_DIR/model.safetensors
+MODEL_CONFIG=$CHECKPOINT_DIR/moshi_lm_kwargs.json
+HF_REPO=kyutai/moshiko-pytorch-bf16
+
+uv run python -m moshi.server_oracle \
+  --hf-repo "$HF_REPO" \
+  --moshi-weight "$MODEL_WEIGHT" \
+  --config-path "$MODEL_CONFIG" \
+  --device cuda
+```
+
+Make sure any required environment variables for your local setup are already set
+before launching the server.
+
+Then open:
+
+- `http://localhost:8998`
+
+If the GPU machine is remote and you need a public URL, add:
+
+- `--gradio-tunnel`
+
+---
+
+Copy everything above this line into the Hugging Face repo `README.md`.
+
+## Internal Notes
+
+This section is for internal team use and can stay in the local copy of this directory.
+
+Current local directory:
+
+- `/home/manatoyaguchi_sakana_ai/kame_finetune_dev/output/moshiko-finetuned_all_en_oracle_emb_separate_copy_single_multi/step_6000_fp32_cleaned`
+
+### Local Files
+
+- `model.safetensors`: model weights
+- `moshi_lm_kwargs.json`: model architecture/config used for this checkpoint
+
+### Important
+
+- This checkpoint is currently shared as `model.safetensors + moshi_lm_kwargs.json`.
+- There is no `config.json` in this directory.
+- When loading this checkpoint, pass `moshi_lm_kwargs.json` explicitly as `--config-path`.
+- Tokenizer and Mimi assets are expected to come from `kyutai/moshiko-pytorch-bf16`.
+- The weights are stored in fp32.
+- In `server_oracle.py` and the evaluation scripts used in this repo, inference runs in bfloat16 by default unless a different dtype is explicitly requested.
+
+### Quick Start
+
+Use this checkpoint together with the base Moshi repo:
+
+```bash
+cd /home/manatoyaguchi_sakana_ai/kame_finetune_dev
+
+MODEL_DIR=$PWD/output/moshiko-finetuned_all_en_oracle_emb_separate_copy_single_multi/step_6000_fp32_cleaned
+MODEL_WEIGHT=$MODEL_DIR/model.safetensors
+MODEL_CONFIG=$MODEL_DIR/moshi_lm_kwargs.json
+HF_REPO=kyutai/moshiko-pytorch-bf16
+```
+
+### Main Use Case: UI Inference with `server_oracle.py`
+
+The main intended use of this checkpoint is interactive inference through
+`kame_dev/moshi/moshi/server_oracle.py`.
+
+Start the server like this:
+
+```bash
+cd /home/manatoyaguchi_sakana_ai/kame_dev/moshi
+
+set -a
+source ../.env
+set +a
+
+MODEL_DIR=/home/manatoyaguchi_sakana_ai/kame_finetune_dev/output/moshiko-finetuned_all_en_oracle_emb_separate_copy_single_multi/step_6000_fp32_cleaned
+MODEL_WEIGHT=$MODEL_DIR/model.safetensors
+MODEL_CONFIG=$MODEL_DIR/moshi_lm_kwargs.json
+HF_REPO=kyutai/moshiko-pytorch-bf16
+
+uv run python -m moshi.server_oracle \
+  --hf-repo "$HF_REPO" \
+  --moshi-weight "$MODEL_WEIGHT" \
+  --config-path "$MODEL_CONFIG" \
+  --device cuda
+```
+
+Then open:
+
+- `http://localhost:8998`
+
+If the GPU machine is remote and you need a public URL, add:
+
+- `--gradio-tunnel`
+
+Notes:
+
+- `moshi_lm_kwargs.json` must be passed explicitly as `--config-path`.
+- `kyutai/moshiko-pytorch-bf16` provides the base tokenizer and Mimi assets.
+- By default this runs in `bfloat16`. Add `--half` if you explicitly want `float16`.
+
+### MT-Bench Evaluation (For Internal Use Only)
+
+```bash
+cd /home/manatoyaguchi_sakana_ai/kame_finetune_dev
+
+set -a
+source ../kame_dev/.env
+set +a
+
+INPUT_DIR=/home/shared/sokuroki_sakana_ai/moshi/yaguchi/mt_bench
+MODEL_DIR=$PWD/output/moshiko-finetuned_all_en_oracle_emb_separate_copy_single_multi/step_6000_fp32_cleaned
+MODEL_WEIGHT=$MODEL_DIR/model.safetensors
+MODEL_CONFIG=$MODEL_DIR/moshi_lm_kwargs.json
+HF_REPO=kyutai/moshiko-pytorch-bf16
+
+uv run python -m experimental.mt_bench.run_model_eval \
+  --input-dir "$INPUT_DIR" \
+  --output-dir "$PWD/data/mt_bench_kame_step6000_smoke" \
+  --moshi-weight "$MODEL_WEIGHT" \
+  --config-path "$MODEL_CONFIG" \
+  --hf-repo "$HF_REPO" \
+  --backend openai \
+  --question-ids 106 \
+  --limit 1
+```
+
+### Additional Notes
+
+- For shared use, the most important files are `model.safetensors` and `moshi_lm_kwargs.json`.
+- If you upload this directory to Hugging Face as-is, downstream users should download both files and pass `moshi_lm_kwargs.json` explicitly when loading.
+- `config.json` is not included here. This checkpoint is currently shared in a "documented artifact" style rather than as a standalone self-describing HF repo.

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29ff0aefb06307100183d60da7667d27f7fd6f039fc23115dca37675bcd0fc7a
+size 31275273448

moshi_lm_kwargs.json

@@ -0,0 +1,48 @@
+{
+    "dim": 4096,
+    "text_card": 32000,
+    "existing_text_padding_id": 3,
+    "n_q": 16,
+    "dep_q": 8,
+    "card": 2048,
+    "num_heads": 32,
+    "num_layers": 32,
+    "hidden_scale": 4.125,
+    "causal": true,
+    "layer_scale": null,
+    "context": 3000,
+    "max_period": 10000,
+    "gating": "silu",
+    "norm": "rms_norm_f32",
+    "positional_embedding": "rope",
+    "depformer_dim": 1024,
+    "depformer_dim_feedforward": 4224,
+    "depformer_num_heads": 16,
+    "depformer_num_layers": 6,
+    "depformer_layer_scale": null,
+    "depformer_multi_linear": true,
+    "depformer_context": 8,
+    "depformer_max_period": 10000,
+    "depformer_gating": "silu",
+    "depformer_pos_emb": "none",
+    "depformer_weights_per_step": true,
+    "delays": [
+        0,
+        0,
+        1,
+        1,
+        1,
+        1,
+        1,
+        1,
+        1,
+        0,
+        1,
+        1,
+        1,
+        1,
+        1,
+        1,
+        1
+    ]
+}