Update README.md

README.md

@@ -1,157 +1,22 @@
-# AME Finetuned Moshi Checkpoint
+# KAME
 
-This checkpoint contains a cleaned KAME finetuned Moshi model.
+## Links
 
-Files:
+* **Paper**: [arXiv: 2510.02327](https://arxiv.org/abs/2510.02327) (ICASSP 2026)
+* **Inference code**: [SakanaAI/kame](https://github.com/SakanaAI/kame)
+* **Finetuning code**: [SakanaAI/kame_finetune](https://github.com/SakanaAI/kame_finetune)
 
-- `model.safetensors`: model weights
-- `moshi_lm_kwargs.json`: model architecture/config for this checkpoint
+## Abstract
+Real-time speech-to-speech (S2S) models excel at generating natural, low-latency conversational responses but often lack deep knowledge and semantic understanding. Conversely, cascaded systems combining automatic speech recognition, a text-based Large Language Model (LLM), and text-to-speech synthesis offer superior knowledge representation at the cost of high latency, which disrupts the flow of natural interaction.
+This paper introduces a novel hybrid architecture that bridges the gap between these two paradigms. Our framework processes user speech through an S2S transformer for immediate responsiveness while concurrently relaying the query to a powerful back-end LLM. The LLM’s text-based response is then injected in real time to guide the S2S model’s speech generation, effectively infusing its output with rich knowledge without the full latency penalty of a cascaded system.
+We evaluated our method using a speech-synthesized variant of the MT-Bench benchmark that consists of multi-turn question-answering sessions. The results demonstrate that our system substantially outperforms a baseline S2S model in response correctness, approaching that of a cascaded system, while maintaining a latency on par with the baseline.
 
-Notes:
 
-- The weights are stored in fp32.
-- Inference in the server scripts used in this repo typically runs in bfloat16 by default unless a different dtype is explicitly requested.
-- This release does not include a `config.json`.
-- When loading this checkpoint, pass `moshi_lm_kwargs.json` explicitly as `--config-path`.
-- Tokenizer and Mimi assets are expected to come from `kyutai/moshiko-pytorch-bf16`.
+## Base Model
 
-## Quick Start
+The front-end S2S model is based on Moshi: a speech-text foundation model for real-time dialogue, a full-duplex speech-to-speech foundation model for real-time dialogue. ([arxiv.org][1])
 
-Example with `server_oracle.py`:
+[1]: https://arxiv.org/abs/2410.00037?utm_source=chatgpt.com "Moshi: a speech-text foundation model for real-time dialogue"
 
-```bash
-cd /path/to/moshi
 
-CHECKPOINT_DIR=/path/to/checkpoint
-MODEL_WEIGHT=$CHECKPOINT_DIR/model.safetensors
-MODEL_CONFIG=$CHECKPOINT_DIR/moshi_lm_kwargs.json
-HF_REPO=kyutai/moshiko-pytorch-bf16
 
-uv run python -m moshi.server_oracle \
-  --hf-repo "$HF_REPO" \
-  --moshi-weight "$MODEL_WEIGHT" \
-  --config-path "$MODEL_CONFIG" \
-  --device cuda
-```
-
-Make sure any required environment variables for your local setup are already set
-before launching the server.
-
-Then open:
-
-- `http://localhost:8998`
-
-If the GPU machine is remote and you need a public URL, add:
-
-- `--gradio-tunnel`
-
----
-
-Copy everything above this line into the Hugging Face repo `README.md`.
-
-## Internal Notes
-
-This section is for internal team use and can stay in the local copy of this directory.
-
-Current local directory:
-
-- `/home/manatoyaguchi_sakana_ai/kame_finetune_dev/output/moshiko-finetuned_all_en_oracle_emb_separate_copy_single_multi/step_6000_fp32_cleaned`
-
-### Local Files
-
-- `model.safetensors`: model weights
-- `moshi_lm_kwargs.json`: model architecture/config used for this checkpoint
-
-### Important
-
-- This checkpoint is currently shared as `model.safetensors + moshi_lm_kwargs.json`.
-- There is no `config.json` in this directory.
-- When loading this checkpoint, pass `moshi_lm_kwargs.json` explicitly as `--config-path`.
-- Tokenizer and Mimi assets are expected to come from `kyutai/moshiko-pytorch-bf16`.
-- The weights are stored in fp32.
-- In `server_oracle.py` and the evaluation scripts used in this repo, inference runs in bfloat16 by default unless a different dtype is explicitly requested.
-
-### Quick Start
-
-Use this checkpoint together with the base Moshi repo:
-
-```bash
-cd /home/manatoyaguchi_sakana_ai/kame_finetune_dev
-
-MODEL_DIR=$PWD/output/moshiko-finetuned_all_en_oracle_emb_separate_copy_single_multi/step_6000_fp32_cleaned
-MODEL_WEIGHT=$MODEL_DIR/model.safetensors
-MODEL_CONFIG=$MODEL_DIR/moshi_lm_kwargs.json
-HF_REPO=kyutai/moshiko-pytorch-bf16
-```
-
-### Main Use Case: UI Inference with `server_oracle.py`
-
-The main intended use of this checkpoint is interactive inference through
-`kame_dev/moshi/moshi/server_oracle.py`.
-
-Start the server like this:
-
-```bash
-cd /home/manatoyaguchi_sakana_ai/kame_dev/moshi
-
-set -a
-source ../.env
-set +a
-
-MODEL_DIR=/home/manatoyaguchi_sakana_ai/kame_finetune_dev/output/moshiko-finetuned_all_en_oracle_emb_separate_copy_single_multi/step_6000_fp32_cleaned
-MODEL_WEIGHT=$MODEL_DIR/model.safetensors
-MODEL_CONFIG=$MODEL_DIR/moshi_lm_kwargs.json
-HF_REPO=kyutai/moshiko-pytorch-bf16
-
-uv run python -m moshi.server_oracle \
-  --hf-repo "$HF_REPO" \
-  --moshi-weight "$MODEL_WEIGHT" \
-  --config-path "$MODEL_CONFIG" \
-  --device cuda
-```
-
-Then open:
-
-- `http://localhost:8998`
-
-If the GPU machine is remote and you need a public URL, add:
-
-- `--gradio-tunnel`
-
-Notes:
-
-- `moshi_lm_kwargs.json` must be passed explicitly as `--config-path`.
-- `kyutai/moshiko-pytorch-bf16` provides the base tokenizer and Mimi assets.
-- By default this runs in `bfloat16`. Add `--half` if you explicitly want `float16`.
-
-### MT-Bench Evaluation (For Internal Use Only)
-
-```bash
-cd /home/manatoyaguchi_sakana_ai/kame_finetune_dev
-
-set -a
-source ../kame_dev/.env
-set +a
-
-INPUT_DIR=/home/shared/sokuroki_sakana_ai/moshi/yaguchi/mt_bench
-MODEL_DIR=$PWD/output/moshiko-finetuned_all_en_oracle_emb_separate_copy_single_multi/step_6000_fp32_cleaned
-MODEL_WEIGHT=$MODEL_DIR/model.safetensors
-MODEL_CONFIG=$MODEL_DIR/moshi_lm_kwargs.json
-HF_REPO=kyutai/moshiko-pytorch-bf16
-
-uv run python -m experimental.mt_bench.run_model_eval \
-  --input-dir "$INPUT_DIR" \
-  --output-dir "$PWD/data/mt_bench_kame_step6000_smoke" \
-  --moshi-weight "$MODEL_WEIGHT" \
-  --config-path "$MODEL_CONFIG" \
-  --hf-repo "$HF_REPO" \
-  --backend openai \
-  --question-ids 106 \
-  --limit 1
-```
-
-### Additional Notes
-
-- For shared use, the most important files are `model.safetensors` and `moshi_lm_kwargs.json`.
-- If you upload this directory to Hugging Face as-is, downstream users should download both files and pass `moshi_lm_kwargs.json` explicitly when loading.
-- `config.json` is not included here. This checkpoint is currently shared in a "documented artifact" style rather than as a standalone self-describing HF repo.