File size: 4,411 Bytes
7f9dfed | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | # Plant Model And Training How-To
This answers three common questions:
1. Does Plant Discovery use an LLM?
2. Does it use a specially trained plant model?
3. How do we train and then use one?
## Current Model Modes
Plant Discovery has three explicit modes.
| Mode | Command | Uses an LLM/VLM? | Trained specifically for this app? |
| --- | --- | --- | --- |
| Demo | `python -m plant.app --model-mode demo --port 7861` | No | No |
| OpenBMB zero-shot | `python -m plant.app --model-mode openbmb --port 7861` | Yes, MiniCPM-V | No, uses base OpenBMB model |
| Fine-tuned adapter | `python -m plant.app --model-mode finetuned --port 7861` | Yes, MiniCPM-V plus adapter | Yes, after you train/publish an adapter |
The app defaults to `openbmb` mode. The `--no-model` flag is an alias for demo mode and exists for
tests, screenshots, and machines without GPU/model dependencies.
## What Runs Today
- Demo mode is fully verified locally.
- OpenBMB mode is implemented but requires optional dependencies and model weights.
- Fine-tuned mode is implemented as an adapter-loading path, but there is no real adapter yet.
This means the app can use a real OpenBMB VLM, but this workspace has not installed the heavy
`plant/requirements.txt` dependencies or downloaded model weights.
## Step 1 - Use OpenBMB MiniCPM-V Zero-Shot
Install optional plant dependencies:
```powershell
.venv\Scripts\python.exe -m pip install -r plant\requirements.txt
```
Run the app with the real OpenBMB model path:
```powershell
.venv\Scripts\python.exe -m plant.app --model-mode openbmb --port 7861
```
Open `http://127.0.0.1:7861`, upload a public-safe plant image, and click `Identify plant`.
Expected behavior:
- The UI shows model status with `uses_llm: true`.
- The first identify action loads `openbmb/MiniCPM-V-4.6`.
- If confidence is low and auto-thinking is enabled, the app can try
`openbmb/MiniCPM-V-4.6-Thinking`.
- Output is parsed into the `PlantID` JSON schema.
## Step 2 - Collect Corrections
Use the app normally:
1. Upload plant image.
2. Identify.
3. Add a human correction when the model is wrong.
4. Save correction.
5. Open `Corrections`.
6. Click `Export training JSONL`.
The export path is:
```text
data/plant_training.jsonl
```
Do not train on one or two examples. The current minimum recommendation is 30 corrected examples,
and more is better.
## Step 3 - Plan Training
Generate a non-executing training plan:
```powershell
.venv\Scripts\python.exe scripts\plan_plant_training.py --corrected-examples 30
```
This prints:
- dependency availability,
- SWIFT command preview,
- LLaMA-Factory command preview,
- publish commands,
- command to run the trained adapter.
The Gradio UI also exposes the same non-executing plan from the `Corrections` tab.
## Step 4 - Train Locally
Preferred path for MiniCPM-V vision LoRA is SWIFT or another tool that supports multimodal LoRA.
The generated SWIFT command looks like this:
```powershell
swift sft --model openbmb/MiniCPM-V-4.6 --dataset data/plant_training.jsonl --lora_rank 16 --num_train_epochs 3 --per_device_train_batch_size 4 --gradient_accumulation_steps 4 --learning_rate 2.0e-4 --freeze_vit true --output_dir checkpoints/plant_lora
```
Run it only after:
- optional dependencies are installed,
- GPU memory is sufficient,
- exported JSONL has been reviewed,
- private images/notes are removed,
- you accept that training may take time and disk space.
## Step 5 - Publish Or Configure The Adapter
After training, publish the adapter or keep it local. If publishing:
```powershell
huggingface-cli repo create your-username/minicpm-v46-plant-lora --type model
huggingface-cli upload your-username/minicpm-v46-plant-lora checkpoints/plant_lora .
```
Then edit `plant/models.yaml`:
```yaml
plant_vlm_finetuned:
base_model: openbmb/MiniCPM-V-4.6
adapter_id: your-username/minicpm-v46-plant-lora
```
Run:
```powershell
.venv\Scripts\python.exe -m plant.app --model-mode finetuned --port 7861
```
## Important Honesty Rule
Do not claim Plant Discovery uses a specially trained model until:
- a real adapter exists,
- it is configured in `plant/models.yaml`,
- `--model-mode finetuned` has been run,
- evaluation shows it improves over the base OpenBMB model.
Until then, the honest claim is:
> Plant Discovery uses OpenBMB MiniCPM-V zero-shot, with a correction loop and documented path to
> fine-tune a plant adapter.
|