Domain fine-tuning

RAG gives Cradle the facts (prices, specs, clauses). Fine-tuning gives it the style and reasoning patterns of a narrow domain — legal, medical, construction, accounting, and so on. The two features work together.

The training pipeline uses Python only for the LoRA steps. The Cradle runtime itself remains Python-free.

What you can train on a MacBook Pro M3 Pro 18 GB

With 36 GB or more of RAM you can use Qwen2.5-7B with rank 16.

Step 1 — build a dataset

Collect source documents in one folder:

~/corpus/legal/
├── snippets/
│   ├── contract-terms.md
│   └── court-practice.pdf
├── gost/
│   └── civil-code-sections.pdf
└── examples/
    └── faq-operator.md

Run the dataset builder:

pnpm build-dataset \
  --files ~/corpus/legal \
  --out "~/Library/Application Support/cradle/finetune-datasets/legal-v1" \
  --mode qa \
  --model "~/Library/Application Support/cradle/models/qwen3-8b-q4.gguf" \
  --qa-per-chunk 2 \
  --language ru

• --mode qa — synthesizes Q&A pairs from each chunk with a local model. Slow but produces an instruction-tuning dataset.
• --mode raw — fast; emits { "text": "<chunk>" } for continued pretraining.
• --model — path to a downloaded chat GGUF used for Q&A synthesis. Qwen3 8B is recommended.

Output:

legal-v1/
├── train.jsonl
├── valid.jsonl
└── meta.json

Inspect 5–10 pairs manually. Questions should sound natural and answers must not contain made-up numbers. If the synthesizer hallucinates, try a larger model or --mode raw.

Step 2 — install MLX

Python is needed once on the training machine. Use uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

uv venv cradle-finetune --python 3.11
source cradle-finetune/bin/activate
uv pip install -r /path/to/cradle/tools/finetune/requirements.txt

python -c "import mlx_lm; print(mlx_lm.__version__)"
# expect ≥ 0.31.0

Step 3 — train a LoRA

Copy and edit the example config:

cp tools/finetune/lora_config.example.yaml ~/lora-legal.yaml
# edit: data, adapter_path, base model

Run training:

python -m mlx_lm.lora --config ~/lora-legal.yaml

Watch the log:

Iter 1:   Val loss 2.134, Val took 4.2s
Iter 10:  Train loss 1.987, It/sec 0.85, Tokens/sec 420
...
Iter 750: Train loss 1.234, Val loss 1.456
Saved final adapter weights to adapters-raw/legal-v1/adapters.safetensors

Checkpoints:

• Validation loss should decrease. If it stalls or rises, reduce iters.
• It/sec of 0.5–2 on an M3 Pro is normal. Below 0.2 means RAM is tight: reduce batch_size or enable grad_checkpoint.
• OOM mid-run: reduce num_layers from 16 to 8, or rank from 16 to 8.

Step 4 — export to GGUF

Clone and build llama.cpp for the conversion tools:

git clone https://github.com/ggml-org/llama.cpp ~/repos/llama.cpp
cd ~/repos/llama.cpp
cmake -B build && cmake --build build --config Release -j

Merge the LoRA into the base model and quantize:

cd /path/to/cradle
BASE_MODEL=Qwen/Qwen2.5-3B-Instruct \
ADAPTER_PATH="~/Library/Application Support/cradle/adapters-raw/legal-v1" \
OUT_DIR=/tmp/legal-v1-gguf \
LLAMA_CPP=~/repos/llama.cpp \
QUANT=Q4_K_M \
./tools/finetune/export_gguf.sh

Result: /tmp/legal-v1-gguf/model-Q4_K_M.gguf (~1.9 GB for a 3B base).

Step 5 — import into Cradle

Option A — as a new model

1. Move the GGUF into the Cradle models directory: ~/Library/Application Support/cradle/models/legal-v1.gguf.
2. In the desktop app: Models → Add Custom URL.
3. URL: file:///Users/aki/Library/Application%20Support/cradle/models/legal-v1.gguf
4. Role: general, context: 16384 (or matching the base model).
5. In Agents → Edit, assign this model to the relevant agent.

Option B — runtime adapter (experimental)

If you have a separate GGUF adapter:

1. Adapters → Import adapter in Cradle.
2. Select the base model by catalogId.
3. Scale: 1.0.

Then assign the adapter in Agents → Edit → Domain adapter. The base model stays the same.

⚠️ Verified only with node-llama-cpp v3.4, which supports LoRA through createContext({ lora: ... }). MLX-LM does not export GGUF adapters directly, so Option A is recommended until you need multiple independent adapters.

Troubleshooting

• OOM at iter 1 — base model too large. On 18 GB max: 3B with rank 16, or 7B with rank 8 + grad checkpoint.
• Validation loss does not drop — too few examples (less than 100) or wrong format. Check each line of train.jsonl is valid JSON with a messages field.
• Fused GGUF is 16 GB — quantization was skipped. Verify llama-quantize ran and OUT_DIR contains model-Q4_K_M.gguf.
• Agent still answers like the base model — adapter scale too low; try 1.5, or increase iters.

Next steps

Pair the fine-tuned adapter with RAG: the adapter sets the domain language, RAG supplies the current facts.