MLX LM: fine-tune Qwen2.5 on a Mac with QLoRA

This tutorial follows a local LLM fine-tuning workflow with Apple MLX, MLX LM and Qwen2.5-3B-Instruct. The presenter records the run on a MacBook Pro with an M2 Pro and 16 GB of unified memory. He says the commands also apply to a Mac mini M5 Pro; the recorded results are not M5 Pro benchmarks.

Setup uses Python 3.12, MLX 0.32.3 and MLX LM 0.32.0. The workflow covers 4-bit conversion, baseline generation and JSONL chat data before training LoRA adapters. The 20-example dataset is a pipeline test: 16 training examples, two validation examples and two test examples. The presenter reports that 20 iterations took about 16 seconds and peaked near 2.5 GB of memory.

Checkpoint comparison exposes a practical limitation. The step-20 adapter achieved lower test loss but omitted authorization from an answer about authentication and authorization. The step-10 adapter explained both terms in 47 tokens, compared with the baseline's 151. The lesson is to inspect answers alongside loss scores.

The tutorial also covers adapter fusion and a local FastAPI chat route, including an async route fix for an MLX thread error. That API is a single-user demo without streaming, authentication or a queue. Ollama and GGUF export are explained rather than demonstrated because network access was blocked. For changing documents, the presenter recommends retrieval; for tone changes, he suggests trying a system prompt before training.