chapter 58

fine-tune a small specialist

one repeated job, a reviewed example set, a QLoRA adapter, and an eval receipt that beats prompt-only on consistency — without claiming the base model got generally smarter.

8 live lessons · 28 live steps · 65 XP

One job, one adapter, a receipt

You already know how to write a stronger brief and how to retrieve the documents the model does not have. Some production jobs still wobble after both of those levers: the same CI log gets labeled flake on Tuesday and needs-human on Thursday, even though the acceptance test never changed. Fine-tuning a small specialist is the lever for that wobble. A specialist is a narrow behavior — voice, format, or a three-way label — trained into a LoRA adapter on top of a small instruct model. The base weights stay shared. The adapter is the job.

This chapter ships one living artifact: specialist-adapter-receipt/ for a CI-triage labeler that marks a failing job as flake, product bug, or needs-human. That is the same triage surface you met in agent studio, now treated as a consistency problem instead of a tool-contract problem.

The framing is adapted from Sai Rahul (@sairahul1) — How To Fine-Tune a Small LLM on Your Own Data (Full Guide) — into PromptDojo's judgment-and-receipt bar. Read his piece if you want the wider how-to. The unlock here is the gate, the splits, and the eval table, not a transcription of that writeup.

Use this chapter when

One repeated job stays inconsistent under a strong prompt.
Retrieval already won or lost on the facts. The leftover problem is how the model writes or labels, not what it knows.
You can review a few hundred examples and spare a T4-class GPU hour.

Skip the adapter when the brief is still weak, when the missing piece is a document that changes, when nobody will read the training rows, or when you want a generally smarter assistant. Fine-tune will not deliver that last one.

What the lab actually grades

The in-browser drills run in ordinary Python. They grade the config and the receipt, not a live GPU train. The Colab notebook under specialist-adapter-receipt/ is the T4 path: Qwen2.5-1.5B-Instruct (or a peer in the 0.5–3B class), QLoRA via PEFT + bitsandbytes, TRL SFTTrainer, adapter export. Free-tier unlock is a fixture receipt plus a correct judgment payload. Running it live on a T4 is the stretch.

By the end you can:

Pick prompt, RAG, or fine-tune from a failure mode, and hold the run when someone fine-tunes a knowledge gap.
Name the specialist in one sentence with acceptance tests the CI-triage labels can fail.
Build a human-reviewed example set with company- or doc-level splits, so a random row split cannot leak.
Audit a QLoRA payload and emit an adapter receipt the notebook (or the fixture) can attach to.
Score prompt-only vs RAG vs FT on the same held-out cases, and refuse hallucinated numbers or contaminated eval.
Describe how vLLM LoRA (or Modal/RunPod) loads the adapter while the base stays shared.

lessons in this chapter

01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps 01the specialist, not the genius3 steps

02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps 02judgment gate — prompt, rag, or fine-tune4 steps

03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps 03name the narrow job3 steps

04data as the product4 steps 04data as the product4 steps 04data as the product4 steps 04data as the product4 steps 04data as the product4 steps 04data as the product4 steps 04data as the product4 steps 04data as the product4 steps 04data as the product4 steps 04data as the product4 steps 04data as the product4 steps

05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps 05lab — ship the adapter3 steps

06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps 06eval receipt2 steps

07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps 07deploy adapters3 steps

08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps 08fail-case gallery6 steps