Optimising is turning a lucky prompt into a reliable one. You measure the output against criteria you wrote down, change one thing, measure again — and keep the phrasing that wins. Without measurement, prompt tuning is superstition.
Measure before you tune
- Write the criteria first. Correctness, format, tone, length — the same criteria you should have put in the prompt.
- Build a tiny evaluation set. Ten to twenty real inputs with known-good answers catch far more regressions than intuition.
- Score consistently. Exact match or a rubric applied the same way each run; a second model can grade if the rubric is explicit.
- Change one variable. Prompt wording, context, model, temperature — one at a time, or you learn nothing.
Levers, cheapest first
- Sharper wording. Numbers instead of quantifiers, exclusions stated — see Precision.
- Better context. Retrieval that pulls the right passage beats a longer prompt full of noise.
- Examples in the prompt. A couple of worked examples often outperform paragraphs of instruction.
- A different model size. Route easy cases to a small model and hard ones to a frontier model; measure both.
- Fine-tuning, last. Only once the prompt is stable and you have data — see Fine-Tuning & Alignment.
Example
Vague
Classify these support tickets.
Better
Classify each ticket as exactly one of: BILLING, BUG, HOWTO, OTHER.
Output JSON: {"id": "", "label": "", "confidence": 0.0-1.0}. No other text.
If the ticket fits none of the labels, use OTHER with confidence 0.
Examples:
"card declined twice" -> BILLING
"export button spins forever" -> BUG