Write the evals before you write the prompts
If you can't measure whether the model is doing the right thing, you're not engineering — you're guessing.
The fastest way to build an AI feature that quietly breaks is to start with the prompt. You tweak, it looks better, you ship — and you have no idea whether the next tweak made it worse.
Evals are the test suite for behaviour
In normal software, you wouldn't merge a change without tests. AI features are no different, except the "behaviour" is fuzzier. An eval is just a labelled set of inputs and a way to score outputs against what "good" means for your task.
Write them first, and the prompt becomes an optimization target instead of a vibe. You can compare two approaches objectively, catch regressions, and know when you're actually done.
What a minimal eval looks like
- Collect 30–50 real inputs that cover the easy, hard, and adversarial cases.
- Define a score — exact match, a rubric, or a judge model with a clear rubric.
- Run it on every change and track the number.
That's it. You don't need a framework to start; you need the discipline to measure before you tune.
The payoff
Once evals exist, prompt engineering stops being folklore. You ship changes because the number went up, not because the latest output happened to look nice — and that's the difference between a demo and a system.
More insights
- What observability actually means for AI agentsLogs, traces, and metrics for systems that make decisions — so when an agent does something strange, you can see why.
- Building a RAG system that stays accurate as it growsRetrieval quality degrades quietly as your corpus changes. Here's how to catch it before your users do.