AI Engineering
Build the eval harness before you build the feature
By Pavan Jegurupati, Founder, InferLearn
The ordering problem
Every team says they will add evals later. Later arrives, the surface area has tripled, and nobody can name what "better" means any more.
What a harness actually needs
A useful harness is smaller than people expect. Three things carry most of the weight: a frozen set of representative inputs, a scoring function you trust enough to argue with, and a gate that fails a pull request.
Why LLM-as-judge keeps disappointing you
Judges inherit the biases of the model behind them. They reward fluency and punish terseness, which is exactly backwards for most production tasks.
Where to start on Monday
Take the last ten bugs someone filed against your AI feature. That is your golden set. It is not elegant, but it is real, and real beats elegant.
Taggedevaluationllmtesting
