Evals and production monitoring — knowing your AI feature still works
Lesson 3 of 5 in AI for Software Engineers — Co-pilots, RAG & Agents · 24 minutes
AI features that work in demos often break in production. This lesson covers evals (offline and online) and the production monitoring loop that shows quality, cost, latency and user outcomes.
What you'll learn
- Explain the difference between offline and online evals
- Build an eval set with happy, hard, failure and adversarial cases
- Name the four metrics every production AI feature should track
- Describe the model-update and data flywheel routines
From the lesson
Most AI features that work in a demo do not survive production. The reasons are predictable: untested edge cases, hallucinations that reach customers, latency spikes and cost overruns. The fix is not magic. It is rigorous evals and steady monitoring. This lesson explains both.
1. Why evals matter
Traditional tests check deterministic behaviour. LLM features are different: the same input can give different outputs, edge cases are endless, and quality can drop silently when a model or the data changes. Evals ('evaluations') measure whether your feature does what you want, on the inputs that matter.
Offline evals run before deployment. You keep a fixed set of inputs with expected outputs (from…