- The production loop
- Serving a model
- Version everything
- Rollouts without heroics
- Monitoring & drift
- LLMOps extras
- Interview questions
The production loop
train → evaluate → package → deploy → predict
↖—— monitor / collect labels / retrain ——↗
Notebooks are for discovery. Production needs reproducible training, a scorable artifact, a serving path, and feedback. If any leg is missing, you don’t have MLOps — you have a demo.
Serving a model
- Batch — score overnight tables (reports, campaigns)
- Online API — low-latency request/response (fraud check, ranking)
- Streaming — events on a bus; score continuously
A minimal online path: serialize model (and preprocessing!) → FastAPI `/score` → validate schema → return score + model version id. Keep preprocessing identical to training.
Version everything
code commit · data snapshot / query · model artifact metrics report · prompt text · retrieval index build id
If you can’t answer “which model/prompt served this request?”, you can’t debug or roll back. Log request ids with those versions.
Rollouts without heroics
- Shadow — new model scores in parallel; no user impact
- Canary — small % of traffic; compare metrics; promote or revert
- Blue/green — two environments; flip traffic when healthy
Same discipline for prompt changes — they are production config, not chat scratchpads.
Monitoring & drift
Models rot when the world changes.
- Data drift — input distributions shift
- Concept drift — input→label relationship shifts
- Performance drift — live metric drops (when labels arrive late, use proxies carefully)
- Ops signals — latency, errors, CPU/GPU, queue depth, cost
Alert on schema breaks and sudden null spikes first — they kill more systems than subtle KL-divergence dashboards.
LLMOps extras
- Prompt + tool schema versioning
- Eval harness on every change (offline set)
- Token/cost budgets and caches
- Safety filters / PII redaction
- Retrieval index freshness monitors
- Trace logs for tool calls (without leaking secrets)
For FraudWatch → FilingPilot: score API first, then a RAG service with citations and an eval CSV you re-run on every prompt/index bump.
Interview questions
Also in the interview bank.
How do you know a model is rotting in production?
Watch input/prediction distributions, delayed labels or proxies, latency/errors, and business KPIs. Compare to a shadow baseline. For LLMs, also track eval regressions and retrieval quality.
What do you log for a production ML / LLM service?
Request id, model/prompt/index versions, latency, token/cost, redacted inputs/outputs, retrieval ids, user feedback.
Why canary a model rollout?
Limit blast radius. Compare champion vs challenger on live metrics before full promote — same for prompts.
What is training/serving skew?
Features computed differently in training vs production, so the live model sees a different world than it learned. Share one pipeline.