The most expensive MLOps problems are often ordinary: stale data, unreproducible features, an invisible schema change, a failed schedule, or a model nobody can roll back safely.
A broad maturity assessment can produce a long list of best practices while missing the path that actually hurts the business. I prefer to select one training, inference, or data workflow and trace it from input to operational outcome.
Choose the failure that matters
The review should start with an observable symptom: a forecast arrives late, a batch silently skips records, a deployment cannot be reproduced, an online score diverges from offline evaluation, or a downstream team no longer trusts the output.
That symptom defines the first boundary. The task is to reconstruct how data, code, artifacts, schedules, infrastructure, and ownership combine to produce it.
Trace six layers end to end
- Data contract: sources, schema, freshness, completeness, late arrivals, and ownership.
- Transformation: feature logic, versioning, leakage risk, and training-serving consistency.
- Reproducibility: code, configuration, environment, randomness, lineage, and artifact identity.
- Orchestration: schedules, dependencies, retries, idempotency, backfills, and failure visibility.
- Deployment: promotion, compatibility, rollback, resource limits, and access controls.
- Operation: monitoring, alert ownership, recovery, user feedback, and the decision affected.
Distinguish silent failures from visible failures
A visible failed job is often safer than a successful-looking job that produces incomplete or stale output. The review should ask which defects can pass every infrastructure check while corrupting the business result.
Reliability covers the running pipeline, the trustworthy result and the recovery path.
Make recovery part of the definition of done
For the selected workflow, the team should know how to identify the last valid input, reproduce the affected run, isolate the bad artifact, backfill safely, and verify the repaired output. A recovery path stored in one person’s memory remains fragile.
End with the smallest credible remediation
The first fix may be a schema check, a lineage record, an idempotent write, a deployment manifest, or an alert tied to data freshness. A new platform is rarely the automatic answer.
A useful review separates the critical correction from worthwhile later improvements, attaches an acceptance test to each, and identifies the owner who will operate the result.