MLOps: Running AI in Production Without Losing Sleep
MLOps: Running AI in Production Without Losing Sleep
TL;DR: A model that performs well in a notebook is not a product. It is a hypothesis. MLOps is the engineering discipline that turns that hypothesis into a production system — one that runs reliably, degrades gracefully, and improves over time. This article covers the full ML lifecycle, why CI/CD for models is fundamentally different from software CI/CD, how to detect and respond to model drift, the four pillars of AI observability, and how to build an MLOps pipeline your team can actually maintain without a dedicated platform engineering org.
Introduction
There is a gap that every AI initiative eventually hits. It is the gap between a model that works in a controlled environment and a model that works in production, with real users, real data, and real consequences.
In the lab, the data is clean. The questions are known. The model is evaluated against a test set that was carefully curated. If something breaks, you fix it and run the experiment again. There is no urgency. There is no user. There is no cost to being wrong.
In production, none of that holds. The data is messy, incomplete, and constantly changing. Users ask questions the model was never trained on. The distribution shifts — sometimes gradually, sometimes overnight. A model that was accurate last month starts making errors that nobody notices until a customer complains or a metric drops. And unlike a software bug, which either crashes or does not, model failures are often silent. The system keeps running. It keeps producing outputs. The outputs are just wrong.
This is the gap that MLOps exists to bridge. MLOps — a portmanteau of Machine Learning and Operations — is the set of practices, tools, and disciplines that take a model from prototype to production and keep it running. It is not a single tool. It is not a platform you buy. It is the engineering culture and pipeline that makes AI a reliable, maintainable part of your business rather than a science experiment that works until it does not.
If your AI initiative stops at "the model works in our notebook," you do not have an AI product. You have a demo. MLOps is how you get from the demo to the product.
What MLOps actually means (beyond the buzzword)
MLOps borrows from DevOps — the philosophy that transformed software delivery by automating the path from code to production. But ML introduces complications that make traditional DevOps insufficient on its own.
Software is deterministic. The same input produces the same output. If a build passes in CI, it will behave the same way in production. Version control tracks every change. Rollbacks are clean. Testing is well-understood: given an input, assert the output matches the expected value.
Machine learning is probabilistic. The same input can produce different outputs depending on the model's state, the retrieved context, and inherent stochasticity. A model's behaviour depends on the data it was trained on — data that changes over time. Version control does not just track code; it has to track data, model weights, training configurations, and evaluation metrics. Rollbacks are not clean because the production environment itself may have shifted. And testing is fundamentally harder: you are not asserting an exact output, you are measuring whether the model's statistical behaviour meets an acceptable threshold across a distribution of inputs.
This is why MLOps is a distinct discipline, not a subset of DevOps. The practices overlap — automation, CI/CD, monitoring, infrastructure-as-code — but the ML lifecycle has properties that require specialised tooling and workflows. Treating a model like a software component will get you to deployment. It will not keep you running.
The ML lifecycle: four stages that never stop
The ML lifecycle is not a linear pipeline. It is a loop. Each stage feeds into the next, and the final stage feeds back into the first. A model that is deployed and never revisited is a model that is decaying.
Stage 1: Training
Training is where the model learns. It is the stage most teams focus on because it is the most visible — it produces the artefact (the model) that everything else depends on. But training is only as good as the data it uses, and the data is where most training problems originate.
The training stage includes data preparation — selecting, cleaning, and formatting the training data. It includes model selection — choosing the architecture, the hyperparameters, and the training configuration. And it includes evaluation — measuring the model's performance against a held-out test set before it is deemed good enough to deploy.
The critical practice at this stage is reproducibility. Every training run should be fully traceable: what data was used, what version of the code, what hyperparameters, what random seed. Without reproducibility, you cannot diagnose why a model's behaviour changed. You cannot roll back to a known-good state. You cannot compare two versions of a model fairly because you do not know whether a performance difference is due to the model or due to the data it was trained on.
Stage 2: Deployment
Deployment is the stage where the model moves from a training environment to a production environment where it can serve predictions. This sounds simple — package the model, expose an endpoint, route traffic to it. In practice, deployment involves decisions that determine whether the model will be useful in production.
Will the model serve predictions in real time, where a user is waiting for a response? Or will it run in batch mode, processing large datasets on a schedule? Real-time serving requires low-latency infrastructure — model optimisation, GPU allocation, caching strategies. Batch serving is simpler but introduces a freshness gap: the prediction was generated hours ago, and the world may have changed since then.
Will the model be deployed alongside a fallback? A new model, no matter how well it tested, should not replace the existing model in a single cutover. The standard practice is a canary deployment — route a small percentage of traffic to the new model, monitor its behaviour, and gradually increase the traffic if the metrics hold. This is the ML equivalent of a feature flag, and it prevents a bad model from taking down your production system.
Stage 3: Monitoring
Monitoring is where most teams fail. A model that was deployed and never monitored is a model that is silently degrading. The data distribution shifts. The input patterns change. The model's accuracy erodes. And because the system does not crash — it keeps producing outputs — nobody knows until the damage is visible.
Monitoring an ML system is different from monitoring a software system. Software monitoring tracks infrastructure metrics: CPU, memory, latency, error rates. These are necessary for ML systems too, but they are not sufficient. An ML system can have perfect infrastructure metrics and still be producing wrong answers because the model's behaviour has drifted.
ML monitoring needs to track model-specific signals: prediction distribution (is the model producing the same mix of outputs it was at deployment?), input distribution (are the inputs the model is receiving in production similar to the data it was trained on?), and outcome metrics (are the model's predictions still accurate, measured against ground truth that arrives later). These signals tell you whether the model is still healthy — and when it is not.
Stage 4: Retraining
Retraining is the stage that closes the loop. When monitoring reveals that a model's performance has degraded — or when new data becomes available that could improve it — the model is retrained and redeployed.
Retraining is not as simple as running the same training pipeline with more data. The new data may have a different distribution than the old data. Adding it to the training set without care can introduce bias or degrade performance on previously well-handled cases. The retraining decision needs to be informed by analysis: what changed, why did it change, and what is the right response?
The retraining pipeline should also be automated to the extent possible. A system that requires a data scientist to manually trigger retraining, evaluate the new model, and deploy it is a system that will not be maintained. The goal is a pipeline that detects degradation, triggers retraining, evaluates the new model against the old one, and deploys only if the new model is better — with a human in the loop for the final approval.
CI/CD for ML: why it is different from software CI/CD
In software engineering, CI/CD is a solved problem. You write code, push it to a repository, and a pipeline automatically builds, tests, and deploys it. The pipeline is deterministic — the same code produces the same build. Testing is binary — the test either passes or fails. Deployment is atomic — the new version replaces the old one.
ML CI/CD is fundamentally different because the artefact being deployed is not just code. It is a combination of code, data, model weights, and configuration. A change in any one of these can change the model's behaviour. And unlike code, data and model weights are not text files that can be diffed in a pull request.
An ML CI/CD pipeline needs to handle three types of changes. The first is a code change — the model serving code, the preprocessing logic, or the training script. This is closest to traditional CI/CD and can be handled with standard tools. The second is a data change — new training data, updated feature definitions, or a shift in the data schema. This requires a data validation step that checks whether the new data is consistent with expectations before it enters the training pipeline. The third is a model change — a retrained model with new weights. This requires an evaluation step that compares the new model against the current production model on a shared benchmark before approving the deployment.
The evaluation step is the critical difference. In software CI/CD, tests are pass-or-fail. In ML CI/CD, evaluation is a statistical comparison. The new model may be better on some metrics and worse on others. It may be better overall but regress on a specific segment of users. The decision to deploy is not binary — it is a judgement call informed by metrics, segment analysis, and business context.
This is why ML deployment needs guardrails that go beyond software CI/CD. A shadow deployment runs the new model alongside the production model, serving the production model's outputs to users while logging what the new model would have produced. This lets you compare real-world behaviour without risking the user experience. A canary deployment routes a small percentage of traffic to the new model and monitors for regressions. An automatic rollback triggers if the new model's metrics drop below a threshold within a defined window.
These patterns exist because the question "is this model safe to deploy?" is harder to answer than "does this code pass tests?" The CI/CD pipeline for ML is not just automation — it is a risk management framework.
Model drift: detection and response
Model drift is the gradual degradation of a model's performance over time. It is not a bug. It is the natural consequence of deploying a static model in a dynamic world. Understanding drift — and building the systems to detect and respond to it — is one of the most important aspects of MLOps.
There are two types of drift, and they require different responses.
Data drift (also called covariate shift) occurs when the input distribution changes. The model was trained on data from one context, and production data comes from a different context. A model trained to classify support tickets during a product's early adoption phase may receive very different tickets when the product matures and the user base grows. The inputs have shifted, even if the model's internal logic has not changed. Data drift is detectable by comparing the statistical properties of production inputs against the training data — feature distributions, value ranges, missing-value patterns.
Concept drift occurs when the relationship between inputs and outputs changes. The model was trained to predict a relationship that no longer holds. A churn prediction model trained on data from a period of stable pricing may become inaccurate when a competitor launches an aggressive promotion — the factors that drive churn have changed, and the model's learned patterns are no longer valid. Concept drift is harder to detect because it requires ground truth labels, which often arrive with a delay. You only know the model's prediction was wrong when the outcome becomes known.
The detection strategy depends on the type of drift. For data drift, statistical tests can compare the distribution of incoming features against the training baseline. If the distribution shifts beyond a threshold, an alert fires. For concept drift, the approach is to track performance metrics over time — accuracy, precision, recall, or business-specific metrics like conversion rate — and detect when they trend downward.
The response to drift is not always retraining. Sometimes the drift is temporary — a seasonal shift, a one-time event, a data pipeline issue that should be fixed, not trained around. Sometimes the drift is real and permanent, and the model needs to be retrained on data that reflects the new reality. And sometimes the drift reveals that the model was never robust to begin with — it was overfit to a narrow data distribution, and the production environment has exposed that limitation.
The key practice is to make drift visible. A model whose performance is silently degrading is a liability. A model whose drift is detected, analysed, and responded to is a system that is being managed. The difference between a reliable AI system and an unreliable one is often not the model — it is the operational discipline around it.
Observability for AI: four pillars
Observability is the practice of instrumenting a system so that you can understand its behaviour from the outside — not just whether it is running, but how well it is performing and where problems are emerging. For AI systems, observability has four pillars.
1. Latency
Latency is the time it takes for the model to produce a response. It is the most user-visible metric — a slow AI system feels broken even if the outputs are perfect. But latency in AI systems has nuances that traditional software latency does not.
A large language model generating a 500-token response has an inherent latency floor — the time it takes the model to generate tokens sequentially. This is different from a database query, where the response is available all at once. AI systems also have variable latency — the same query can take different amounts of time depending on the complexity of the retrieved context, the length of the generation, and the load on the inference infrastructure.
Latency monitoring for AI needs to track not just average response time but the full distribution — the 95th and 99th percentile latencies that reveal the worst user experience. It needs to break latency into components: retrieval time, model inference time, post-processing time. And it needs to correlate latency with output quality — a model that is fast but wrong is not better than one that is slower but accurate.
2. Accuracy
Accuracy monitoring tracks whether the model's outputs are still correct in production. This is harder than it sounds because ground truth — the correct answer — often arrives with a delay. A churn prediction is only verifiable when the customer either churns or does not, weeks or months later. A support ticket classification is only verifiable when a human reviews it.
The practical approach is to use proximate signals — metrics that correlate with accuracy but are available in real time. For a RAG system, this might be the retrieval confidence score: if the retrieved chunks have low similarity to the query, the answer is likely to be weak. For a classification model, it might be the model's confidence in its prediction: a model that was previously confident and is now uncertain may be encountering inputs it was not trained on. For a generation model, it might be user feedback — thumbs up/down, follow-up questions, or whether the user rephrased and asked again.
When ground truth does arrive, it should be logged and used to compute retrospective accuracy. This closes the loop — it tells you whether your real-time signals were accurate proxies and whether the model's performance is trending up or down.
3. Cost
Cost monitoring tracks the financial impact of running the model. AI systems have variable costs that are unlike traditional software costs. A software API has a fixed cost per request. An AI system's cost depends on the number of tokens processed, the model used, the retrieval infrastructure, and the compute resources consumed.
Cost can escalate quickly and silently. A RAG system that retrieves too many chunks per query increases the generation prompt size, which increases the cost per query. A model that is called more often than expected — because of a spike in usage, a retry loop, or a misconfigured caching layer — can produce a bill that is multiples of the forecast. Cost monitoring needs to track cost per query, cost per user, and cost per outcome — and alert when any of these trend upward unexpectedly.
Cost is also a design constraint. The choice between a powerful but expensive model and a smaller but cheaper model is not just a technical decision — it is a business decision that depends on the value of the AI system's output. A customer-facing chatbot that handles sales inquiries can justify a higher per-query cost than an internal tool that summarises meeting notes. Cost monitoring makes these trade-offs visible.
4. Bias and fairness
Bias monitoring tracks whether the model's outputs are fair across different user segments. This is both an ethical and a business concern. A model that performs well overall but poorly for a specific demographic, language, or user segment is a model that is creating harm — and a model that will eventually create a business problem when that harm becomes visible.
Bias monitoring requires segmenting the model's performance by relevant dimensions — user geography, language, account type, or any other attribute that defines a meaningful user segment. If accuracy is 95% overall but 70% for a specific segment, that gap needs to be visible, investigated, and addressed.
Building an MLOps pipeline your team can maintain
The biggest failure mode in MLOps is not technical — it is operational. Teams build elaborate pipelines that require specialised knowledge to run, and when the person who built the pipeline leaves, nobody can maintain it. The pipeline becomes a black box that everyone is afraid to touch, and the system degrades.
The goal is not to build the most sophisticated MLOps pipeline. It is to build the simplest pipeline that your team can reliably operate. Here is what that looks like in practice.
Automate the repetitive, manual the judgement. Data validation, model training, and deployment can be automated. The decision to deploy a new model — especially one that regresses on some metrics — should involve human judgement. Build automation for the mechanical steps and create clear decision points for the ones that require context.
Version everything. Every training run, every dataset, every model weight, every configuration. This is not optional. Without versioning, you cannot reproduce a training run, cannot diagnose why a model's behaviour changed, and cannot roll back to a known-good state. Use a model registry — MLflow, Weights & Biases, or a simple versioned artifact store — to track every model version alongside its training metadata.
Start with monitoring, not automation. Before you automate anything, make sure you can see what your model is doing in production. A system with no automation but good monitoring is better than a fully automated system with no visibility. Monitoring tells you what to automate — it reveals the bottlenecks, the failure modes, and the manual steps that are consuming the most time.
Document the pipeline, not just the code. A pipeline that is documented as "run these scripts in this order" is a pipeline that dies when its author leaves. Document the why, not just the what. Why was this threshold chosen? Why does the pipeline retrain weekly instead of daily? Why is this feature included and that one excluded? A new team member should be able to understand the pipeline's design, not just execute its steps.
Design for the team you have, not the team you wish you had. If you do not have a dedicated MLOps engineer, do not build a pipeline that requires one. Use managed services — cloud-hosted model registries, managed training pipelines, serverless inference. The trade-off is less control and higher cost, but the benefit is a system that your existing team can operate without hiring a specialist. You can migrate to a more custom setup when the system's value justifies the investment.
FAQs: MLOps
Do we need MLOps if we are using a managed AI service (OpenAI API, Anthropic, etc.)?
If you are calling a managed API — OpenAI, Anthropic, Google — you do not need to manage model training or deployment. But you still need MLOps for everything around the model. You need to monitor latency and cost. You need to track which prompts and configurations are in production. You need to detect when the API provider changes the model's behaviour (which they do, without notice). You need to evaluate whether the model's outputs are still accurate for your use case. You need versioning for your prompts, your retrieval pipelines, and your system configurations. Managed services handle the model lifecycle. They do not handle your application lifecycle. The observability, evaluation, and operational discipline still belong to you.
How is MLOps different from traditional DevOps?
DevOps automates the path from code to production. The artefact is deterministic — the same code produces the same build, and tests pass or fail. MLOps automates the path from data to model to production. The artefact is probabilistic — the same training data can produce different model weights, and evaluation is a statistical comparison, not a binary test. MLOps also tracks a wider set of artefacts: code, data, model weights, training configurations, and evaluation metrics. And MLOps has a lifecycle that DevOps does not — models degrade over time and need retraining, while software either works or has a bug. The practices overlap (CI/CD, monitoring, infrastructure-as-code) but the ML lifecycle requires specialised tooling and workflows.
What tools should we use for MLOps?
Start with what you already use. If your team uses GitHub, GitHub Actions can handle basic CI/CD for model evaluation and deployment. If you use AWS, SageMaker provides end-to-end MLOps. If you use Google Cloud, Vertex AI. For model registries and experiment tracking, MLflow is open-source and widely adopted. Weights & Biases is excellent for experiment tracking and collaboration. For monitoring, tools like Arize, Fiddler, or WhyLabs specialise in ML observability. The right tool stack depends on your cloud provider, your team's expertise, and your budget. The wrong approach is to start with the tools and work backwards. Start with the problems you need to solve — versioning, monitoring, deployment — and choose the simplest tool that solves each one.
How often should we retrain our model?
It depends on how fast your data distribution changes and how quickly the model's performance degrades. A model trained on product documentation that changes weekly should be evaluated for retraining at least monthly. A model trained on customer behaviour data that shifts seasonally may need quarterly retraining. A model used in a fast-changing domain like fraud detection may need daily or even real-time retraining. The answer is not a fixed schedule — it is a monitoring signal. If your drift detection shows the model's performance dropping below your acceptable threshold, it is time to retrain. If the performance is stable, retraining adds risk without benefit. Let the data tell you when.
What is the most common MLOps mistake teams make?
Deploying a model without a monitoring strategy. Teams spend months building and training a model, deploy it, and then assume the job is done. The model runs, it produces outputs, and nobody checks whether those outputs are still accurate. By the time someone notices a problem — usually through a customer complaint or a business metric dropping — the model has been degrading for weeks or months, and the root cause is hard to trace. The second most common mistake is building a pipeline that is too complex for the team to maintain. A simple pipeline that runs reliably is worth more than a sophisticated one that nobody understands. Start simple, monitor everything, and add sophistication only when the system demands it.
Conclusion
A model that works in a notebook is a starting point, not a deliverable. The gap between a prototype and a production system is not bridged by a bigger model or more training data. It is bridged by the operational discipline that keeps the model running, accurate, and trustworthy over time.
MLOps is that discipline. It is the lifecycle — training, deployment, monitoring, retraining — that treats a model as a living system rather than a static artefact. It is the CI/CD pipeline that evaluates model changes with statistical rigour rather than a simple pass-or-fail. It is the drift detection that makes degradation visible before it becomes damage. It is the observability that tracks latency, accuracy, cost, and bias so you know not just whether the system is running but whether it is running well.
The businesses that succeed with AI are not the ones with the best models. They are the ones with the best operations around their models. A mediocre model with excellent MLOps will outperform a great model with no operations — because the mediocre model will be monitored, maintained, and improved, while the great model will silently degrade until it is neither great nor useful.
If you are building an AI system and want to make sure it runs reliably in production — not just in a demo, not just in the first week, but over months and years — we can help. We set up the pipelines, the monitoring, the deployment guardrails, and the evaluation discipline that turn a model into a product.
Set up your MLOps pipeline — we will help you run AI in production without losing sleep.