From Notebook to Production: Understanding ML in Production

The Distance between research and production.

In research, or notebook settings, one is concerned with experimentation:

1. Try multiple algorithms
2. Tune hyperparameters
3. Compare metrics
4. Improve validation scores

Nonetheless, the following are needed in production systems:

1. Reliability
2. Scalability
3. Reproducibility
4. Observability

One model performing well on the example can fail when implemented to the real world as a result of latency, memory constraints or unanticipated input patterns. It is here this gap becomes critical that MLOps is important.

The 3 Major steps of a Data Science Workflow.

The workflow in the majority of ML consists of three stages:

1. Data preparation
2. Model training
3. Deployment

Most attention is paid to data preparation and training. Deployment is most of the time seen as an end. The reality is, deployment happens to be the most complicated stage since it integrates the model to actual users and live systems.

The Dilemma of Static Files.

One of the typical anti-patterns of ML systems is to keep datasets or model artifacts as a static file located in a repository.

Typical issues include:

1. Large repository sizes
2. Pickle file versioning challenges.
3. Retrieving all the data into the memory.
4. Heavy Docker images
5. Slow deployments

There must be separation of data and models of application code in production systems. Store the artifacts in an appropriate system and tools rather than storing everything in the code.

Problems with Model and Data Versioning.

In that case, in contrast with conventional software, ML systems require monitoring more than code. You must also track:

1. Model versions
2. Training datasets
3. Feature engineering logic
4. Hyperparameters
5. Evaluation metrics

In the absence of versioning, teams end up having to deal with a confusing file name such as:

a. modelfinalv2
b. modellatestfinal_new

This poses problems of reproducibility. To create ML, a mapping is necessary between:

1. Code version
2. Data version
3. Model artifact

Training Code is Not Inference Code.

A large proportion of the teams directly use training code to infer. This can be a source of performance issues.

The code used in training is batch-optimized. Many production inferences usually involve:

1. Real-time predictions
2. Low latency
3. High throughput
4. Efficient memory usage

NumPy and pandas libraries are great to do some experimentation but do not seem to be the best choice when working with high-throughput APIs. Inference pipelines of production should be slim and lean.

Decision making in architecture is important.

The type of architecture that you adopt determines the level of scalability and maintainability.

Typical architectural decisions are:

1. Unified systems to be simple.
2. Modularity microservices.
3. Asynchronous workflow systems that are event based.
4. Cost minimization serverless components.

The right decision suggests traffic, complexity and business needs. ML systems should have scale consideration at their inception.

The least considered component is monitoring.

Models work in dynamic environments once they are deployed. Data distributions change. User behavior changes. Business logic evolves.

Thou can not see without keeping watch:

1. Data drift
2. Authentic performance degradation.
3. Latency spikes
4. Infrastructure failures

Monitoring should include:

1. System measures (CPU, memory, latency)
2. Measures of the model (accuracy, prediction distribution, drift detection)

The ML is not a one-time implementation anymore because it is monitored as a continuously running system.

Production ML Necessitates Engineering Discipline.

Research code puts a lot of emphasis on speed and experimentation. The priorities of production code should be:

1. Reliability
2. Maintainability
3. Observability
4. Scalability

ML engineers have to fill the differences between data science and software engineering. Coming up with the model is not the only thing to do. It is also important to augment the system with building around it.

Conclusion: production is a System, not a model, ML.

Machine learning production does not involve opening up an endpoint of prediction. It is concerning the establishment of a strong ecosystem that governs:

1. Data pipelines
2. Model artifacts
3. Versioning
4. Deployment workflows
5. Monitoring and retraining

Companies that make adequate investments in MLOps are less affected by failures, have higher scalability, and achieve the long-term reliability of their models.