--- title: "Crossing the machine learning pilot to product chasm through mlops" page_name: "Crossing the machine learning pilot to product chasm through MLOps" type: "blog" slug: "mlops-framework" published_at: "2024-04-09" modified_at: "2025-11-10" url: "https://www.sumologic.com/blog/mlops-framework" canonical: "https://www.sumologic.com/blog/mlops-framework" markdown_url: "https://www.sumologic.com/blog/mlops-framework.md" lang: "en" excerpt: "Sumo Logic’s MLOps methodology helps accelerate machine learning-based features. Discover how you can adopt this framework for your production-grade features." taxonomy_blog_category: - "DevOps & IT Operations" --- [ All blogs ](https://www.sumologic.com/blog "blog")[DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) # Crossing the machine learning pilot to product chasm through MLOps [Bashyam Anant](#blog-author-block-42)[David Andrzejewski](#blog-author-block-93)[Aaishwarya Bansal](#blog-author-block-10) April 9, 2024 4 min read [DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) ##### Table of contents Numerous companies keep launching AI/ML features, specifically “ChatGPT for XYZ” type productization. Given the buzz around Large Language Models (LLMs), consumers and executives alike are growing to assume that building AI/ML-based products and features is easy. LLMs can appear to be magical as users experiment with them. However, in our experience, building production-grade AI/ML features requires hypotheses and problem formulation, curating training datasets, defining success metrics, modeling/algorithms, DevOps and most importantly, doing all these activities at scale in a performant, reliable and cost-efficient manner. Let us walk you through an MLOps framework that has helped Sumo Logic deliver several recent AI/ML-based features, including our [AI-driven alerting](https://www.sumologic.com/blog/ai-driven-low-noise-alerts/) capability, which is used as an illustrative example. We are also making available our [MLOps Observability dashboards](https://github.com/SumoLogic-Labs/MLOPs-Dashboards), which are designed to work with logs telemetry from your MLOps stack ## **MLOps framework for production-grade features** ### **Gathering customer requirements and success metrics** Our MLOps framework begins with understanding the customer problem and outcomes we want to solve. In the AI-driven alerting example, our customers reported experiencing alerts from their modern app stacks at rates higher than they are able to attend to. Such noisy monitors are common: 5% of monitors used by Sumo Logic customers trigger more than four times per day, 60-90% of these are false alarms, and anecdotally, half of these alerts are triggered after normal work hours. In short, for AI-driven alerting, our success criteria were to reduce such noisy monitors by 50% (i.e. no more than two times per day for the top 5% of monitors by detection rate). While the success criteria were defined in aggregate customer terms, such a metric was unsuitable for continuous model evaluation in our MLOps framework. As a result, we came up with a proxy for this success criteria we will describe later. As part of understanding the customer problem, we also suggest envisioning the ideal user experience. For AI-driven alerting, very early on, we assessed that the customer’s on-call engineers were looking to set up monitors without having to spend too much time tweaking monitor thresholds or parameters of an AI-driven alerting feature. As a result, we were looking for an approach that could support a “set it and forget it” experience with minimal user inputs. ### Collect data Next, you must figure out data sources relevant to your problem. For AI-driven alerting, customer logs (and eventually, time series’) from their apps and infrastructure stacks were our focus. More importantly, at this stage, you also want to preprocess the data to account for data aggregation, data privacy and compliance concerns (e.g. data minimization), data augmentation (e.g. join datasets based on downstream modeling requirements) and other factors. For AI-driven alerting, we only require a time-sliced signal from the customer’s logs (e.g. latency or throughput at five-minute intervals) – such aggregation is defined by the user as part of authoring a monitor. Conversely, aggregation ensures that we do not collect raw logs and also ensures that only the signal of interest is collected (versus everything else that is mentioned in the logs). Other considerations at this stage are to figure out how much data you would need based on model requirements that provide a good balance of model accuracy, cost and performance. For AI-driven alerting, we determined that 60 days of time-sliced logs were more than adequate for most of our customer use cases. You also have to set up data pipelines to automate data collection, which in turn require their own monitoring and operations. ### **Building the model** The Model: Train-Test-Predict is the heart of any AI/ML feature. Understanding the archetype of the problem and formulating the best-fit model for it is a crucial challenge. For AI-driven alerting, while the user could specify the signal of interest, their challenge is to figure out what makes a particular value of the signal unusual and worthy of an alert in the middle of the night. AI-driven alerting uses a forecasting approach where the model predicts the expected value of the signal in a given detection window (say five minutes) to detect unusual data points. This is augmented with some additional context provided by the user such as the percentage of data points in a detection window that should be unusual before triggering an alert. At this stage, you also have to translate the original success criteria into a measurement that can be evaluated continuously. For our example, we chose the prediction error in a given detection window for evaluating the model. In addition, we run the train-test-evaluate process in a closed loop to pick the best model hyperparameters. These automated machine learning (AutoML) features cut down the time-consuming, iterative process behind machine learning model development and are key to achieving the “set it and forget it” experience for users. A crucial and time consuming aspect of modeling is to figure out failure modes for the ML model. For example, how does the model handle incomplete or sparse data (e.g. you could set an alert on an error signal to allow the model to handle incomplete or sparse data), how do you handle cases where the AutoML is unable to converge on a well-tuned model in a reasonable amount of time, etc. As a best practice, previewing AI/ML features with customers and exposing the work-in-progress models to numerous diverse customer datasets is the only useful way to iron out failure modes in our experience. While LLMs may seem like a discontinuous break from prior ML technologies, similar practices can be adapted to help deploy and manage product features powered by LLMs. Ultimately, the modeling phase involves identifying a step in the customer experience where classic deterministic software cannot solve the target problem, but an ML/AI approach (whether Classical ML or LLMs) based on patterns or regularities in the data can deliver ideal results. As before, this problem formulation and framing exercise is foundational to the feature development process and heavily influences all subsequent design decisions. The dataset and training aspects may manifest differently in the LLM setting, depending on whether you are doing fine-tuning (actually updating model weights, as in Classical ML) or in-context learning (dynamically supplying training data in the LLM prompt at query time). In either case, it is still necessary to test the performance of a given configuration, which may be considered to include the base model, the prompt itself, as well as the specific learning procedure and hyperparameters (when fine-tuning) or the training instance population and selection procedure (for in-context learning). ### Deploy and Observe Finally, once a model has been tested and validated, you must determine how to expose the model to consumers, deploy it via your CI/CD process and observe it for accuracy, reliability and cost. At Sumo Logic, ML model predictions are served internally to other microservices via an API endpoint. The early stages of a rollout are also a good time to collect cost metrics compared to earlier estimates for the AI/ML feature and construct SLO targets for ML-as-a-service endpoints. Learn more about [AI/ML and log analysis](https://www.sumologic.com/guides/machine-data-analytics/). ### Article Tags - [DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) Bashyam Anant Sr Director, Advanced Analytics As a general manager and innovation leader, Bashyam has driven 30+ software products ($1B ARR, 7 patents) from concept to market leadership. At Sumo Logic, Bashyam leads AI experiences and platform products, which feature petabyte-scale machine learning to drive insights and action for cybersecurity and application reliability outcomes. In a prior strategy consulting career, Bashyam advised leadership teams at Boeing, HP and Motorola on new market and product assessments leading to multimillion dollar businesses. David Andrzejewski Director, Engineering David Andrzejewski is a Director of Engineering at Sumo Logic, where he works on applying machine learning to problems related to the reliability and security of modern software applications. Prior to Sumo Logic, David held a postdoctoral research position working on knowledge discovery at Lawrence Livermore National Laboratory (LLNL). He completed his PhD in Computer Sciences with a focus on Machine Learning at the University of Wisconsin-Madison. Aaishwarya Bansal Software Engineer II [](https://www.sumologic.com/feed "RSS Feed")[](https://twitter.com/intent/tweet?text=Crossing%20the%20machine%20learning%20pilot%20to%20product%20chasm%20through%20MLOps&url=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fmlops-framework "X")[](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fmlops-framework "Facebook")[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fmlops-framework "Linkedin") [Previous blog Responding to CVE-2024-3094 – Supply chain compromise of XZ Utils](https://www.sumologic.com/blog/respond-xz-utils)[Next blog How AI will impact cybersecurity: the beginning of fifth-gen SIEM](https://www.sumologic.com/blog/how-ai-will-impact-cybersecurity-the-beginning-of-fifth-gen-siem) People who read this also enjoyed [ Sumo Logic AWS Region European Sovereign Cloud is now generally available June 2, 2026 ](https://www.sumologic.com/blog/sumo-logic-aws-region-european-sovereign-cloud-generally-available)[ How to secure cloud workloads without building a full-scale SOC April 30, 2026 ](https://www.sumologic.com/blog/secure-cloud-workloads-with-limited-resources)[ Join operator and Query Agent for smarter log analysis April 22, 2026 ](https://www.sumologic.com/blog/using-the-join-operator)[ 92% of security leaders say their SIEM is effective. 51% say it’s exceptional. What’s living in that gap? April 16, 2026 ](https://www.sumologic.com/blog/from-effective-to-exceptional-siem) [AI Instructions](https://www.sumologic.com/ai-instructions.md)