--- title: "From stateful to stateless: Sumo Logic’s transition from lucene to parquet-based architecture" page_name: "From stateful to stateless: Sumo Logic’s transition from Lucene to Parquet-based architecture" type: "blog" slug: "stateful-to-stateless-architecture" published_at: "2024-10-29" modified_at: "2025-05-09" url: "https://www.sumologic.com/blog/stateful-to-stateless-architecture" canonical: "https://www.sumologic.com/blog/stateful-to-stateless-architecture" markdown_url: "https://www.sumologic.com/blog/stateful-to-stateless-architecture.md" lang: "en" excerpt: "Sumo Logic’s architecture innovation has boosted performance and facilitated the Flex pricing model. Discover why the team moved to a stateless architecture with Parquet." taxonomy_blog_category: - "Apache" - "Containers" - "DevOps & IT Operations" --- [ All blogs ](https://www.sumologic.com/blog "blog")[Apache](https://www.sumologic.com/blog/apache), [Containers](https://www.sumologic.com/blog/containers), [DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) # From stateful to stateless: Sumo Logic’s transition from Lucene to Parquet-based architecture [Priyansh Goel](#blog-author-block-339) October 29, 2024 3 min read [Apache](https://www.sumologic.com/blog/apache), [Containers](https://www.sumologic.com/blog/containers), [DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) ##### Table of contents Ensuring scalability, performance, and cost-effectiveness is a constant challenge for cloud-native [log management](https://www.sumologic.com/guides/log-management/) and [observability](https://www.sumologic.com/observability/). At Sumo Logic, we faced this challenge head-on by transitioning from a stateful, [Lucene](https://lucene.apache.org/)-based architecture to a completely stateless, [Parquet](https://parquet.apache.org/)-based architecture. This transformation lets us improve data storage efficiency, streamline operational complexity, and meet the demands of an ever-increasing data scale. Let’s walk through the journey of how we made this architectural shift, the reasoning behind it, the challenges we faced, and the benefits that come from adopting a stateless architecture powered by Parquet. ## The starting point: Lucene-based architecture We used to create Lucene indexes from customers’ [logs](https://www.sumologic.com/glossary/log-file/). Lucene is a famous search engine library that helps query the data quickly. Because Lucene had the inverted index created, we used to get a compression of only 4x. That means that for every 100GB of raw logs, we created a 25GB index. Downloading of these indexes took time, so we had to cache the indexes locally on disk. Also, because 95% of queries were of the last 15 days, we cached just the last 15 days’ indexes. So, we maintained two clusters – cached and cacheless. The cached cluster was mainly used to serve queries for the past 15 days. But in case the cached cluster was unable to serve the query, we would also route it to the cacheless cluster. The cacheless cluster was used to serve all the queries beyond 15 days and act as a fallback for the cached cluster. Because of this stateful architecture, we faced several challenges: - **Scalability** To cache the last 15 days indexes, we would provision the hardware for a customer according to the daily ingest. But there were cases when customers did a lot of querying making it hard to serve via the cached cluster. Because cached clusters are not autoscalable, the queries would go to the cacheless cluster, making it slow. Those slow queries would trigger our internal alerts, making our on-call load very high. - **Operational overhead** In cases where queries were not getting served via the cached cluster, the customers would complain of slow performance and we had to provision more hardware for them. This led to a lot of operational overhead for the engineering team. ## The transition: moving to a stateless, Parquet-based architecture Recognizing these challenges, we decided to pivot to a stateless architecture using Parquet as the primary data storage format. This shift marked a significant change in how Sumo Logic managed, stored, and queried data. ### Why Parquet? Apache Parquet is a columnar storage format optimized for analytics use cases, making it a natural fit for log management and observability. Its design allows for efficient compression and encoding schemes, which translates into reduced storage costs and faster query performance. Parquet’s columnar storage format provides better compression and performance compared to Lucene. By storing data in columns, similar data types are grouped together, allowing for more efficient compression. This reduces the storage footprint significantly, which is crucial for handling the large volumes of data Sumo Logic ingests. With Parquet we could achieve 16x compression, which means for every 100 GB of raw data we create a 6GB Parquet file. Because of this, we were able to unlock the stateless architecture as it reduced our need to keep the indexes cached. Since we moved to cacheless architecture completely, these are a few of the benefits that we observed: - **Scalable** This architecture is completely auto-scalable. We moved from EC2 instances to Kubernetes making the autoscaling even faster. - **No operational overhead** Now that the architecture is auto-scalable, we do not need hardware provisioning. To scale, we just needed to put minReplicas and leave them on K8s. - **Performance predictability** In earlier architecture, query performance was not predictable. The same query in the same time range could take two different times just because it was either served from cached or cacheless. With this architecture, we were able to bring predictability to query performance and remove such discrepancies. - **Unlocking Flex Licensing** This architecture allowed us to [unlock a new scan-based pricing model in Flex](https://www.sumologic.com/blog/log-everything-zero-dollar-ingest/). When we used to provision on the basis of ingest, it was extremely hard to charge customers on scan but with this architecture change, we could charge based on the usage. ## Final thoughts Moving from a stateful Lucene-based architecture with 15-day cached data on disk to a stateless Parquet-based system has been a monumental shift at Sumo Logic. The transition has provided us with a more scalable, resilient, and cost-effective platform that can keep pace with the growing demands of modern observability and log management. At Sumo Logic, we are committed to continuous innovation, and this transition is just one of the many steps we’re taking to build a platform that delivers unmatched performance, scalability, and reliability for our customers. Discover more about our latest product innovations on our [what’s new page](https://www.sumologic.com/whats-new/), or [start your free trial](https://www.sumologic.com/sign-up/) to try it for yourself. ### Article Tags - [Apache](https://www.sumologic.com/blog/apache) - [Containers](https://www.sumologic.com/blog/containers) - [DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) Priyansh Goel Priyansh Goel is an Engineering Manager in Logs Team responsible for ingesting petabytes of data per day and supporting query scans of exabytes per day. He has been at Sumo for 5 years. In his free time, he likes to read books and listen to music. [](https://www.sumologic.com/feed "RSS Feed")[](https://twitter.com/intent/tweet?text=From%20stateful%20to%20stateless%3A%20Sumo%20Logic%E2%80%99s%20transition%20from%20Lucene%20to%20Parquet-based%20architecture&url=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fstateful-to-stateless-architecture "X")[](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fstateful-to-stateless-architecture "Facebook")[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fstateful-to-stateless-architecture "Linkedin") [Previous blog Asia-based SMBs: leverage logs to enhance cloud security and scale your operations](https://www.sumologic.com/blog/asia-smb-logs-cloud-security)[Next blog Differentiating Sumo Logic Mo Copilot using Amazon Bedrock](https://www.sumologic.com/blog/copilot-amazon-bedrock) People who read this also enjoyed [ Monitoring and troubleshooting – Apache error log file analysis April 4, 2023 ](https://www.sumologic.com/blog/analyzing-apache-errors)[ Monitoring Apache Kafka clusters with Sumo Logic July 15, 2021 ](https://www.sumologic.com/blog/monitoring-apache-kafka-clusters-with-sumo-logic)[ How to Monitor Apache Web Server Performance May 15, 2019 ](https://www.sumologic.com/blog/monitor-apache-web-server)[ Apache Log Analysis with Sumo Logic October 11, 2017 ](https://www.sumologic.com/blog/apache-log-analysis) [AI Instructions](https://www.sumologic.com/ai-instructions.md)