What you should know about Datadog Flex Logs

Late last year, Datadog announced something called Flex Logs, a “more affordable” warm storage tier for log data. Designed for high-volume datasets that are infrequently queried and don’t require real-time analysis, the Flex Tier offers Datadog Log Management customers a third option for data storage.

It was basically Datadog’s way of saying, “Hey, we know storing all your log data with us costs an arm and a leg, so here’s a cheaper option.” It’s a warm storage tier designed for all those logs you collect but never really look at—kind of like that stack of New Yorker magazines you swear you’ll read someday. The Flex Tier offers Datadog Log Management customers a third option for data storage, Standard, Flex and Archive tiers.

*To analyze data stored in the Archive Tier, you need to “rehydrate” it into the Standard Tier and pay the applicable fees.

The trade-off: cost vs. performance

Datadog can apply a $0.05-per-million-event charge to Flex Logs because it separates compute and storage retention costs. Compute refers to the querying capacity needed to run queries for Flex Logs. This is used when querying logs in the Flex Logs tier. The available compute tiers, based on log volume and price, are listed below:

Choosing Datadog’s Flex Tier for storage comes with a trade-off: you save on storage costs at the expense of search performance. While it’s cheaper to store data in the Flex Tier, running queries and retrieving results will take longer compared to the standard index. Essentially, if you want faster performance, you’ll need to sacrifice some of those cost savings.

Making this decision isn’t as straightforward as it seems. It requires a deep understanding of how your organization will use the data once it’s ingested—a question that’s often easier asked than answered.

Which storage tier should you use?

To truly appreciate the value of Datadog’s Flex Logs, it’s crucial to categorize your logs into the appropriate storage tiers: Standard (hot), Flex (warm), and Archive (cold). But doing this effectively means you need to have a thorough grasp of the data you’re feeding into Datadog’s log management solution. Consider these key questions:

• What’s your daily data ingestion volume, and what types of log files are you ingesting?

• How valuable is the data? Is it redundant and noisy, or is it critical to your observability strategy?

• How does your team use this data? Is it for troubleshooting or compliance purposes?

• How often are these logs queried, and how urgently do users need results – yesterday, or whenever?

Unfortunately, Datadog doesn’t provide clear guidance on answering these questions. The responsibility of determining which logs to route to which tier falls squarely on the customer. And getting this wrong can be costly.

Picture this: It’s the middle of an outage, and the logs you desperately need are stored in the Flex Tier. Now your team has to play a game of “Wait for the Data” while they all hit the system at once and max out compute resources, potentially even dropping queries, all before they can even start fixing the problem. Fun, right?

The impracticality of storage tiers

In theory, storage tiers sound great as a way to save money and keep focused on the logs you need. But in practice, they’re not always as useful, with outsized drawbacks.

This is one reason why Sumo Logic has moved away from storage tiers in favor of a usage-based pricing model called Flex Pricing. This model lets you ingest as much data as you need without worrying about performance trade-offs or deciding which tier to use. It provides a scalable and efficient log analytics architecture that adapts to the demands of enterprise-wide and cloud-scale log ingestion, without forcing you to compromise on cost or performance. It’s like having your cake, eating it, and then getting another cake because you really can ingest everything.

Learn more about Sumo Logic pricing or sign up for your free trial.