As human beings, we share quite a few life events that we keep track of, like birthdays, holidays, anniversaries, and so on. These are structured events that occur on exact dates or during specific times of year.
But how do you keep track of the unique, unexpected events that can be life-changing? The first meeting with someone, an inspiring conversation that sparked a realization—events that may seem common to many, but are so special to you.
Computer systems offer the same dilemma. Some events are expected, like adding a new user. Other events look routine, but from time to time they carry crucial, unexpected information. Unfortunately we most often realize how important pivotal events were after we experience a malfunction.
That’s where logs come in.
Virtually every computer program has some append-only structure for logs. Usually, it is as simple as a text file with a new line for each event. Sometimes the messages are saved to a database if the information may be used later. Why does it work that way? Well, it’s very easy to use and implement–usually it’s just one line of code. Don’t let the simplicity fool you. Logs provide a very powerful way of understanding and debugging systems. In many cases, logs are the sole method of figuring out the reason why something has happened.
From time to time, I’ll read about a new log management tool that converts log data into some standardized format. Well, there is limited value in that approach. Extracting data from logs is useful and could answer many business and operational questions. This works well with things that we expect, and things that answer numerical questions, like determining how many users have signed up in a given period of time.
However, during the process of converting logs to a standardized format, valuable data could be lost. For example, it’s interesting that many users couldn’t log in to your service, but the crucial information is why it happened. The unexpected part is usually very important and often even more valuable.
So do logs have a schema? Well, for the expected things, sure. But for analyzing the unexpected events it’s hard to think of a schema at all, beyond perhaps some partial structure.
That’s why at Sumo Logic, we accept any kind of log you throw at us. During log collection we just need to understand the events (e.g. separate lines) and the timestamp format. Everything else can be derived when you run a query.
Our query language lets you to find or extract structure, and data can be visualized and/or exported. Sumo Logic’s key advantage is how we handle the unexpected with machine learning algorithms. Our patent-pending LogReduce groups similar events on the fly to find anomalies, enabling our customers to review large sets of events quickly to identify the root cause of unexpected things.
No one ever intends to create bugs, but with the complexity and fast pace of software development they are inevitable. Well-designed systems should be debuggable. Log management tools, such as Sumo Logic, are here to help you deal with the logs that are a huge part of today’s technology.
“These days are only important, which are still unknown to us
These several moments are important, these for which we still wait”
– lyrics from famous Polish song by Marek Grechuta