Let’s face it, when that alert goes off at 2:58am, abruptly shaking you out of a deep slumber because of a high-priority issue hitting the application, you’re not 100% “on”. You need to shake the fog out of your head to focus on the urgent task of fixing the problem. This is where having the best log analytics tool can take on some of that cognitive load.
Sumo Logic recently released new features specific to our Log Search queries that automatically detect log levels. This lets you quickly identify anomalies without having to search through large volumes of logs to find high-severity issues, so engineers like you can quickly troubleshoot and find the root cause to fix the issue and go back to bed.
Find the error spike fast
So, going back to our venerable hero, the on-call engineer (now rudely awakened by an issue alert), who is shuffling to the laptop to log into Sumo Logic. Typically, getting to log levels would require using field extraction rules (FERs) or modifying your log query search. Then you’d find the number of rows with the log level using count by times and log level, then be able to visualize it.
This is asking a lot from your brain at 2:58 am. Plus, it’s a cumbersome process that takes some time to do, which increases your troubleshooting time. Sumo Logic has made this much easier.
Going from the dashboard directly to the logs, you can see a spike in errors or warnings whenever an outage or incident happens, and this is easy to see in the histogram visualization on the logs page.
See exactly when the error occurred
Diving into the problem, running a query for, say, the last 70 minutes shows the distribution of all log levels. Visually the histogram shows the system was running as it should, until a spike in errors occurred then it resumes normal operation after about 15 minutes. You now know the time window to look at for your troubleshooting process, and you didn’t have to keep writing query after query to get to this point.
Filter specific log level messages to find the root cause
Now to dive deeper into the specific log messages related to the error spike shown on the histogram, just click on error legend to show just the errors as you see here. Filtering out the noise lets you focus on, and review, the log messages that are also the error messages. And that’s how to quickly progress in your troubleshooting journey to achieve this.
Finally, grouping the error log messages together pinpoints the log message that references the checkout error having a bad handshake to the SSL which then gets to the root cause of the issue.
Log level is available under the “_loglevel” field that you can use to write custom queries or build visualizations for your dashboard.
