Pricing Login
Interactive demos

Click through interactive platform demos now.

Live demo, real expert

Schedule a platform demo with a Sumo Logic expert.

Start free trial
Back to blog results

April 3, 2015 By David Wynn

Collecting and Analyzing CoreOS (journald) Logs w/ Sumo Logic

Towards More and Better Tools

With Docker becoming an increasingly popular platform for deploying applications, we’re continually looking into how we can best leverage Sumo to help collect all the logs from containerized apps. We’ve already posted about Docker a few times regarding best collection strategies, and our official Docker image.

One other request that we have heard from customers is how to pull logs, not from the containers themselves, but from journald, which CoreOS uses.

An easy way to do this is by setting up a new systemd service that forwards those logs over udp to a Sumo Logic Collector.

How to Set Up Journald Collection with Sumo Logic

First, you’ll need to set up a collector that listens for the udp traffic we’re about to send it. Since CoreOS is built for a containerized world, we recommend setting up the official Sumo Logic Docker image on the localhost, and mapping it to the appropriate ports.

docker run -d -p 514:514 -p 514:514/udp --name="sumo-logic-collector" sumologic/collector:latest-syslog [Access ID] [Access key]

Second, you’ll want to create a new unit that describes the forwarding system we’ll want to set up. An example unit file is provided below, but you can tweak the journalctl output if you want to change the formatting to another iso format or json.

Description=Send Journalctl to Sumo
ExecStart=/bin/sh -c '/usr/bin/journalctl -f | /usr/bin/ncat --udp localhost 514'



In-depth details for creating the service can be found here, though the gist is to save this unit file as journalctl_syslog.service in /etc/systemd/system and run the following commands:

$ sudo systemctl enable /etc/systemd/system/journalctl_syslog.service
$ sudo systemctl start journalctl_syslog.service

Once the service is up and running, that’s all there is to it. Restarts will be handled by systemd, and all the data should be forwarded appropriately to the cloud from the collector.

Example Queries

Once the data is present inside of Sumo Logic, you might want to try some of the following searches:

Message Count by Unit

| parse "\"MESSAGE\" : \"*\"" as message nodrop
| parse "\"UNIT\" : \"*\"" as unit nodrop
| where !(isNull(unit) OR unit="")
| timeslice by 1m
| count by unit, _timeslice
| transpose row _timeslice column unit

Log Levels Over Time

| parse "\"MESSAGE\" : \"*\"" as message nodrop
| parse "\"UNIT\" : \"*\"" as unit nodrop
| where isNull(unit) OR unit=""
| parse regex field=message "(?<level>[A-Z]{2,})"
| timeslice by 1m
| count by level, _timeslice
| where level !=""
| transpose row _timeslice column level

Outlier Detection on Total Number of Journald Messages

| timeslice by 1m
| count by _timeslice
| outlier _count

Complete visibility for DevSecOps

Reduce downtime and move from reactive to proactive monitoring.


Sumo Logic cloud-native SaaS analytics

Build, run, and secure modern applications and cloud infrastructures.

Start free trial

David Wynn

More posts by David Wynn.

People who read this also enjoyed