Free Trial

Collecting and Analyzing CoreOS (journald) Logs w/ Sumo Logic

Towards More and Better Tools

With Docker becoming an increasingly popular platform for deploying applications, we’re continually looking into how we can best leverage Sumo to help collect all the logs from containerized apps. We’ve already posted about Docker a few times regarding best collection strategies, and our official Docker image.

One other request that we have heard from customers is how to pull logs, not from the containers themselves, but from journald, which CoreOS uses.

An easy way to do this is by setting up a new systemd service that forwards those logs over udp to a Sumo Logic Collector.

How to Set Up Journald Collection with Sumo Logic

First, you’ll need to set up a collector that listens for the udp traffic we’re about to send it. Since CoreOS is built for a containerized world, we recommend setting up the official Sumo Logic Docker image on the localhost, and mapping it to the appropriate ports.

docker run -d -p 514:514 -p 514:514/udp --name="sumo-logic-collector" sumologic/collector:latest-syslog [Access ID] [Access key]

Second, you’ll want to create a new unit that describes the forwarding system we’ll want to set up. An example unit file is provided below, but you can tweak the journalctl output if you want to change the formatting to another iso format or json.

[Unit]
Description=Send Journalctl to Sumo
 
[Service]
TimeoutStartSec=0
ExecStart=/bin/sh -c '/usr/bin/journalctl -f | /usr/bin/ncat --udp localhost 514'

Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

In-depth details for creating the service can be found here, though the gist is to save this unit file as journalctl_syslog.service in /etc/systemd/system and run the following commands:

$ sudo systemctl enable /etc/systemd/system/journalctl_syslog.service
$ sudo systemctl start journalctl_syslog.service

Once the service is up and running, that’s all there is to it. Restarts will be handled by systemd, and all the data should be forwarded appropriately to the cloud from the collector.

Example Queries

Once the data is present inside of Sumo Logic, you might want to try some of the following searches:

Message Count by Unit

_sourceCategory=journald 
| parse "\"MESSAGE\" : \"*\"" as message nodrop
| parse "\"UNIT\" : \"*\"" as unit nodrop
| where !(isNull(unit) OR unit="")
| timeslice by 1m
| count by unit, _timeslice
| transpose row _timeslice column unit

Log Levels Over Time

_sourceCategory=journald 
| parse "\"MESSAGE\" : \"*\"" as message nodrop
| parse "\"UNIT\" : \"*\"" as unit nodrop
| where isNull(unit) OR unit=""
| parse regex field=message "(?<level>[A-Z]{2,})"
| timeslice by 1m
| count by level, _timeslice
| where level !=""
| transpose row _timeslice column level

Outlier Detection on Total Number of Journald Messages

_sourceCategory=journald 
| timeslice by 1m
| count by _timeslice
| outlier _count

Get Started Today!

Sign up for your FREE Sumo Logic Trial.

Free Trial
Sign up for your 30 day free trial!
Sign up for
Sumo Logic Free
  • No credit card required to sign-up
  • Create your account in minutes
  • No expiration date
  • After 30 day trial period, reverts to Sumo Logic Free
    View All Pricing Options Privacy Policy