Collecting and Analyzing CoreOS (journald) Logs w/ Sumo Logic | Blog
Sign Up Free Request Demo

Collecting and Analyzing CoreOS (journald) Logs w/ Sumo Logic

Towards More and Better Tools

With Docker becoming an increasingly popular platform for deploying applications, we’re continually looking into how we can best leverage Sumo to help collect all the logs from containerized apps. We’ve already posted about Docker a few times regarding best collection strategies, and our official Docker image.

One other request that we have heard from customers is how to pull logs, not from the containers themselves, but from journald, which CoreOS uses.

An easy way to do this is by setting up a new systemd service that forwards those logs over udp to a Sumo Logic Collector.

How to Set Up Journald Collection with Sumo Logic

First, you’ll need to set up a collector that listens for the udp traffic we’re about to send it. Since CoreOS is built for a containerized world, we recommend setting up the official Sumo Logic Docker image on the localhost, and mapping it to the appropriate ports.

docker run -d -p 514:514 -p 514:514/udp --name="sumo-logic-collector" sumologic/collector:latest-syslog [Access ID] [Access key]

Second, you’ll want to create a new unit that describes the forwarding system we’ll want to set up. An example unit file is provided below, but you can tweak the journalctl output if you want to change the formatting to another iso format or json.

Description=Send Journalctl to Sumo
ExecStart=/bin/sh -c '/usr/bin/journalctl -f | /usr/bin/ncat --udp localhost 514'



In-depth details for creating the service can be found here, though the gist is to save this unit file as journalctl_syslog.service in /etc/systemd/system and run the following commands:

$ sudo systemctl enable /etc/systemd/system/journalctl_syslog.service
$ sudo systemctl start journalctl_syslog.service

Once the service is up and running, that’s all there is to it. Restarts will be handled by systemd, and all the data should be forwarded appropriately to the cloud from the collector.

Example Queries

Once the data is present inside of Sumo Logic, you might want to try some of the following searches:

Message Count by Unit

| parse "\"MESSAGE\" : \"*\"" as message nodrop
| parse "\"UNIT\" : \"*\"" as unit nodrop
| where !(isNull(unit) OR unit="")
| timeslice by 1m
| count by unit, _timeslice
| transpose row _timeslice column unit

Log Levels Over Time

| parse "\"MESSAGE\" : \"*\"" as message nodrop
| parse "\"UNIT\" : \"*\"" as unit nodrop
| where isNull(unit) OR unit=""
| parse regex field=message "(?<level>[A-Z]{2,})"
| timeslice by 1m
| count by level, _timeslice
| where level !=""
| transpose row _timeslice column level

Outlier Detection on Total Number of Journald Messages

| timeslice by 1m
| count by _timeslice
| outlier _count

Get Started Today!

Sign up for your FREE Sumo Logic Trial.

Sign Up Free

Request a Demo

Thank you! We will get in touch with you shortly to schedule your Sumo Logic demo.
“Sumo Logic brings everything together into one interface where we can quickly scan across 1,000 servers and gigabytes of logs and quickly identify problems. It’s awesome software and awesome support.”

Jon Dokuli,
VP of Engineering

Thank you for signing up for Sumo Logic.

We are creating your account now.
Please check your email.
Need more help? Contact Us
Sign up for free trial
Sign up for free trial

Full functionality for 30 days. Ingest and analyze data in minutes.

    • Please Enter your email address.
    • Please enter a valid email address.
    • This email is already in use for another account.
    • Please use your company email to create an account.
    • Please agree to the Service License.
    • Free trial provisioning is temporarily offline, please call 855-LOG-SUMO to get started.
    Deployment region: North America: US Change
    Plan: Professional - Free Trial Change
    View All Pricing Options
    Already have an account? Login