Towards More and Better Tools
With Docker becoming an increasingly popular platform for deploying applications, we’re continually looking into how we can best leverage Sumo to help collect all the logs from containerized apps. We’ve already posted about Docker a few times regarding best collection strategies, and our official Docker image.
One other request that we have heard from customers is how to pull logs, not from the containers themselves, but from journald, which CoreOS uses.
An easy way to do this is by setting up a new systemd service that forwards those logs over udp to a Sumo Logic Collector.
How to Set Up Journald Collection with Sumo Logic
First, you’ll need to set up a collector that listens for the udp traffic we’re about to send it. Since CoreOS is built for a containerized world, we recommend setting up the official Sumo Logic Docker image on the localhost, and mapping it to the appropriate ports.
docker run -d -p 514:514 -p 514:514/udp --name="sumo-logic-collector" sumologic/collector:latest-syslog [Access ID] [Access key]
Second, you’ll want to create a new unit that describes the forwarding system we’ll want to set up. An example unit file is provided below, but you can tweak the journalctl output if you want to change the formatting to another iso format or json.
[Unit]
Description=Send Journalctl to Sumo
[Service]
TimeoutStartSec=0
ExecStart=/bin/sh -c '/usr/bin/journalctl -f | /usr/bin/ncat --udp localhost 514'
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
In-depth details for creating the service can be found here, though the gist is to save this unit file as journalctl_syslog.service in /etc/systemd/system and run the following commands:
$ sudo systemctl enable /etc/systemd/system/journalctl_syslog.service
$ sudo systemctl start journalctl_syslog.service
Once the service is up and running, that’s all there is to it. Restarts will be handled by systemd, and all the data should be forwarded appropriately to the cloud from the collector.
Example Queries
Once the data is present inside of Sumo Logic, you might want to try some of the following searches:
Message Count by Unit
_sourceCategory=journald
| parse ""MESSAGE" : "*"" as message nodrop
| parse ""UNIT" : "*"" as unit nodrop
| where !(isNull(unit) OR unit="")
| timeslice by 1m
| count by unit, _timeslice
| transpose row _timeslice column unit
Log Levels Over Time
_sourceCategory=journald
| parse ""MESSAGE" : "*"" as message nodrop
| parse ""UNIT" : "*"" as unit nodrop
| where isNull(unit) OR unit=""
| parse regex field=message "(?<level>[A-Z]{2,})"
| timeslice by 1m
| count by level, _timeslice
| where level !=""
| transpose row _timeslice column level
Outlier Detection on Total Number of Journald Messages
_sourceCategory=journald
| timeslice by 1m
| count by _timeslice
| outlier _count
