Collecting Log Data from Amazon ECS & Docker Containers

# How Trek10 Uses Sumo Logic to Monitor Amazon Container Service

Guest Blog Post by Andrew Warzon, Founder at Trek10 and Jared Short, Director of DevOps at Trek10

You’ve probably heard of Docker by now – maybe you’ve toyed with Dockerizing your app or maybe you’re even running Docker in staging or production (We are!). For us, Docker means we can focus on building solid infrastructure and monitoring without having to worry about managing all sorts of things, like application specific dependencies, bootstrapping scripts, baking AMIs, or environment management. Docker also enables high fidelity parity from dev through to production, since everything runs in the same Docker containers. No more “but it works on my machine” bugs, or nasty dependency surprises between staging and production.

With the explosive growth of Docker, it is no surprise that AWS announced the EC2 Container Service (ECS) in November 2014. ECS entered General Availability in April 2015. At Trek10, we have been running apps in ECS since May. ECS enables you to take your lovingly Dockerized applications and processes and distribute them across a cluster. It handles tasks like load balancing, rolling deploys, service/application updates, healing, and more. Only because of Docker & ECS can we confidently say, “Running eight different applications on a cluster is no different than running one or two applications.” Powerful stuff!

As great as Docker & ECS seem to be, one of the biggest hurdles we faced was logging in a reliable, easy-to-manage way that let Docker & ECS do what they do best, with minimal engineering, configuration, and resource overhead.

Unfortunately, collecting logs from a container has no accepted, perfect solution. There are a lot of options out there…

Our primary goal is simplicity; we just want the logging problem to “get out of the way” so we can push ahead with building value. We’ve opted for a relatively simple solution: installing the Sumo collector on the ECS host and using mounted volumes.

For the impatient, here is the quick summary of how we make this work:

1. Install the Sumo agent unattended on the ECS host with user data
2. Make sure your sumosources.json file points to a new directory like /tmp/logs which you will map into your containers
3. Make sure your logs inside the container are written to some directory like /tmp/clogs
4. Use the ECS task definition to map /tmp/clogs to /tmp/logs/mycontainer

Here is some more detail on each step:

Step 1: Write a script to install Sumo on the host unattended. We will run this script in the EC2 user data. User data is EC2’s way to let you run scripts upon launching an instance. In this way, we can customize the host without maintaining our own AMIs; we simply add this script to the existing ECS AMI.

There are many ways to accomplish this, but our script includes the following:
Copy some configs out of an S3 bucket including Sumo access keys and a Sumo sources JSON file.
Create /etc/sumo.conf and /etc/sumosources.json on the host machine.
Actually install the Sumo collector

The key here is the sumosources.json file. Here is ours:

{
"api.version": "v1",
"sources": [
{
"sourceType" : "LocalFile",
"name" : "${CLIENT_NAME}_${ECS_CLUSTER}-ecs_apps",
"pathExpression" : "${LOGDIR}/**", "category": "${CLIENT_NAME}_${ECS_CLUSTER}-ecs", "hostName": "${CLIENT_NAME}_${INSTANCE_ID}", "useAutolineMatching": false, "multilineProcessingEnabled":${MULTILINE},
"manualPrefixRegexp": "${APP_REGEX}", "timeZone": "UTC", "automaticDateParsing": true, "forceTimeZone": false, "defaultDateFormat": "MMM dd HH:mm:ss" }, { "sourceType" : "LocalFile", "name" : "${CLIENT_NAME}_${ECS_CLUSTER}-ecs_messages", "pathExpression" : "/var/log/messages", "category": "${CLIENT_NAME}_${ECS_CLUSTER}-ecs", "hostName": "${CLIENT_NAME}_${INSTANCE_ID}", "useAutolineMatching": false, "multilineProcessingEnabled": false, "timeZone": "UTC", "automaticDateParsing": true, "forceTimeZone": false, "defaultDateFormat": "MMM dd HH:mm:ss" }, { "sourceType" : "LocalFile", "name" : "${CLIENT_NAME}_${ECS_CLUSTER}-ecs_secure", "pathExpression" : "/var/log/secure", "category": "${CLIENT_NAME}_${ECS_CLUSTER}-ecs", "hostName": "${CLIENT_NAME}_${INSTANCE_ID}", "useAutolineMatching": false, "multilineProcessingEnabled": false, "timeZone": "UTC", "automaticDateParsing": true, "forceTimeZone": false, "defaultDateFormat": "MMM dd HH:mm:ss" } ] }  Note line 7, pathExpression… this is the key. We define$LOGDIR to be some path on the host instance where we will later put our logs. This config just says to push anything in this directory into Sumo.

Step 2: Pick some directory inside the container where your logs will exist. How you accomplish this will vary significantly based on your application. We point ours to a separate directory inside the container, /tmp/clogs.

One key tip here: if whatever you are doing is different than how you would usually run this container, use the ECS Task Definition “Command” to override the default command for your container.

Here we are basically telling ECS to map all of the log files from inside the container (/tmp/clogs in our case) to outside the container where Sumo will be looking for log files as defined in sumosources.json

In the ECS task definition, this is done with two pieces. First, you must define a Volume. This is the path on the host that will now be available to be mapped to containers. Here is where to edit this in the AWS Management Console “Task Definition Builder” GUI:

One key note here: Make sure that this source path is a subdirectory of \$LOGDIR as defined in sumosources.json, and that subdirectory is unique for each container you define across all task definitions in your cluster. This way, any given host can have an arbitrary number of containers and an arbitrary number of tasks running on it and Sumo will get all of the logs and keep them separate.

The second piece of the task definition required is the “mount points” section of each container defined in your task definition. Use the volume name defined above, and map it to the log path inside the container. Below is how this looks in the Task Definition Builder:

If you prefer to write the JSON for the Task Definition directly, here is a generic Task Definition with these two pieces:

{
"family": "my-container",
"containerDefinitions": [
{
"name": "MyContainer",
"image": "python",
"cpu": 400,
"memory": 800,
"entryPoint": [],
"environment": [
{
"name": "MY_VAR",
"value": "foo"
}
],
"command": [],
"portMappings": [
{
"hostPort": 8080,
"containerPort": 8080
}
],
"volumesFrom": [],
"mountPoints": [
{
"sourceVolume": "logs",
"containerPath": "/tmp/clogs",
}
],
"essential": true
}
],
"volumes": [
{
"name": "logs",
"host": {
"sourcePath": "/tmp/logs/mycontainer"
}
}
]
}


So that’s it… a simple, low-maintenance, and flexible way to get all of your logs from ECS-run Docker containers into Sumo Logic. Good luck!

### Request a Demo

Thank you! We will get in touch with you shortly to schedule your Sumo Logic demo.
“Sumo Logic brings everything together into one interface where we can quickly scan across 1,000 servers and gigabytes of logs and quickly identify problems. It’s awesome software and awesome support.”

Jon Dokuli,
VP of Engineering

### Thank you for signing up for Sumo Logic.

We are creating your account now.