“How should we be monitoring our containerized apps, as we move into production?” With Docker’s capabilities and methods for monitoring and logging evolving so quickly, even experienced Docker admins are revisiting this question. Our Data Collection Engineering team is continuously working to stay aligned with Docker’s capabilities and to improve and simplify our docker integrations. Just last month, with the release of Docker Enterprise Edition, v17.06, Docker released support for logging plugins. And, last week at Illuminate, our inaugural user conference, we announced that Sumo Logic published the first Certified Logging Plugin to the Docker Store ! We will come back to talking about this new logging plugin later in the post.
API Based Collector Agent
Now that the bragging is out of the way, let’s get back to the question at hand. There are many ways to get logs, metrics, and events from Docker. For that matter, there are too many. So we are going to try and keep it as simple as we can, and promote our preferred method and one alternative. We highly recommend running our containerized Sumo Logic Collector on your host, in its default packaging that runs our Docker Log Source and our Docker Stats Source.
This is as simple as:
docker run -d -v /var/run/docker.sock:/var/run/docker.sock
--name="sumo-logic-collector" sumologic/collector:latest <yourAccessID> <yourAccessKey>
In its default configuration, our containerized Collector agent will use the Docker API to collect the logs and statistics (metrics) from all containers, and the events that are emitted from the Docker Engine. Unless configured otherwise, the Collector will monitor all containers that are currently active, as well as any containers that are started and stopped subsequently. Within seconds, the latest version of the Collector container will be downloaded, and all of the signals coming from your Docker environment will be pumped up to Sumo Logic’s platform.
Using the API has its advantages. It allows us to get all 3 telemetry types (logs, metrics, and events), we can query for additional metadata during container startup, we don’t have to accommodate for different log file locations, and the integration is the same regardless of whether you log to files, or to journalD. Now, truth be told, we have dealt with our share of growing pains working with a relatively young logging API. However, since its initial release, the API has stabilized and our implementation has become far more robust.
The Benefits of Agent-Based Collection
The other advantage of this approach is the availability of a data collection agent that provides additional data processing capabilities and ensures reliable data delivery. Data processing capabilities include multiline processing, and data filtering and masking of data before leaving the host. This last capability is important when considering compliance requirements such as PCI or HIPAA. Also important from a compliance standpoint is reliability. All distributed logging systems must be able to accommodate networking issues or impedance mismatches, such as latency or endpoint throttling. These are all well covered issues when using the Sumo Logic Collector Agent.
Multiline Logging with Docker
One last point of key differentiation… Lack of multiline logging support has always plagued Docker logging. The default Docker logging drivers, and the existing 3rd party logging drivers, have not supported multiline log messages, and for the most part, they still do not.
One of Sumo Logic’s strengths has always been its ability to rejoin multiline log messages back into a single log message. This is an especially important issue to consider when monitoring JVM-based apps, and working with stack traces. Sumo Logic automatically infers common boundary patterns, and supports custom message boundary expressions. We ensure that our Docker Log Source and our Docker Logging Plugin maintain these same multiline processing capabilities. The ability to maintain multiline support is one of the reasons why we recommend using our custom Docker API based integration over simply reading the log files from the host.
Generally speaking, reading container logs from the file system is a fine approach. However, when the logs are wrapped in JSON, and ornamented with additional metadata, it makes the multiline processing far more difficult. Other logging drivers are starting to consider this issue, no doubt based on market feedback. However, their capabilities are far less mature than Sumo’s.
The installation of the containerized agent couldn’t be simpler. And with a simple query, you can see the data from all of the containers on your host, with all of the fields extracted and ready to explore. From there, it is easy to install our Docker App to monitor your complete Docker Environment as you scale this out to all of your hosts.
Logs Query : _sourceCategory=docker* | json auto
Going Beyond the Basics
Now, there is no need to B.S. you. When you deploy the Sumo Logic Collector container across a fleet of hosts, monitoring hundreds or thousands of containers, you will want to be a bit more sophisticated than just running with the default container settings. However, that is beyond the scope of this discussion. When you deploy our Collector Agent as a container, all of the Collector agent’s features are available, and all parameters can be configured. To read about how to dive into the advanced configuration options, check out the container’s readme on Docker Hub and read more details in our documentation .
Sometimes You Gotta Go Agentless
Ok, let’s get back to what I mentioned up at the top. It would be great to have just one simple answer, but there are times when you require an agentless solution – or you may just prefer one. If you have another way to collect container metrics, and you just need container logs, then a Docker Logging Plugin (earlier versions referred to as Logging Drivers) may be the perfect solution. The agentless approach is an ideal solution for AWS ECS users that rely on CloudWatch for their container metrics and events.
Our Docker Logging Plugin is written in Go, and runs within the Docker Engine. It is configured on a per container basis, and sends data directly to Sumo Logic’s HTTP Endpoint, using a pre-configured “HTTP Source.” You can access our plugin on the new Docker Store , but the best place to read about how to use it is on its Github repo.
Following the theme set out earlier, it is very easy to use in its default configuration, with a host of advanced options available. Follow these simple steps:
- Register the plugin with the Docker Engine :
$ docker plugin install –grant-all-permissions store/sumologic/docker-logging-driver:<ver>
(make sure you go to the Docker Store, and get the latest version number. As of this publishing, the latest version is 1.0.1 , and Docker Store does not support the ‘latest’ parameter. So, here is the corresponding command line for this version:
$ docker plugin install –grant-all-permissions store/sumologic/docker-logging-driver:1.0.1 )
- Specify the driver when you run a container:
$ docker run –log-driver=sumologic –log-opt sumo-url=<sumo_HTTP_url>
Docker Logging Plugin Capabilities
This plugin provides some very important capabilities:
- Buffering and batching. You can configure the size of each HTTP POST
- Compression: Configurable gzip compression levels to minimize data transfer costs
- Proxy support: Critical for highly secure enterprise deployment
- TLS Required: This is a Sumo Logic requirement. All data transfer must meet PCI compliance requirements.
- Multiline Support: Multiline stitching is processed within the Sumo Logic cloud platform rather than in the logging plugin. This keeps the plugin fast and efficient. However, we made specific design considerations to ensure that the we preserved multiline support while providing rich metadata support.
- Configurable Metadata per Container: The Docker Logging Plugin framework supports a flexible templating system that is used by our plugin to construct dynamic Source Category metadata that varies per container. The template syntax gives you access to environment vars, docker labels, and the ability to pass in custom values when starting containers. Our Docker Logging Plugin is the first of our integrations to support this capability. A similar capability will be supported by our Docker Log and Stats Sources with our next Collector release.
Let’s Wrap This Up
If, for some reason, these two methods do not satisfy your needs, then one of our many other collection methods (aka “Sources”) will most likely do the trick. Sumo Logic also integrates with various other open source agents and cloud platform infrastructures, and relies on some of them for certain scenarios. Details on all of the above integrations are available in our docs. If you have been using Docker for a while, and have implemented a solution from the early days, such as syslog or logspout, we encourage you to review the approaches defined here, and migrate your solution accordingly.