Pricing Login
Interactive demos

Click through interactive platform demos now.

Live demo, real expert

Schedule a platform demo with a Sumo Logic expert.

Start free trial
Back to blog results

July 27, 2017 By Brien Posey

Monitoring and troubleshooting using AWS CloudWatch logs

AWS CloudWatch is a monitoring tool for the AWS platform. CloudWatch logs are an important resource for monitoring and helping to interpret the data in your AWS cloud.

This article covers the essentials of working with AWS CloudWatch and CloudWatch Logs.

What can you do with CloudWatch?

As a tool, CloudWatch is quite versatile. IT Pros can use it for several purposes, including tracking performance metrics, setting threshold alarms, and even taking automated action when a monitored resource exceeds a predetermined threshold.

Monitor Amazon EC2 instances

One of the most common uses of AWS CloudWatch is monitoring EC2 instances. The nice thing about this functionality is that it is enabled by default. AWS collects performance metrics from EC2 instances every five minutes and stores those metrics for 15 months so that you can monitor performance changes over time.

For instances that require more timely performance data, AWS does provide an option to collect performance data every minute. Doing so requires you to enable detailed monitoring for the instance, which is a simple process but incurs an additional cost.

Monitor events logged by CloudTrail

AWS CloudWatch logs can do far more than monitor the performance of EC2 instances. You can also use CloudWatch to gather the events monitored by AWS CloudTrail. For those who might not be familiar with CloudTrail, it is designed to be an auditing mechanism for AWS.

As you are no doubt aware, AWS is made up of an extremely diverse collection of services. The one thing that all of these services have in common is that they are built around the use of APIs. An API is at work in the background whenever you interact with an AWS service. This holds regardless of whether the service is accessed programmatically, through the AWS console, or the AWS CLI. CloudTrail’s job is to capture a record of all API activity that occurs across an AWS account. An activity log is written to an S3 bucket, but delivering the logging data to CloudWatch is also possible.

Kinesis streams and AWS Lambda

AWS Kinesis Streams are designed to help AWS subscribers process or analyze extremely high volumes of streaming data. A Kinesis stream can simultaneously capture data from hundreds of thousands of sources and process or analyze multiple terabytes of data every hour. Kinesis is often used in conjunction with AWS Lambda, allowing for automatic streaming data processing. Lambda is designed to log data through CloudWatch logs.

Filtering and searching AWS CloudWatch logs

AWS CloudWatch logs can accumulate vast amounts of data, so it is important to filter the log data based on your needs. Filtering is achieved through the use of metric filters. Perhaps the most important thing to understand about metric filters is that they do not support retroactive filtering. Only events that have been logged since the filter was created will be reported in the filtered results. Log entries that existed before the filter’s creation are not included in the filtered results.

Creating a metric filter

Log into the AWS console and choose the CloudWatch service to create a metric filter. When the CloudWatch dashboard appears, click on the Logs option and then click on the number of metric filters displayed within your log group. (The number of metric filters will initially be set at zero.) You must create a log group before continuing if no log groups exist.

Click the Add Metric Filter button, and you will be taken to a screen that asks you to specify a few pieces of information. First, you will need to provide a filter pattern. A filter pattern specifies what the metric filter will look for within the log. (For instance, entering the word Error will cause the filter to look for occurrences of the word Error.)

Next, you must select the log data you plan to test. Once you have selected, click the Test Pattern button to ensure the results are what you expect, and then click on the Assign Metric button.

The resulting screen requires you to enter a filter name. The filter name is a friendly name used to identify the metric filter within the log group. You will also need to specify a metric namespace. A metric namespace is nothing more than a group that contains related metrics. By default, AWS uses LogMetrics as the metric namespace name.

Finally, you will have to specify a metric name. The metric name is the name of the CloudWatch metric where the log information will be published. AWS also allows you to write a metric value to the log when a pattern match occurs.

When you are done, click the Create Filter button, and the metric filter will be created. You can monitor your metrics from the CloudWatch Metrics dashboard.

<div class="at-below-post addthis_tool" data-url="https:="""" blog...<="" a>"="">

Complete visibility for DevSecOps

Reduce downtime and move from reactive to proactive monitoring.


Sumo Logic cloud-native SaaS analytics

Build, run, and secure modern applications and cloud infrastructures.

Start free trial

Brien Posey

Brien Posey is a Fixate IO contributor, and a 15-time Microsoft MVP with over two decades of IT experience. Prior to going freelance, Brien was CIO for a national chain of hospitals and healthcare facilities. He also served as Lead Network Engineer for the United States Department of Defense at Fort Knox. Brien has also worked as a network administrator for some of the largest insurance companies in America. In addition to his continued work in IT, Brien has spent the last three years training as a Commercial Scientist-Astronaut Candidate for a mission to study polar mesospheric clouds from space.

More posts by Brien Posey.

People who read this also enjoyed