AWS CloudWatch is a monitoring tool for the AWS platform. CloudWatch logs are an important resource for monitoring and helping to interpret the data in your AWS cloud.
This article covers the essentials of working with AWS CloudWatch and CloudWatch Logs.
What can you do with CloudWatch?
As a tool, CloudWatch is quite versatile. IT Pros can use it for several purposes, including tracking performance metrics, setting threshold alarms, and even taking automated action when a monitored resource exceeds a predetermined threshold.
Monitor Amazon EC2 instances
One of the most common uses of AWS CloudWatch is monitoring EC2 instances. The nice thing about this functionality is that it is enabled by default. AWS collects performance metrics from EC2 instances every five minutes and stores those metrics for 15 months so that you can monitor performance changes over time.
For instances that require more timely performance data, AWS does provide an option to collect performance data every minute. Doing so requires you to enable detailed monitoring for the instance, which is a simple process but incurs an additional cost.
Monitor events logged by CloudTrail
AWS CloudWatch logs can do far more than monitor the performance of EC2 instances. You can also use CloudWatch to gather the events monitored by AWS CloudTrail. For those who might not be familiar with CloudTrail, it is designed to be an auditing mechanism for AWS.
As you are no doubt aware, AWS is made up of an extremely diverse collection of services. The one thing that all of these services have in common is that they are built around the use of APIs. An API is at work in the background whenever you interact with an AWS service. This holds regardless of whether the service is accessed programmatically, through the AWS console, or the AWS CLI. CloudTrail’s job is to capture a record of all API activity that occurs across an AWS account. An activity log is written to an S3 bucket, but delivering the logging data to CloudWatch is also possible.
Kinesis streams and AWS Lambda
AWS Kinesis Streams are designed to help AWS subscribers process or analyze extremely high volumes of streaming data. A Kinesis stream can simultaneously capture data from hundreds of thousands of sources and process or analyze multiple terabytes of data every hour. Kinesis is often used in conjunction with AWS Lambda, allowing for automatic streaming data processing. Lambda is designed to log data through CloudWatch logs.
Filtering and searching AWS CloudWatch logs
AWS CloudWatch logs can accumulate vast amounts of data, so it is important to filter the log data based on your needs. Filtering is achieved through the use of metric filters. Perhaps the most important thing to understand about metric filters is that they do not support retroactive filtering. Only events that have been logged since the filter was created will be reported in the filtered results. Log entries that existed before the filter’s creation are not included in the filtered results.
Creating a metric filter
Log into the AWS console and choose the CloudWatch service to create a metric filter. When the CloudWatch dashboard appears, click on the Logs option and then click on the number of metric filters displayed within your log group. (The number of metric filters will initially be set at zero.) You must create a log group before continuing if no log groups exist.
Click the Add Metric Filter button, and you will be taken to a screen that asks you to specify a few pieces of information. First, you will need to provide a filter pattern. A filter pattern specifies what the metric filter will look for within the log. (For instance, entering the word Error will cause the filter to look for occurrences of the word Error.)
Next, you must select the log data you plan to test. Once you have selected, click the Test Pattern button to ensure the results are what you expect, and then click on the Assign Metric button.
The resulting screen requires you to enter a filter name. The filter name is a friendly name used to identify the metric filter within the log group. You will also need to specify a metric namespace. A metric namespace is nothing more than a group that contains related metrics. By default, AWS uses LogMetrics as the metric namespace name.
Finally, you will have to specify a metric name. The metric name is the name of the CloudWatch metric where the log information will be published. AWS also allows you to write a metric value to the log when a pattern match occurs.
When you are done, click the Create Filter button, and the metric filter will be created. You can monitor your metrics from the CloudWatch Metrics dashboard.
