AWS EC2 CW Metrics

Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the Amazon Web Services (AWS) Cloud. You can use Amazon EC2 to launch as many or as few virtual servers as you need, configure security and networking, and manage storage.

The Sumo Logic app for AWS EC2 allows you to collect your EC2 instance metrics and display them using predefined dashboards. The app provides dashboards to display analysis of EC2 instance metrics for CPU, disk, network, EBS, Health Status Check, and EC2 CloudTrail Events. Also, it provides detailed insights into all CloudTrail audit events associated with EC2 instances and specifically helps identify changes, errors, and user activities.

Log and metric types

The Sumo Logic app for AWS EC2 CloudWatch Metrics uses the following metrics:

• Amazon CloudWatch Metrics
• Amazon CloudTrail Logs

Sample log messages

Sample CloudTrail Log

{
	"eventVersion":"1.08",
	"userIdentity":{
		"type":"IAMUser",
		"principalId":"AIDAJ7LGGLTBHHDFNMPSM",
		"arn":"arn:aws:iam::9XXXX34567898:user/cloudhealthuser",
		"accountId":"9XXXXXXX898",
		"accessKeyId":"AKIAXXXXXX22BUTQ",
		"userName":"cloudhealthuser"
	},
	"eventTime":"2022-06-30T08:05:38Z",
	"eventSource":"ec2.amazonaws.com",
	"eventName":"DescribeReservedInstancesListings",
	"awsRegion":"us-east-1",
	"sourceIPAddress":"177.20.215.222",
	"userAgent":"aws-sdk-ruby2/2.11.447 jruby/2.5.7 java cloudhealth",
	"errorCode":"Client.OptInRequired",
	"errorMessage":"AccountId '9XXXXXX898', You are not authorized to use the requested product. Please complete the seller registration null.",
	"requestParameters":{
		"reservedInstancesListingSet":{

		},
		"reservedInstancesSet":{

		},
		"filterSet":{

		}
	},
	"responseElements":null,
	"requestID":"fe609b44-dbc5-454b-8f72-9475d1639441",
	"eventID":"6fc6df43-1ba1-4eb3-948a-0aebc569c024",
	"readOnly":true,
	"eventType":"AwsApiCall",
	"managementEvent":true,
	"recipientAccountId":"9XXXXX7898",
	"eventCategory":"Management",
	"tlsDetails":{
		"tlsVersion":"TLSv1.2",
		"cipherSuite":"ECDHE-RSA-XXXXX-SHA",
		"clientProvidedHostHeader":"ec2.us-west-1.amazonaws.com"
	}
}

Sample queries

CPU utilization (CloudWatch metric-based)

account=* region=* namespace=aws/ec2 instanceid=* metric=CPUUtilization Statistic=average | avg

Top 10 Error Codes (CloudTrail log-based)

account={{account}} region={{region}} namespace={{namespace}} eventname eventsource "ec2.amazonaws.com" errorCode
| json "eventSource", "awsRegion", "requestParameters", "responseElements", "recipientAccountId" as event_source, region, requestParameters, responseElements, accountid nodrop
| json "userIdentity", "eventName", "sourceIPAddress", "userAgent", "eventType", "requestID", "errorCode", "errorMessage", "eventCategory", "managementEvent" as userIdentity, event_name, src_ip, user_agent, event_type, request_id, error_code, error_message, event_category, management_event nodrop
| where event_source = "ec2.amazonaws.com"
| "aws/ec2" as namespace
| json field=userIdentity "type", "principalId", "arn", "userName", "accountId" nodrop
| json field=userIdentity "sessionContext.attributes.mfaAuthenticated" as mfaAuthenticated nodrop
| parse field=arn ":assumed-role/*" as user nodrop  
| parse field=arn "arn:aws:iam::*:*" as accountId, user nodrop
| json field=requestParameters "instanceType", "instancesSet", "instanceId", "DescribeInstanceCreditSpecificationsRequest.InstanceId.content" as req_instancetype, req_instancesSet, req_instanceid_1, req_instanceid_2 nodrop
| json field=req_instancesSet "item", "items" as req_instancesSet_item, req_instancesSet_items nodrop
| parse regex field=req_instancesSet_item "\"instanceId\":\s*\"(?<req_instanceid_3>.*?)\"" nodrop
| parse regex field=req_instancesSet_items "\"instanceId\":\s*\"(?<req_instanceid_4>.*?)\"" nodrop
| json field=responseElements "instancesSet.items" as res_responseElements_items nodrop
| parse regex field=res_responseElements_items "\"instanceType\":\s*\"(?<res_instanceType>.*?)\"" nodrop
| parse regex field=res_responseElements_items "\"instanceId\":\s*\"(?<res_instanceid>.*?)\"" nodrop
| if (!isBlank(req_instanceid_1), req_instanceid_1,  if (!isBlank(req_instanceid_2), req_instanceid_2, if (!isBlank(req_instanceid_3), req_instanceid_3, if (!isBlank(req_instanceid_4), req_instanceid_4, "")))) as req_instanceid
| if (!isBlank(req_instanceid), req_instanceid, res_instanceid) as instanceid
| if (!isBlank(req_instancetype), req_instancetype, res_instancetype) as instanceType
| if (isEmpty(error_code), "Success", "Failure") as event_status
| if (isEmpty(userName), user, userName) as user
| tolowercase(instanceid) as instanceid
| count as count by error_code | sort by count, error_code asc | limit 10

Collecting logs and metrics for AWS EC2

Configure Hosted Collector

When you create an AWS Source, you'll need to identify the Hosted Collector you want to use or create a new Hosted Collector. Once you create an AWS Source, associate it with a Hosted Collector. For instructions, see Configure a Hosted Collector and Source.

Collect AWS EC2 CloudWatch metrics

Sumo Logic supports collecting metrics using one of the following source types:

• Configure an AWS Kinesis Firehose for Metrics Source (recommended)

• Configure an Amazon CloudWatch Source for Metrics

  note

  Namespace for Amazon EC2 service is AWS/EC2.

Follow the steps below to add custom metadata fields with your metrics:

1. Click +Add Field under Metadata. Each field consists of a name (key) and a corresponding value.
2. Create a field named account and assign it a value that represents a friendly name or alias to your AWS account from which metrics are collected. This value will appear in the AWS Observability view, and metrics can be queried using the account field.
   
3. After adding fields, check their status indicators:
   • A green check mark indicates the field exists and is enabled in the Fields table schema.
   • An orange exclamation icon indicates the field does not exist or is disabled in the schema.
     • You will have the option to automatically add or enable the field.
     • If a field is sent but not present or enabled in the schema, it is ignored and marked as Dropped.

AWS EC2 automatically monitors functions on your behalf, reporting AWS EC2 metrics through Amazon CloudWatch. These metrics are collected by our Hosted Collector by configuring the Amazon CloudWatch source.

The Sumo Logic app for AWS EC2 (CloudWatch Metrics) allows you to collect your EC2 instance metrics and display them using predefined dashboards. The app provides dashboards to analyze EC2 instance metrics for CPU, disk, network, EBS, and Health Status Check.

Collect AWS EC2 CloudTrail logs

note

CloudTrail data events will be collected under this source.

Prerequisites

1. Grant Sumo Logic access to an Amazon S3 bucket.
2. Create a trail for your AWS account.
3. Confirm that logs are being delivered to the Amazon S3 bucket.

note

Namespace for Amazon EC2 service is AWS/EC2.

Follow the steps below to collect logs for AWS API Gateway:

1. Configure a CloudTrail Logs Source.
2. Add custom metadata fields with your logs:
   1. Click +Add Field under Metadata. Each field consists of a name (key) and a corresponding value.
   2. Create a field named account and assign it a value that represents a friendly name or alias to your AWS account from which logs are collected. This value will appear in the AWS Observability view, and logs can be queried using the account field.
      
   3. After adding fields, check their status indicators:
      • A green check mark indicates the field exists and is enabled in the Fields table schema.
      • An orange exclamation icon indicates the field does not exist or is disabled in the schema.
        • You will have the option to automatically add or enable the field.
        • If a field is sent but not present or enabled in the schema, it is ignored and marked as Dropped.

Centralized AWS CloudTrail log collection

If you have a centralized collection of CloudTrail logs and are ingesting them from all accounts into a single Sumo Logic CloudTrail log source, create following Field Extraction Rule to map proper AWS account(s) friendly name / alias. Create it if not already present / update it as required.

Rule Name: AWS Accounts
Applied at: Ingest Time
Scope (Specific Data): _sourceCategory=<SourceCategory_of_CloudTrail_source_created_in_sumo>

Parse Expression

Enter a parse expression to create an “account” field that maps to the alias you set for each sub account. For example, if you used the “dev” alias for an AWS account with ID "528560886094" and the “prod” alias for an AWS account with ID "567680881046", your parse expression would look like:

| json "recipientAccountId"
// Manually map your aws account id with the AWS account alias you setup earlier for individual child account
| "" as account
| if (recipientAccountId = "528560886094",  "dev", account) as account
| if (recipientAccountId = "567680881046",  "prod", account) as account
| fields account

Installing the AWS EC2 app

Now that you have set up collection for AWS EC2 metrics install the Sumo Logic app to use the pre-configured searches and dashboards that provide visibility into your environment for real-time analysis of overall usage.

To install the app, do the following:

note

Next-Gen App: To install or update the app, you must be an account administrator or a user with Manage Apps, Manage Monitors, Manage Fields, Manage Metric Rules, and Manage Collectors capabilities depending upon the different content types part of the app.

1. Select App Catalog.
2. In the 🔎 Search Apps field, run a search for your desired app, then select it.
3. Click Install App.
   note

   Sometimes this button says Add Integration.

4. Click Next in the Setup Data section.
5. In the Configure section of your respective app, complete the following fields.
   1. Field Name. If you already have collectors and sources set up, select the configured metadata field name (eg _sourcecategory) or specify other custom metadata (eg: _collector) along with its metadata Field Value.
6. Click Next. You will be redirected to the Preview & Done section.

Post-installation

Once your app is installed, it will appear in your Installed Apps folder, and dashboard panels will start to fill automatically.

Each panel slowly fills with data matching the time range query received since the panel was created. Results will not immediately be available but will be updated with full graphs and charts over time.

As part of the app installation process, the following content will be created by default along with dashboards and monitor template:

Fields

• account Name/alias to the AWS account.
• accountid AWS account ID.
• region The region to which the resource name belongs.
• namespace Namespace for EC2 CW Metrics Service.
• instanceid EC2 Instance Id.

Field Extraction Rule(s)

The FER AwsObservabilityEC2CloudTrailLogsFER to extract fields region, namespace, accountid, and instanceid will be created as a part of app installation.

Viewing AWS EC2 dashboards

Overview (CloudWatch Metrics)

The AWS EC2 Overview (CloudWatch Metrics) dashboard provides at-a-glance information about a EC2 CPU, instance disk store, network and EBS volume usage along with EC2 instance health status.

Use this dashboard to:

• Monitor average CPU utilization and dedicated host CPU utilization, along with instances with High and Low CPU utilization
• Monitor CPU credit Usage,Balance,Surplus Credit Charge,Surplus Credit balance
• EBS related metrics like IO Balance, Byte Balance, Read/Write - bytes,ops
• Identify count of Status checks
• Observe all relevant metrics for CPU, Internal Disk Store, Network utilization per instance type

Summary (CloudWatch Metrics)

The AWS EC2 Summary (CloudWatch Metrics) dashboard provides at-a-glance information about a EC2 CPU, instance disk store, network and EBS volume usage along with EC2 instance health status.

Use this dashboard to:

• Monitor CPU utilization along with CPU credit details & status check counts for EC2
• EBS related metrics for EC2 instance
• Observe Instance Disk Store (Disk Read/Write - Bytes & ops) for EC2 instance.
• Monitor Network usage metrics (Network in/out - Byes & packets) for EC2 instance

Events

The AWS EC2 - Events (CloudTrail) dashboard provides detailed insights into all CloudTrail audit events associated with EC2 instances and specifically helps identify changes, errors, and user activities.

Use this dashboard to:

• Monitor the geo location for successful and failed events
• Observed the event status and top error codes
• Track distribution and top event types and trends
• Monitor top IAM Users, Assumed Role Users, and User agents
• Monitor distribution of Successful and failed events with the list of latest events.

CPU (CloudWatch Metrics)

The AWS EC2 CPU (CloudWatch Metrics) dashboard provides detailed information about EC2 CPU usage like CPU utilization and CPU credits for burstable performance instances.

Use this dashboard to:

• Monitor Average CPU utilization over time for EC2.
• Observe CPU Credits metrics (Usage and balance) over time.
• Identify CPU Surplus Credits (Charged and Balance) over time.

EBS (CloudWatch Metrics)

The AWS EC2 EBS (CloudWatch Metrics) dashboard provides detailed information about EC2 EBS volumes for Nitro-based instances based on EBS volumes read and write bytes, operations, and information on the percentage of I/O and throughput credits remaining in the burst bucket.

Use this dashboard to:

• Monitor EBS volumes read and write bytes over time
• Monitor EBS read and write ops over time
• EBS IO balance and Byte Balance % metric over time for Ec2 instances.

Disk (CloudWatch Metrics)

The AWS EC2 Disk (CloudWatch Metrics) dashboard provides detailed information about a EC2 Instance Store Disk usage based on disk read and write bytes, operations.

Use this dashboard to:

• Monitor instance store - Disk metrics like Disk read/write Bytes and Byte rate
• Monitor instance store - Disk netrucs like Disk read/write Operations and Operation rate.

Network (CloudWatch Metrics)

The AWS EC2 Network (CloudWatch Metrics) dashboard provides detailed information about EC2 Network activities based on In and out packets, bytes.

Use this dashboard to:

• Monitor imported network metrics like - Byte rate for input and out put and Bytes going in and out of Ec2 instances
• Observe network metrics for Ec2 for packet in/out and rate of the packets.

Status Check (CloudWatch Metrics)

The AWS EC2 Status Check (CloudWatch Metrics) dashboard provides detailed information about an EC2 instance's health check status based on an instance, system, and overall health status.

Use this dashboard to:

• Monitor status check for instances
• Monitor if the instance has passed the status check at last minute
• Monitor if an instance has passed the system status check at last minute

Create monitors for AWS EC2 app

From your App Catalog:

1. From the Sumo Logic navigation, select App Catalog.
2. In the Search Apps field, search for and then select your app.
3. Make sure the app is installed.
4. Navigate to What's Included tab and scroll down to the Monitors section.
5. Click Create next to the pre-configured monitors. In the create monitors window, adjust the trigger conditions and notifications settings based on your requirements.
6. Scroll down to Monitor Details.
7. Under Location click on New Folder.
   note

   By default, monitor will be saved in the root folder. So to make the maintenance easier, create a new folder in the location of your choice.

8. Enter Folder Name. Folder Description is optional.
   tip

   Using app version in the folder name will be helpful to determine the versioning for future updates.

9. Click Create. Once the folder is created, click on Save.

AWS EC2 alerts

Name	Description	Alert Condition	Recover Condition
AWS EC2 CW - High CPU Utilization	This alert fires when the average CPU utilization based on cloud watch metrics, within a 5 minute interval for an EC2 instance is high (>=85%).	Count > 85	Count <= 85
AWS EC2 CW - Status Check Failed	This alert fires when there is a status check failures within a 5 minute interval for an EC2 instance.	Count > 0	Count <= 0
AWS EC2 - High Disk Utilization	This alert fires when the average disk utilization within a 5 minute time interval for an EC2 instance is high (>=85%).	Count >= 85	Count < 85
AWS EC2 - High Memory Utilization	This alert fires when the average memory utilization within a 5 minute interval for an EC2 instance is high (>=85%).	Count >= 85	Count < 85
AWS EC2 - High System CPU Utilization	This alert fires when the average system CPU utilization within a 5 minute interval for an EC2 instance is high (>=85%).	Count >= 85	Count < 85
AWS EC2 - High Total CPU Utilization	This alert fires when the average total CPU utilization within a 5 minute interval for an EC2 instance is high (>=85%).	Count >= 85	Count < 85
AWS EC2 CW - Low EBS IO Credit Balance	This alert fires when the average EBS IO Balance percentage within a 5 minute interval for an EC2 instance is low (<=10%), indicating the instance is close to being throttled on EBS IOPS.	Count <= 10	Count > 10
AWS EC2 CW - Low CPU Credit Balance	This alert fires when the average CPU Credit Balance within a 30 minute interval for an EC2 instance is low (<=5), indicating a burstable instance is close to losing burst capability and will be limited to baseline performance.	Count <= 5	Count > 5

Upgrade/Downgrade the AWS EC2 app (Optional)

To update the app, do the following:

note

Next-Gen App: To install or update the app, you must be an account administrator or a user with Manage Apps, Manage Monitors, Manage Fields, Manage Metric Rules, and Manage Collectors capabilities depending upon the different content types part of the app.

1. Select App Catalog.
2. In the Search Apps field, search for and then select your app.
   Optionally, you can identify apps that can be upgraded in the Upgrade available section.
3. To upgrade the app, select Upgrade from the Manage dropdown.
   1. If the upgrade does not have any configuration or property changes, you will be redirected to the Preview & Done section.
   2. If the upgrade has any configuration or property changes, you will be redirected to the Setup Data page.
   3. In the Configure section of your respective app, complete the following fields.
      1. Field Name. If you already have collectors and sources set up, select the configured metadata field name (eg _sourcecategory) or specify other custom metadata (eg: _collector) along with its metadata Field Value.
   4. Click Next. You will be redirected to the Preview & Done section.

Post-update

Your upgraded app will be installed in the Installed Apps folder and dashboard panels will start to fill automatically.

note

See our Release Notes changelog for new updates in the app.

To revert the app to a previous version, do the following:

1. Select App Catalog.
2. In the Search Apps field, search for and then select your app.
3. To version down the app, select Revert to < previous version of your app > from the Manage dropdown.

Uninstalling the AWS EC2 app (Optional)

To uninstall the app, do the following:

1. Select App Catalog.
2. In the 🔎 Search Apps field, run a search for your desired app, then select it.
3. Click Uninstall.