Amazon SQS

Amazon Simple Queue Service (Amazon SQS) is a fully managed message queuing service that makes it easy to decouple and scale microservices, distributed systems, and serverless applications. The Sumo Logic app for Amazon SQS is a unified logs and metrics (ULM) app that provides operational insights into your Amazon SQS utilization. The preconfigured dashboards help you monitor the key metrics, view the SQS events for queue activities, and help you plan the capacity of your SQS service utilization.

Log and metric types

The Sumo Logic app for Amazon SNS uses the following logs and metrics:

• Amazon SQS CloudTrail Logs.
• Amazon SQS CloudWatch Metrics.

Sample log messages

Sample CloudTrail Log Message

{
   "eventVersion":"1.08",
   "userIdentity":{
      "type":"IAMUser",
      "principalId":"HN6WE52ULYKOOQWSKPD41",
      "arn":"arn:aws:iam::123456789033:user/ron_di",
      "accountId":"123456789033",
      "accessKeyId":"1AMZ29JHKLK97M7QVQSK",
      "userName":"ron_di",
      "sessionContext":{
        "sessionIssuer":{
        },
        "webIdFederationData":{
        },
         "attributes":{
            "creationDate":"2022-10-19T09:26:18+0000",
            "mfaAuthenticated":"false"
          }
        }
         },
         "eventTime":"2022-10-19T09:26:18+0000",
         "eventSource":"sqs.amazonaws.com",
         "eventName":"SetQueueAttributes",
         "awsRegion":"us-east-1",
         "sourceIPAddress":"195.186.216.125",
         "userAgent":"AWS Internal",
         "requestParameters":{
           "attributes":{
        "Policy":"{\"Version\":\"2008-10-17\",\"Id\":\"__default_policy_ID\",\"Statement\":[{\"Sid\":\"__owner_statement\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"arn:aws:iam::956882708938:root\"},\"Action\":\"SQS:*\",\"Resource\":\"arn:aws:sqs:us-east-1:956882708938:JayanatTest4\"},{\"Sid\":\"topic-subscription-arn:aws:sns:us-east-1:956882708938:SNSAppSomya01\",\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"SQS:SendMessage\",\"Resource\":\"arn:aws:sqs:us-east-1:956882708938:JayanatTest4\",\"Condition\":{\"ArnLike\":{\"aws:SourceArn\":\"arn:aws:sns:us-east-1:956882708938:SNSAppSomya01\"}}}]}"
      },
      "queueUrl":"https://sqs.us-east-1.amazonaws.com/123456789033/pull_private_submodule_jobs.fifo"
         },
         "responseElements":null,
         "requestID":"635ae9dd-83cc-5b42-890e-b273c168cb35",
         "eventID":"874213f3-d852-481d-bfd1-677c20f97427",
         "readOnly":false,
         "eventType":"AwsApiCall",
         "managementEvent":true,
         "recipientAccountId":"123456789033",
         "eventCategory":"Management",
         "sessionCredentialFromConsole":"true"         
}

Sample queries

Messages Received (Metrics-based)

metric=NumberOfMessagesReceived Statistic=Sum account=* region=* namespace=* queuename=* | sum by account, region, namespace, queuename

Top 10 users (CloudTrail Log-based)

account=* region=* namespace=aws/sqs eventname eventsource "sqs.amazonaws.com"
| json "userIdentity", "eventSource", "eventName", "awsRegion", "recipientAccountId", "requestParameters", "responseElements", "sourceIPAddress","errorCode", "errorMessage" as userIdentity, event_source, event_name, region, recipient_account_id, requestParameters, responseElements, src_ip, error_code, error_message nodrop
| json field=userIdentity "accountId", "type", "arn", "userName" as accountid, type, arn, username nodrop
| json field=requestParameters "queueUrl" as queueUrlReq nodrop
| json field=responseElements "queueUrl" as queueUrlRes nodrop
| where event_source="sqs.amazonaws.com"
| if(event_name="CreateQueue", queueUrlRes, queueUrlReq) as queueUrl
| parse regex field=queueUrl "(?<queueName>[^\/]*$)"
| where (tolowercase(queuename) matches tolowercase("*")) or isBlank(queuename)
| if (isBlank(recipient_account_id), accountid, recipient_account_id) as accountid
| if (isEmpty(error_code), "Success", "Failure") as event_status
| count as event_count by username
| top 10 username by event_count, username asc

Collecting logs and metrics for Amazon SQS

Configure Hosted Collector

When you create an AWS Source, you'll need to identify the Hosted Collector you want to use or create a new Hosted Collector. Once you create an AWS Source, associate it with a Hosted Collector. For instructions, see Configure a Hosted Collector and Source.

Collect Amazon SQS CloudWatch metrics

Sumo Logic supports collecting metrics using one of the following source types:

• Configure an AWS Kinesis Firehose for Metrics Source (recommended)

• Configure an Amazon CloudWatch Source for Metrics

  note

  Namespace for Amazon SNS service is AWS/SQS.

Follow the steps below to add custom metadata fields with your metrics:

1. Click +Add Field under Metadata. Each field consists of a name (key) and a corresponding value.
2. Create a field named account and assign it a value that represents a friendly name or alias to your AWS account from which metrics are collected. This value will appear in the AWS Observability view, and metrics can be queried using the account field.
3. After adding fields, check their status indicators:
   • A green check mark indicates the field exists and is enabled in the Fields table schema.
   • An orange exclamation icon indicates the field does not exist or is disabled in the schema.
     • You will have the option to automatically add or enable the field.
     • If a field is sent but not present or enabled in the schema, it is ignored and marked as Dropped.

Collect Amazon SQS CloudTrail logs

Prerequisites

1. Grant Sumo Logic access to an Amazon S3 bucket.
2. Create a trail for your AWS account.
3. Confirm that logs are being delivered to the Amazon S3 bucket.

note

Namespace for Amazon SQS service is AWS/SQS.

Follow the steps below to collect logs for Amazon SQS:

1. Configure a CloudTrail Logs Source.
2. Add custom metadata fields with your logs:
   1. Click +Add Field under Metadata. Each field consists of a name (key) and a corresponding value.
   2. Create a field named account and assign it a value that represents a friendly name or alias to your AWS account from which logs are collected. This value will appear in the AWS Observability view, and logs can be queried using the account field.
   3. After adding fields, check their status indicators:
      • A green check mark indicates the field exists and is enabled in the Fields table schema.
      • An orange exclamation icon indicates the field does not exist or is disabled in the schema.
        • You will have the option to automatically add or enable the field.
        • If a field is sent but not present or enabled in the schema, it is ignored and marked as Dropped.

Centralized AWS CloudTrail log collection

In case you have a centralized collection of CloudTrail logs and are ingesting them from all accounts into a single Sumo Logic CloudTrail log source, create the following Field Extraction Rule to map a proper AWS account(s) friendly name/alias. Create it if not already present/update it as required.

• Rule Name: AWS Accounts
• Applied at: Ingest Time
• Scope (Specific Data): _sourceCategory=aws/observability/cloudtrail/logs

Parse Expression

Enter a parse expression to create an “account” field that maps to the alias you set for each sub account. For example, if you used the “dev” alias for an AWS account with ID "528560886094" and the “prod” alias for an AWS account with ID "567680881046", your parse expression would look like:

| json "recipientAccountId"
// Manually map your aws account id with the AWS account alias you setup earlier for individual child account
| "" as account
| if (recipientAccountId = "528560886094",  "dev", account) as account
| if (recipientAccountId = "567680881046",  "prod", account) as account
| fields account

Installing the Amazon SQS app

Now that you have set up collection for Amazon SQS, install the Sumo Logic app to use the pre-configured dashboards that provide visibility into your environment for real-time analysis of overall usage.

To install the app, do the following:

note

Next-Gen App: To install or update the app, you must be an account administrator or a user with Manage Apps, Manage Monitors, Manage Fields, Manage Metric Rules, and Manage Collectors capabilities depending upon the different content types part of the app.

1. Select App Catalog.
2. In the 🔎 Search Apps field, run a search for your desired app, then select it.
3. Click Install App.
   note

   Sometimes this button says Add Integration.

4. Click Next in the Setup Data section.
5. In the Configure section of your respective app, complete the following fields.
   1. Field Name. If you already have collectors and sources set up, select the configured metadata field name (eg _sourcecategory) or specify other custom metadata (eg: _collector) along with its metadata Field Value.
6. Click Next. You will be redirected to the Preview & Done section.

Post-installation

Once your app is installed, it will appear in your Installed Apps folder, and dashboard panels will start to fill automatically.

Each panel slowly fills with data matching the time range query received since the panel was created. Results will not immediately be available but will be updated with full graphs and charts over time.

As part of the app installation process, the following content will be created by default along with dashboards and monitor template:

Fields

• account Name / alias to the AWS account.
• accountid AWS account id.
• region The region to which the resource name belongs to.
• namespace Namespace for Amazon SQS Service is AWS/SQS.
• queuename Amazon SQS Service Queue Name.

Field Extraction Rule(s)

The FER AwsObservabilitySQSCloudTrailLogsFER to extract fields region, namespace, accountid, and queuename will be created as a part of app installation.

Viewing Amazon SQS dashboards

Amazon Simple Queue Service (Amazon SQS) is a fully managed message queuing service that makes it easy to decouple and scale microservices, distributed systems, and serverless applications.

The Sumo Logic app for Amazon SQS provides operational insights into your Amazon SQS utilization. The app’s preconfigured dashboards help you monitor the key metrics, view the SQS events for queue activities, and help you plan the capacity of your SQS service utilization.

note

We recommend using the AWS Observability view.

Overview

The 1. Amazon SQS - Overview dashboard provides insights into SQS metrics and CloudTrail audit logs including the age, delayed, visible, sent and deleted messages, size of the messages and information about events.

Use this dashboard to:

• Monitor events by status, type, queues, location and users.
• Monitor number of messages received, sent, deleted and other metrics.
• Monitor message states, queue health, and message lag.

Queue Stats

The 1. Amazon SQS - Queue Stats dashboard provides details of SQS queue metrics including the delayed, invisible, deleted, lag, size, received and sent messages. This dashboard contains line chart panels showing trends for all the SQS metrics and a few use cases of them.Use this dashboard to:

• Monitor trend of messages received, sent, deleted and other metrics.
• Monitor message states, queue health and message lag.

Audit Events

The 2. Amazon SQS - Audit Events dashboard provides the details of SQS from CloudTrail audit logs including the top users, event locations, event status associated with queues. The dashboard has panels regarding successful and failure event locations, error code Use this dashboard to:

• Monitor events by status, type, queues, location and users.
• Monitor successful, failure event locations and trends.
• Monitor event details by users.
• Monitor successful and error event details.

Threat Intel

The 3. Amazon SQS - Threat Intel dashboard provides insights into incoming requests to your Amazon SQS services from malicious sources determined via Sumo Logic’s Threat Intel feature. Panels show detailed information on malicious IPs and the malicious confidence of each threat.

Use this dashboard to:

• Monitor details of threat locations and count.
• Get details of threats by malicious confidence and malicious IPs.
• Get details of all threats by IPs.

Performance Trends

The 1. Amazon SQS - Performance Trends dashboard provides derived performance insights including true consumer lag, empty receive rate trends, and cross-queue rankings by backlog and message staleness.

Use this dashboard to:

• Monitor true consumer lag by tracking combined visible and delayed messages in the backlog.
• Identify queues with high empty receive rates to optimize polling behavior and reduce costs.
• Rank queues by consumer backlog size to prioritize capacity planning.
• Identify message staleness risks by tracking the age of the oldest message per queue.

Create monitors for AWS SQS app

From your App Catalog:

1. From the Sumo Logic navigation, select App Catalog.
2. In the Search Apps field, search for and then select your app.
3. Make sure the app is installed.
4. Navigate to What's Included tab and scroll down to the Monitors section.
5. Click Create next to the pre-configured monitors. In the create monitors window, adjust the trigger conditions and notifications settings based on your requirements.
6. Scroll down to Monitor Details.
7. Under Location click on New Folder.
   note

   By default, monitor will be saved in the root folder. So to make the maintenance easier, create a new folder in the location of your choice.

8. Enter Folder Name. Folder Description is optional.
   tip

   Using app version in the folder name will be helpful to determine the versioning for future updates.

9. Click Create. Once the folder is created, click on Save.

AWS SQS alerts

These alerts are available for the AWS SQS app.

Alert Name	Alert Description and Conditions	Alert Condition	Recover Condition
AWS SQS - Access from Highly Malicious Sources	This alert fires when an Application AWS - SQS is accessed from highly malicious IP addresses within last 5 minutes.	Count > 0	Count <= 0
AWS SQS - Message processing not fast enough	This alert fires when we detect message processing is not fast enough. That is, the average approximate age of the oldest non-deleted message in the queue is more than 5 seconds for an interval of 5 minutes.	Seconds > 5	Seconds <= 5
AWS SQS - Messages not processed	This alert fires when we detect messages that have been received by a consumer, but have not been processed (deleted/failed). That is, the average number of messages that are in flight are >=20 for an interval of 5 minutes.	Count >= 20	Count < 20
AWS SQS - Queue has stopped receiving messages	This alert fires when we detect that the queue has stopped receiving messages. That is, the average number of messages received in the queue <1 for an interval of 30 minutes.	Count < 1	Count >= 1

Upgrade/Downgrade the Amazon SQS app (Optional)

To update the app, do the following:

note

Next-Gen App: To install or update the app, you must be an account administrator or a user with Manage Apps, Manage Monitors, Manage Fields, Manage Metric Rules, and Manage Collectors capabilities depending upon the different content types part of the app.

1. Select App Catalog.
2. In the Search Apps field, search for and then select your app.
   Optionally, you can identify apps that can be upgraded in the Upgrade available section.
3. To upgrade the app, select Upgrade from the Manage dropdown.
   1. If the upgrade does not have any configuration or property changes, you will be redirected to the Preview & Done section.
   2. If the upgrade has any configuration or property changes, you will be redirected to the Setup Data page.
   3. In the Configure section of your respective app, complete the following fields.
      1. Field Name. If you already have collectors and sources set up, select the configured metadata field name (eg _sourcecategory) or specify other custom metadata (eg: _collector) along with its metadata Field Value.
   4. Click Next. You will be redirected to the Preview & Done section.

Post-update

Your upgraded app will be installed in the Installed Apps folder and dashboard panels will start to fill automatically.

note

See our Release Notes changelog for new updates in the app.

To revert the app to a previous version, do the following:

1. Select App Catalog.
2. In the Search Apps field, search for and then select your app.
3. To version down the app, select Revert to < previous version of your app > from the Manage dropdown.

Uninstalling the Amazon SQS app (Optional)

To uninstall the app, do the following:

1. Select App Catalog.
2. In the 🔎 Search Apps field, run a search for your desired app, then select it.
3. Click Uninstall.