Sign up for a live Kubernetes or DevSecOps demo

Click here
Back to blog results

November 13, 2019 By Sadequl Hussain

How to Monitor AWS CloudTrail Logs with Sumo Logic

This is the third and last in a series of articles on Amazon CloudTrail. In the first part of the series, we introduced AWS CloudTrail and how it works and saw where and how it saves its data. We then learned how to query CloudTrail logs in the second part of the series where we used Amazon Athena to find meaningful information from large volumes of CloudTrail data.

Using one of the AWS built-in tools is great, but to gain useful and actionable intelligence, IT operation teams need a tool that can dig deeper, check anomalies, show trends, and send warnings when something is not quite right.

In this article, we will see how Sumo Logic is an ideal solution for this.

What’s Sumo Logic?

Sumo Logic is an industry-leading, Software-as-a-Service (SaaS) monitoring and SIEM tool that helps users to connect to a number of data sources, collect data to a centralized managed environment and then search, analyze, drill-down and visualize the aggregated data.

The ability to connect to, and collect data from a wide variety of platforms is made possible with Sumo Logic integrations. Using built-in integrations, Sumo Logic can collect data from a diverse set of sources. One of the built-in integrations available is for AWS CloudTrail.

With CloudTrail integration, Sumo Logic can connect to an AWS account and collect its CloudTrail logs into its own SaaS platform in a highly secured manner.

Users can then run real-time analytics on the logs to rapidly identify trends and anomalies. Sumo Logic apps like the “threat intel for aws” can easily show breaches and threats of cyberattacks based on latest data.

With a tool like this, Sumo Logic becomes the go-to application not just for the ops team, but also the DevOps and SecOps.

Installing Sumo Logic AWS CloudTrail Integration

To install CloudTrail integration in Sumo Logic, follow these steps:

  • From the “Manage Data” menu, choose “Collection”
  • Click on the “Setup Wizard” link from the top menu.
  • This will start the Setup Wizard. Click on the “Set up Streaming Data” link:
  • Select “AWS CloudTrail” link from the “Select Data Type” screen:
  • In the “Configure Source: AWS CloudTrail” screen:
    • Leave the source to “aws/cloudtrail”
    • Provide AWS Access Key ID and Secret Acces Key for the AWS account
    • Specify the bucket name where CloudTrail is saving its log files
    • Provide an expression for the path to the CloudTrail logs folder
    • Select the S3 region. In this case, we are selecting “Others”
  • Sumo Logic also provides a policy for the specified IAM user:
  • Copy this code and:
    • Create an inline IAM policy for the user whose credentials were provided
    • Paste the code in that policy document.
    • Save the inline policy
  • In the Sumo Logic “Configure Source” screen, click “Continue”
  • The initial data load starts. Depending on the existing volume of CloudTrail data, this can be a lengthy process:
  • Once the process is finished, Sumo Logic shows the number of records added:
  • Click on the “Exit Set up Wizard” button to exit the wizard.

Once the setup wizard closes, Sumo Logic opens a search screen with a default search criteria to show all CloudTrail records in the last sixty minutes:

Analyzing CloudTrail Logs with Sumo Logic

As the logs start to stream in, you can start searching and analyzing using Sumo’s proprietary query language. The search query language is both rich and well-documented. In the following image, we start by stipulating a source category of aws/cloudtrail. We then pipe the output through a number of operators. The first two operators find the event names from each CloudTrail log entry and the third one groups the events by their total number of occurrences. Finally, the sort operator sorts the figures:

The result for the query is a simple table:

As you can see, it’s possible to save the query or run it as Live Tail. With Live Tail, the query continuously updates the results as more logs stream in. Also, it’s very simple to create instant visualization from the resultset and save that as a dashboard widget:

The image below shows a pie-chart form of the resultset returned.

What’s a Sumo Logic App?

A Sumo Logic app is nothing but a collection of common searches, and pre-built dashboards based on those searches. The apps are freely accessible from the Sumo Logic interface: users can easily install an app for any data source installed.

Using an app has the benefit of not reinventing the wheel. Since the searches and dashboards already provide meaningful insights for common use cases, users don’t need to create widgets and dashboards from scratch.

There are two Sumo Logic apps users can make use of for CloudTrail:

  • Sumo Logic CloudTrail App
  • Sumo Logic Threat Intel for AWS App

Installing the Sumo Logic CloudTrail App

To add the Sumo Logic app for CloudTrail, follow these steps:

  • Navigate to the App Catalog screen by selecting the option from the left menu.
  • From the App Catalog, click the “AWS CloudTrail” icon
  • The AWS CloudTrail app’s page appears. You can see the developer for this app is Sumo Logic. You can also preview some of the dashboards available in the app.
  • Click on the “Add to Library” button. The “Add AWS CloudTrail to Library” dialog box appears. Provide a name for the app. We are giving it a name of “AWS CloudTrail”.

Next, specify a source category for the CloudTrail logs. This is the category Sumo Logic will use during searching. Here, we specify the category as “aws/cloudtrail”.

  • Also, we want the app to be installed in a separate folder. We already created a folder for it - “AWS_CloudTrail” and have selected that in the dialog box.
  • Click on “Add to Library”. The app will be added to the library in the specified folder and a tab will open, showing the dashboards that come with the app.

Sample Insights from Sumo Logic CloudTrail App

Here are a few sample insights from the CloudTrail app dashboards:

  • The Logins Over Time widget can give an idea about successful and failed logins to your AWS account over a period of time.
  • The Top 10 Users widget shows users by their number of activities over a period of time. From here, you can easily check for any unknown IAM user activity.
  • Network Security Events Over Time is another important widget. This can give a clear idea about Network related events that need to be closely monitored. For example, under normal circumstances, there should not be too many events of Security Group change events. If there is a sudden spike in this event for no planned reason, you can drill down further to discover when and who made the changes.
  • The Action Events widget shows a relative comparison of different types of events by their total number of occurrences over a period of time. Previously we had created one such visualization from our query.

Sumo Logic Threat Intel for AWS App

Sumo Logic has partnered with well-known industry player CrowdStrike and maintains an up-to-date cyber threat intelligence database. Using information from the CrowdStrike threat intelligence database, Sumo Logic can cross match with AWS logs and immediately report on breaches and cyber attacks.

The Threat Intel App for AWS is a visual representation of these attack patterns. This particular app makes use of AWS logs like VPC flow logs, ELB logs and AWS CloudTrail logs.

Installing the Threat Intel for AWS app is similar to installing the CloudTrail app. When adding to the library, you can specify the data source categories for CloudTrail, VPC flow logs and ELB logs:

A number of dashboards are installed with the package, one of those is specifically for CloudTrail events:

There are some very good widgets available for CloudTrail data within the overview dashboard and the CloudTrail dashboard.

  • The “Threats Over Time - CloudTrail” widget can easily show the number of security-critical events over time:
  • The “Threats by Geo Location” widget shows the geo location where threats are coming from:
  • “Threats by Events and Result” widget shows events which you need to investigate further. For example, in the image below, we can see the “GetAccountPasswordPolicy” event has a relatively large number of access denied errors. A security engineer would investigate such occurrences.

If the CloudTrail dashboard is blank, it does not necessarily mean the app is not working. It means Sumo could not find any data in the CrowdStrike database that could be matched with the CloudTrail events. The image below shows one such case:

Conclusion

Using Sumo Logic for monitoring CloudTrail has a number of benefits, for example:

  • As a SaaS tool, Sumo ensures CloudTrail data is saved in a compressed format and continuously indexed for easy searching, advanced analytics, and machine learning. This is a better option than using Amazon Athena to run SQL queries on the raw data, which can often take a long time to finish. Features like LogReduce can dramatically improve query performance.
  • It’s easy to build widgets and dashboards from custom queries. Queries can be saved for reuse and shared with other Sumo users in the same organization. This can be useful for different teams trying to get different views from the same set of data. A series of custom-written queries can form the basis of more complex analysis.
  • Advanced analytic options like future trends, time compare, outlier detection can easily identify possible threats and alert users.
  • Pre-built apps with out-of-box dashboards (both written by Sumo and those available from reputable communities) lets operations teams concentrate on the actual task of monitoring instead of building charts and widgets.

Organizations interested in taking Sumo Logic for a test drive can sign up for Sumo Free, a free tier option for storing and analyzing limited volumes of logs, or a 30-day free trial of Sumo Professional which offers more features. Advanced users can also opt for a 30-day free trial of Sumo Enterprise.

Complete visibility for DevSecOps

Reduce downtime and move from reactive to proactive monitoring.

Sadequl Hussain

Sadequl Hussain

Sadequl Hussain is an information technologist, trainer, and guest blogger for Machine Data Almanac. He comes from a strong database background and has 20 years experience in development, infrastructure engineering, database management, training, and technical authoring. He loves working with cloud technologies and anything related to databases and big data. When he is not blogging or making training videos, he can be found spending time with his young family.

More posts by Sadequl Hussain.

People who read this also enjoyed