How to Monitor AWS S3

In the first part of our AWS S3 series, we discussed what AWS S3 buckets are, the difference between S3 and EC2s, advantages of AWS S3 object storage, and AWS S3 API integration. In this next post, we’ll be covering AWS S3 Monitoring, including the importance of leveraging data and monitoring metrics, and how Sumo Logic provides insight into your infrastructure with S3 logs.

What is AWS S3 Monitoring?

Amazon CloudWatch and CloudTrail both offer visibility into your infrastructure when enabled on your S3 buckets. Taking advantage of AWS S3 monitoring capabilities help you gain valuable insights which you can use to improve the overall performance of your web applications.

AWS S3 Monitoring and Logging

If your cloud environment were a hotdog, monitoring and logging would be the ketchup and mustard. Each one is good to use on its own, but they compliment each other so well that you should be utilizing both. What’s the difference you may ask?

• Monitoring - provides a report, usually for analyzing metrics or incident response teams. For AWS, this would be Amazon CloudWatch, for monitoring operational health and performance.
• Logging - creates a detailed log or record of events that occur within your cloud environment. Within AWS, this would mainly be CloudTrail, for auditing changes and API calls made to particular AWS services.

Gathering Data

AWS offers an abundance of ways to gather data for your monitoring services and applications. The more information your team can gather, the greater insight and visibility you have to make better decisions about your infrastructure.

By default, Amazon S3 doesn’t collect server access logs. Your team is going to want to enable Server Access Logging on your S3 buckets for detailed records for the requests of your objects within S3.

Together with Amazon S3 Server Access Logging, AWS CloudWatch, and AWS CloudTrail, your team can construct monitors and rules around your buckets for security and reliability.

How to Leverage Data

To demonstrate how data can be leveraged, let’s use a practical example. A DevOps team can monitor logs for a specific event, such as a user trying to gain access to a specific S3 bucket. While monitoring the logs, AWS can send a notification to the incident response team if someone does happen to log in using root credentials. For this specific example, AWS CloudWatch and AWS CloudTrail would both be used, in addition to AWS SNS and SQS.

Besides security reasons, teams can also leverage data access logs from AWS S3 for business purposes. With CloudWatch monitoring and CloudTrail logs, your team can ingest access logs into a service such as Sumo Logic. Sumo Logic can help spot and identify trends in requests, such as high traffic periods, or patterns from certain geo locations around the world.

Being able to predict and prepare for trends based on historical data can help teams mitigate outages or times of slow performance.

AWS S3 Monitoring Metrics

Metrics for web applications detail how a web application is performing on a certain system. Amazon AWS CloudWatch can ingest all the metrics your cloud environment puts out via logging and trails, which can then be used for search, querying, graphing, and creating alarms.

With CloudWatch request metrics, you have visibility with up to the minute logging and metrics, which you can then create alarms for, view via dashboards, filter for specific metrics using object tags, and more.

Visibility into Web Resources for AWS S3

It’s important to make sure your data is protected in your S3 buckets. Amazon offers several tools to ensure your team can gain major visibility into your web resources for AWS S3.

Amazon S3 offers Audit Logging, which provides your team with the insight of who is making requests and against what S3 resources. Protecting your data also includes identifying where your data is being requested from.

Amazon also offers AWS Macie. Macie is a security service from AWS that uses machine learning to recognize sensitive data and information which can be personally identifiable. This includes social security numbers, intellectual property and other private information.

How AWS S3 is Monitored with Sumo Logic

With the word “log” right in its name, Sumo Logic clearly makes for a great service to add to your infrastructure for real-time visibility and monitoring capabilities. How does Sumo Logic do this?

• Powerful Monitoring - gain a deeper insight into your infrastructure and systems. Sumo Logic’s proactive and predictive monitoring tools help you identify, prepare for, and resolve issues faster. Spot traffic trends and scale when necessary. Sumo Logic can help you track and identify spikes in data requests using geo location mapping.

• Troubleshooting and Root Cause Analysis - speaking of solving issues, Sumo Logic helps you identify the root cause of issues much quicker than having to parse through thousands of lines of logs. Is a region reporting latency issues or loss of connection? Sumo Logic can help identify specifically which error codes are being relayed, which allows your team to respond faster for remediation.

• Reporting and Auditing - While Amazon AWS may provide an abundance of resources for data and logging, Sumo Logic makes it possible for your team to parse through the information and make sound judgements and decisions based on the information available. What good is 500 pages of Server Access Logs if you can’t find the specific IP address that tried to access the Private Information S3 bucket 5 months ago. Sumo Logic makes that possible--and easy.

Conclusion

In Part 2 of our AWS S3 Monitoring with Sumo Logic series, we’ve covered the basics of AWS S3 logging, why it’s important to log all the information in your cloud environment, and also the benefits of monitoring those logs.

With Sumo Logic, teams gain valuable insight into their infrastructure with data metrics, monitoring, reporting, and logging, that just isn’t scalable or possible without the help of Sumo Logic.

In part three of AWS S3 Monitoring, we’ll be covering howSumo Logic integrates with Amazon S3 and how to get started with Sumo Logic, then we’ll take a deep look into Sumo Logic dashboards.