Best Practices for Analyzing Elastic Load Balancer Logs

The AWS Elastic Cloud Compute (EC2) service offers a simple, robust vehicle for performing real-time load balancing of applications hosted within the Amazon Cloud. Elastic Load Balancer (ELB) is designed to optimize performance and scalability, and maximize resource utilization by balancing loads across multiple AWS instances. To use AWS effectively, you need continuous monitoring, detection, troubleshooting and reporting. For these tasks, analyzing Elastic Load Balancer logs is crucial. This post explains what you need to know about load balancing logs, and how to analyze them.

Overview of Elastic Load Balancer Logs

EC2 load balancing works by taking a single cloud-based application and creating two or more EC2 instances. Each instance is capable of resolving an access request in its own right. Access requests can be routed in real time to the EC2 instance under least load.

By recording each and every access request made to the EC2 platform, the resulting Elastic Load Balancing logs that are produced can be used to:

• Analyze access and traffic patterns
• Troubleshoot applications
• Perform security monitoring
• Improve the user experience
• Discover and debug problems with the EC2 platform

Planning for ELB Log Analysis

To get the most from ELB logs, you should perform the following tasks before you begin logging:

• Turn on access logging at every point possible within ELB.
• Make a plan for log storage. Determine where logs will be stored, how long they will be kept, and which users will be able to access them.
• Develop a set of processes for analyzing logs. This includes setting up multiple analytics views over the same logs to discern specific facts. It also includes defining how frequently each type of log entry needs to be analyzed.
• Set up a standard convention for naming log entries, reports and all other log-related data.

Using ELB Logs

Elastic Load Balancer logs can be produced by EC2 at a rate ranging from every five minutes to every 60 minutes. Deciding how frequently logs need to be produced will depend on how often there is a need to re-analyze logs.

Each load balancer will have its own log, and the filename of each log created will have the following format:
bucket[/prefix]/AWSLogs/aws-account-id/elasticloadbalancing/region/yyyy/mm/dd/aws-account-id_elasticloadbalancing_region_load-balancer-name_end-time_ip-address_random-string.log
A full explanation of how this filename is composed is available in the AWS documentation.

Each log file entry also has a standard format, which looks like this:
timestamp elb client:port backend:port request_processing_time backend_processing_time response_processing_time elb_status_code backend_status_code received_bytes sent_bytes "request" "user_agent" ssl_cipher ssl_protocol
Once again, the AWS documentation provides additional info on this format.

Using Analytics Tools to Examine EC2 Logs

You can begin the process of analyzing ELB logs manually by downloading log files in a format suitable and feeding them to a spreadsheet or database application. But that would be very time-consuming and inefficient. It would also be very difficult to derive real value from a large amount of log data if you attempt to analyze it by hand.

A much better and more effective approach is to leverage an analytics platform like Sumo Logic. Sumo Logic offers an analytics app designed specifically for ELB. It provides quick visualizations to help users interpret traffic data, discover choke points in AWS app performance, and so on.

Plus, the Sumo Logic app for AWS ELB can go further than simple analysis by allowing you to configure triggers to automate changes to the ELB configuration in response to given events. This feature allows you to correct load balancing issues automatically in order to prevent them from affecting users.

Additional Best Practices for ELB Logging

Whether you are going to export raw EC2 logs and perform analysis by hand or use a pre-built application such as Sumo Logic, there is a need to operate in a methodical and logical manner. Towards this end, here are some additional best practices that apply to all forms of logging, not just ELB logs:

• Keep it simple: Develop a bare-minimum approach to log file analysis. Log files are there to help, not guide. Spending too much time obsessing over log files takes valuable time away from other tasks such as application support and development.
• Log as much as you can: If you can log it, then log it. Even if you never actually analyze the additional things you log, you never know when you may need to in the future. Log it now, so that you have a full set of historic log data.
• Keep logs: Log files are so small, and current storage technology so vast in volume, that there is rarely a good reason for deleting historical log files. Keep them, as they can be used as part of a long-term analysis of application efficiency. They might also be necessary for auditing purposes.
• Centralize Storage: Keep all log files in a secure, centralized repository with easy access for real-time log monitoring and analysis.