Log Management and Analytics for the AWS ELB Classic Service
Sign Up Free Request Demo

Log Management and Analytics for the AWS ELB Classic Service

Quick Refresher

Earlier this year, we showed you how to monitor Amazon Web Services Elastic Load Balancer (AWS ELB) with Cloudwatch. This piece is a follow up to that, and will focus on Classic Load Balancers. Classic Load Balancers provide basic load balancing across multiple Amazon EC2 instances and operate at both the request level and connection level. Classic Load Balancers are intended for applications that were built within the EC2-Classic network. AWS provides the ability to monitor your ELB configuration with detailed logs of all the requests made to your load balancers. There is a wealth of data in the logs generated by ELB, and it is extremely simple to set up.

How to Get Started: Setting up AWS ELB Logs

Logging is not enabled in AWS ELB by default. It is important to set up logging when you start using the service so you don’t miss any important details!

Step 1: Create an S3 Bucket and Enable ELB Logging

Note: If you have more than one AWS account (such as ops, dev, and so on) or multiple regions that generate Elastic Load Balancing data, you’ll probably need to configure each of these separately.

Here are the key steps you need to follow

  1. Create an S3 Bucket to store the logs

Note: Want to learn more about S3? Look no further (link)

  1. Allow AWS ELB access to the S3 Bucket
  2. Enable AWS ELB Logging in the AWS Console
  3. Verify that it is working

Step 2: Allow Access to external Log Management Tools

To add AWS ELB logs to your log management strategy, you need to give access to your log management tool! The easiest way to do that is by creating a special user and policy.

  1. Create a user in AWS Identity and Access Management (IAM) with Programmatic Access. For more information about this, refer to the appropriate section of the AWS User Guide.

Note: Make sure to store the Access Key ID and Secret Access Key credentials in a secure location. You will need to provide these later to provide access to your tools!

  1. Create a Custom Policy for the new IAM user. We recommend you use the following JSON policy:

{
“Version”:”2012-10-17″,
“Statement”:[
{
“Action”:[
“s3:GetObject”,
“s3:GetObjectVersion”,
“s3:ListBucketVersions”,
“s3:ListBucket”
],
“Effect”:”Allow”,
“Resource”:[
“arn:aws:s3:::your_bucketname/*”,
“arn:aws:s3:::your_bucketname”
]
}
]
}

Note: All of the Action parameters shown above are required. Replace the “your_bucketname” placeholders in the Resource section of the JSON policy with your actual S3 bucket name.

Refer to the Access Policies section of the AWS User Guide for more info.

What do the Logs look like?

ELB logs are stored as .log files in the S3 buckets you specify when you enable logging.

The file names of the access logs use the following format:

bucket[/prefix]/AWSLogs/aws-account-id/elasticloadbalancing/region/yyyy/mm/dd/aws-account-id_elasticloadbalancing_region_load-balancer-name_end-time_ip-address_random-string.log

bucket

The name of the S3 bucket.

prefix

The prefix (logical hierarchy) in the bucket. If you don’t specify a prefix, the logs are placed at the root level of the bucket.

aws-account-id

The AWS account ID of the owner.

region

The region for your load balancer and S3 bucket.

yyyy/mm/dd

The date that the log was delivered.

load-balancer-name

The name of the load balancer.

end-time

The date and time that the logging interval ended. For example, an end time of 20140215T2340Z contains entries for requests made between 23:35 and 23:40 if the publishing interval is 5 minutes.

ip-address

The IP address of the load balancer node that handled the request. For an internal load balancer, this is a private IP address.

random-string

A system-generated random string.

The following is an example log file name:

s3://my-loadbalancer-logs/my-app/AWSLogs/123456789012/elasticloadbalancing/us-west-2/2014/02/15/123456789012_elasticloadbalancing_us-west-2_my-loadbalancer_20140215T2340Z_172.160.001.192_20sg8hgm.log

Syntax

Each log entry contains the details of a single request made to the load balancer. All fields in the log entry are delimited by spaces. Each entry in the log file has the following format:

timestamp elb client:port backend:port request_processing_time backend_processing_time response_processing_time elb_status_code backend_status_code received_bytes sent_bytes “request” “user_agent” ssl_cipher ssl_protocol

The following table explains the different fields in the log file. Note: ELB can process HTTP requests and TCP requests, and the differences are noted below:

Field Description
timestamp The time when the load balancer received the request from the client, in ISO 8601 format.
elb The name of the load balancer
client:port The IP address and port of the requesting client.
backend:port The IP address and port of the registered instance that processed this request.
request_processing_time [HTTP listener] The total time elapsed, in seconds, from the time the load balancer received the request until the time it sent it to a registered instance.

[TCP listener] The total time elapsed, in seconds, from the time the load balancer accepted a TCP/SSL connection from a client to the time the load balancer sends the first byte of data to a registered instance.

backend_processing_time [HTTP listener] The total time elapsed, in seconds, from the time the load balancer sent the request to a registered instance until the instance started to send the response headers.

[TCP listener] The total time elapsed, in seconds, for the load balancer to successfully establish a connection to a registered instance.

response_processing_time [HTTP listener] The total time elapsed (in seconds) from the time the load balancer received the response header from the registered instance until it started to send the response to the client. This includes both the queuing time at the load balancer and the connection acquisition time from the load balancer to the client.

[TCP listener] The total time elapsed, in seconds, from the time the load balancer received the first byte from the registered instance until it started to send the response to the client.

elb_status_code [HTTP listener] The status code of the response from the load balancer.
backend_status_code [HTTP listener] The status code of the response from the registered instance.
received_bytes The size of the request, in bytes, received from the client (requester).

[HTTP listener] The value includes the request body but not the headers.

[TCP listener] The value includes the request body and the headers.

sent_bytes The size of the response, in bytes, sent to the client (requester).

[HTTP listener] The value includes the response body but not the headers.

[TCP listener] The value includes the request body and the headers.

request The request line from the client enclosed in double quotes and logged in the following format: HTTP Method + Protocol://Host header:port + Path + HTTP version.

[TCP listener] The URL is three dashes, each separated by a space, and ending with a space (“- – – “).

user_agent [HTTP/HTTPS listener] A User-Agent string that identifies the client that originated the request. The string consists of one or more product identifiers, product[/version]. If the string is longer than 8 KB, it is truncated.
ssl_cipher [HTTPS/SSL listener] The SSL cipher. This value is recorded only if the incoming SSL/TLS connection was established after a successful negotiation. Otherwise, the value is set to -.
ssl_protocol [HTTPS/SSL listener] The SSL protocol. This value is recorded only if the incoming SSL/TLS connection was established after a successful negotiation. Otherwise, the value is set to -.

You can find more details here.

Example Log Message

2017-01-20T23:00:26.059475Z elb-shop-com 10.15.120.181:80 10.34.7.122:80 0.000026 0.315185 0.000027 200 200 51 1230 “POST https://examplesite.com:443/Common/path HTTP/1.1”

“Mozilla/5.0 (Safari; Touch) AppleWebKit/537.35+ (HTML, like Gecko) Version/10.3.2.2239 Mobile Safari/517.35+”

How do you analyze AWS ELB Logs?

The logs from AWS ELB contain a wealth of information that you can use to get deep insights into your application performance. Your first step is to extract all of the relevant fields using your tool. Then you can analyze the data in lots of interesting ways.

The possibilities for analysis on the ELB logs are virtually endless! We will show you some ways to do this with basic grep and with the Sumo Logic query language as references. Here are some key use cases for you to consider:

Tracking Errors and Exceptions

Log Fields: elb_status_code and backend_status_code

One of the most important pieces of data is the status code of the request, both from the ELB side and your server side. You can track redirects with 3xx status codes, client side issues with 4xx status codes (ex. an incorrect URL) and 5xx status codes for serious server errors (ex. a process on your server crashed). You can see more details on status codes here.

You can then aggregate those results over time by other fields in the message. Here are some examples:

  • Overall Statistics
    • Total requests across the entire application by status code or status code category
  • By ELB Instance (using the elb field)
    • Total requests by each instance
    • If you see an overall issue, you need to see what instances are having issues
  • Failed Dispatch
    • This has happened if request_processing_time OR backend_processing_time OR response_processing_time are set to -1
    • This typically happens because:
      • The load balancer can’t dispatch the request to a registered instance.
      • The registered instance closes the connection before the idle timeout or if the client sends a malformed request.
      • The registered instance does not respond before the idle timeout.
Example Sumo Logic Query
Look for specific status codes _sourceCategory = “aws_elb”

| split _raw delim=’ ‘ extract 8 as elb_status_code

| where elb_status_code = 500

Look for categories of status code _sourceCategory = “aws_elb”

| split _raw delim=’ ‘ extract 8 as elb_status_code

| where elb_status_code matches “5*”

Show lines where the elb and front end codes don’t match _sourceCategory = “aws_elb”

| split _raw delim=’ ‘ extract 8 as elb_status_code, 9 as backend_status_code

| where !(elb_status_code = backend_status_code)

Count request by status code _sourceCategory = “aws_elb”

| split _raw delim=’ ‘ extract 8 as elb_status_code, 9 as backend_status_code

| count by elb_status_code

Failed Dispatch (where any response times are set to -1) _sourceCategory = “aws_elb”

| parse “* * *:* ” as date_time, elb, client_ip, client_port

| split _raw delim=’ ‘ extract 5 as request_processing_time, 6 as backend_processing_time, 7 as response_processing_time

| where (request_processing_time=-1 OR backend_processing_time=-1 OR response_processing_time=-1)

| count by clientIP

| sort by _count

| limit 10

Tracking Request Behavior

You can understand the nature of your requests by examining different parts of the log message, and looking at the number of requests by those characteristics

  • Overall Statistics
    • Total process time latency trend across the entire application (we recommend using 95th and 99th percentiles, as well as a basic average). You can do this for request, response, and backend processing time.
    • By comparing these 3 basic measurements overall, you can see what part of the request processing is causing issues
  • Geolocation
    • If you have access to a geolocation (IP to physical location), you understand the origin of your requests. Sumo Logic recently partnered with Neustar for access to a new, highly accurate geolocation database.
    • This is extremely useful for both understanding your traffic and finding bad actors
  • By ELB Instance (using the elb field)
    • Total requests by each instance
    • If you see an overall issue, you need to see what instances are having issues
  • Data Volume
    • Data Sent and Received in MB: the data being sent and received by client IP
Example Sumo Logic Query
Look for the top client IPs _sourceCategory = “aws_elb”

| parse “* * *:* ” as date_time, elb, client_ip, client_port

| count as requests by client_ip

| sort by requests

GeoLocation _sourceCategory = “aws_elb”

| parse “* * *:* ” as date_time, elb, client_ip, client_port

| lookup latitude, longitude, country_code, country_name, region, city, postal_code, area_code, metro_code from geo://location on ip = client_ip

| count as requests by city, region, country_name

| sort by requests

Total Data Sent and Received by ELC instance _sourceCategory = “aws_elb”

| split _raw delim=’ ‘ extract 2 as elb, 10 as received_bytes, 11 as sent_bytes

| ((received_bytes/(1024))/(1024)) as received_bytes

| ((sent_bytes/(1024))/(1024)) as sent_bytes

| sum(received_bytes) as data_received, sum(sent_bytes) as data_sent by elb

Application Latency

Log Fields:  request_processing_time, backend_processing_time, response_processing_time

To understand how your users experience your application, you must understand the time requests take. Users judge their experience, first and foremost, by how long an action in your application takes. Using these fields, you can do some key analysis of your application experience:

  • Overall Statistics
    • Total process time latency trend across the entire application (we recommend using 95th and 99th percentiles, as well as a basic average). You can do this for request, response, and backend processing time
    • By comparing these 3 basic measurements overall, you can see what part of the request processing is causing issues.
  • By ELB Instance (using the elb field)
    • Total process time latency for by each instance
    • If you see an overall issue, you need to see what instances are having issues
  • By Backend Instance (using the backend:port field)
    • Total process time latency for by each instance
    • If you see an overall issue, you need to see what EC2 instances are having issues
  • By URL Domain (using the request field)
    • You usually don’t want to track statistics by unique URLs. You need to split the URL up. You can do that with this regex:
Example Sumo Logic Query
Total Latency by ELB Instance sourceCategory = “aws_elb”

| split _raw delim=’ ‘ extract 2 as elb, 5 as request_processing_time, 6 as backend_processing_time, 7 as response_processing_time

| avg(request_processing_time) as request_processing_time, avg(backend_processing_time) as backend_processing_time,avg(response_processing_time) as response_processing_time by elb

| (request_processing_time + backend_processing_time + response_processing_time) as total_process_time

Total Latency by Backend Instance _sourceCategory = “aws_elb”

| split _raw delim=’ ‘ extract 3 as backend, 5 as request_processing_time, 6 as backend_processing_time, 7 as response_processing_time

| avg(request_processing_time) as request_processing_time, avg(backend_processing_time) as backend_processing_time,avg(response_processing_time) as response_processing_time by backend

| (request_processing_time + backend_processing_time + response_processing_time) as total_process_time

Related Resources

Get Started Today!

Sign up for your FREE Sumo Logic Trial.

Sign Up Free

Request A Free Sumo Logic Demo

Fill out the form below and a Sumo Logic representative will contact you to schedule your free demo.
“Sumo Logic brings everything together into one interface where we can quickly scan across 1,000 servers and gigabytes of logs and quickly identify problems. It’s awesome software and awesome support.”

Jon Dokuli,
VP of Engineering

Thank you for signing up for Sumo Logic.

We are creating your account now.
Please check your email.
Need more help? Contact Us
Sign up for Sumo Logic Free*
Sign up for Sumo Logic Free*
  • No credit card required to sign-up
  • Create your account in minutes
  • No expiration date*
  • *After 30 day trial period, reverts to Sumo Logic Free
    • Please Enter your email address.
    • Please enter a valid email address.
    • This email is already in use for another account.
    • Please use your company email to create an account.
    • Please agree to the Service License.
    • Free trial provisioning is temporarily offline, please call 855-LOG-SUMO to get started.
    View All Pricing Options
    Already have an account? Login