VPC flow logging

What is VPC flow logging?

Virtual Private Cloud (VPC) Flow logging provides built-in power to monitor information about how your network resources are operating in Amazon Web Services.

VPC Flow logging lets you capture and log data about network traffic in your VPC. VPC Flow logging records information about the IP data going to and from designated network interfaces, storing this raw data in Amazon CloudWatch, where it can be retrieved and viewed.

Click here to learn more about AWS VPC Firewall.

Uses for VPC logging

Rather than the old days of collecting this critical data through add-on applications and services—which add overhead and use computing power—Amazon has brought native flow AWS monitoring to the cloud. It is the equivalent of NetFlow monitoring in the on-premises world. VPC Flow logging is critical for security and compliance in your AWS cloud environment.

Use VPC flow logs to identify latencies, establish performance baselines and tweak applications. VPC flow logs can reveal flow duration, latency, and bytes sent, allowing you to identify performance issues and deliver a better user experience quickly. Security. By logging all of the traffic from a given interface or an entire subnet, root cause analysis can reveal critical gaps in security where malicious traffic is moving around your network. Key in on suspicious traffic and tighten security loopholes using VPC flow log data information.

A VPC flow logging dashboard from Sumo Logic

Catch the flow: enable VPC logging

By default, you will have to enable VPC. There are two different methods for turning on logging and capturing your network flow logs in Amazon Cloudwatch:

Flow logging can also be enabled and configured for more advanced users from the AWS Command Line Interface (CLI), a unified scripting tool for managing your AWS services.

Use the AWS Management Console to enable and configure VPC Flow logs.

Though enabling flow logs for every resource on your network may be tempting, do so judicially. Flow logs can quickly swell into hundreds of gigabytes, and this mountain of data has a capture and storage fee. Work with your DevOps/operations team to determine what flow logs are beneficial and check Amazon Cloudwatch pricing to plan your budget.

Click here to learn more about AWS Traffic Mirroring.

Three kinds of flow logs

After enabling VPC Flow logging in AWS, it’s important to understand what you’re monitoring and how the logs compile data. Amazon offers flow logging at three separate levels:

Virtual private cloud

Monitor all the activity within your cloud environment for a bird’s eye view of your operations but note the pricing above. Analysis of VPC logging should reveal popular or vulnerable resources to watch closely moving forward.

Subnet

VPCs are often divided into subnets spanning multiple availability zones in the region. Subnets can be private or public subnet. Private subnets isolate internal resources from public-facing traffic, among other uses. Public subnets require an elastic IP to communicate to the Internet. Create a flow log for a specific subnet where you may want to monitor all activity. In this example, you want to monitor flow logs to ensure no internet traffic goes to the private subnet.

Network Interface

One can monitor specific interfaces on AWS EC2 instances and capture flow logs from an interface. Capture full flow logs from critical connection points in your network to stay ahead of issues like latency and malicious intrusions.

After choosing what resources you will log, define the logging parameters. These include:

• Traffic type: You can filter by all, accepted, or rejected traffic.
• Log name and destination: Specify a functional name for the log and where to store it in CloudWatch.
• Necessary permissions: Ensure the log owner has identity access management (IAM) privileges to publish and work with the flow log.

After setting up a flow log for a given resource, scaling is simple. The rules you outline will automatically replicate to additional instances, saving you time and trouble duplicating flow logs.

Limits to the flow

VPC flow logs can’t capture everything. VPC flow logs exclude certain types of traffic. Here are a few instances where you can’t rely on VPC logging:

• DNS traffic

You can log request resolution traffic if you’re running your DNS server. But many users rely on internal AWS DNS servers, and VPC flow logs will not capture activity between the servers and AWS DNS services.

• DHCP

Similarly, dynamic host configuration protocol (DHCP) traffic is not recorded. Depending on the size of your VPC, this can represent a notable amount of traffic.

• Multiple IP Addresses

Sometimes a virtual NIC will pool IP addresses for better performance. Flow logs only display traffic on the primary address.

• Legacy limitations

AWS instances before December 2013 running in the EC2 Classic format are incompatible with VPC Flow logging. Consider migrating to the current AWS format.

Get your VPC flowing with Sumo Logic

With VPC Flow logging, Amazon adds a powerful deep analysis tool for your AWS cloud, including in a DevOps environment. Knowing how to turn it on, what critical data to collect, and what you can’t find in your VPC logs is a step in the right direction toward mastering VPC logging.

Integrating directly with Google Stackdriver, Sumo Logic provides real-time observability for your GCP-generated log data. With the Sumo Logic app for Google Cloud VPC, gain real-time insights and analytics into network activity through interactive, customizable dashboards. You can look for unusual traffic patterns and suspicious activity with outlier detection.