Packer and Sumo Logic – Build Monitoring Into Your Images

Whether you’re new to automating your image builds with Packer, new to Sumo Logic, or just new to integrating Packer and Sumo Logic, this post guides you through creating an image with Sumo Logic baked in. We’ll use AWS as our cloud provider, and show how to create custom machine images in one command that allow you to centralize metrics and logs from applications, OSs, and other workloads on your machines.

Overview

When baking a Sumo Logic collector into any machine image, you’ll need to follow three main steps:

First, create your sources.json file, and add it to the machine.

• This file specifies what logs and metrics you’d like to collect
• It’s usually stored at /etc/sources.json, although you can store it anywhere at point to it

Next, download, rename, and make the collector file executable.

• Collector downloads for various operating systems and Sumo Logic deployments can be found here
• An example command might look like:
  sudo wget 'https://collectors.us2.sumologic.com/rest/download/linux/64' -O SumoCollector.sh && sudo chmod +x SumoCollector.sh

Finally, run the install script and skip registration.

• The most important part here is to use the -VskipRegistration=true flag so that the collector doesn’t register to the temporary machine you are trying to built the image with
• Other important flags include
  • -q > Run the script in quiet mode
  • -Vephemeral=true > This tells Sumo Logic to auto-remove old collectors that are no longer alive, usually applicable for autoscaling use cases where VMs are ephemeral
  • -Vsources=/etc/sources.json > Point to the local path of your sources.json file
  • -Vsumo.accessid=<id> -Vsumo.accesskey=<key> > This is your Sumo Logic access key pair
  • See all installation options here
• An example command might look like:
  sudo ./SumoCollector.sh -q -VskipRegistration=true -Vephemeral=true -Vsources=/etc/sources.json -Vsumo.accessid=<id> -Vsumo.accesskey=<key>

Packer and Sumo Logic – Provisioners

Packer Provisioners allow you to communicate with third party software to automate whatever tasks you need to built your image. Some examples of what you’d use provisioners for are:

• installing packages
• patching the kernel
• creating users
• downloading application code

In this example, we’ll use the Packer Shell Provisioner, which provisions your machine image via shell scripts. The basic steps that Packer will execute are:

1. Start up an EC2 instance in your AWS account
2. Download your sources.json file locally, which describes the logs and metrics you’d like to collect
3. Download the Sumo Logic collector agent
4. Run the collector setup script to configure the collector, while skipping registration (this creates a user.properties config file locally)
5. Create the AMI and shut down the EC2 instance
6. Print out the Amazon Machine Image ID (AMI ID) for your image with Sumo baked in

Instructions: Packer and Sumo Logic Build

Before You Begin

To ensure Packer can access your AWS account resources, make sure you have an AWS authentication method to allow Packer to control AWS resources:

• Option 1: User key pair
• Option 2: Set up the AWS CLI or SDKs in your local environment

I have chosen option 2 here so my Packer build command will not need AWS access key pair information.

After setting up your local AWS authentication method, create a Sumo Logic free trial here if you don’t already have an account. Then, generate a Sumo Logic key pair inside you Sumo Logic account. Copy this key down, as the secret key will only be shown once.

Step 1 – Get Your Files

After downloading Packer, download the Packer+Sumo_template.json and the packer_variables.json files, and place all 3 in the same directory.

Step 2 – Customize Variables and Test Your Template

Use the command ./packer validate packer_sumo_template.json to validate your packer template.

• This template automatically finds the latest Amazon Linux image in whatever region you use, based on the source_ami_filter in the builders object:"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "amzn-ami-hvm-????.??.?.x86_64-gp2",
"root-device-type": "ebs"
},
"owners": ["amazon"],
"most_recent": true
}

• Customize the Region in the packer_variables.json file to the AWS Region you want to build your image in
• You can also change the Sumo collector download URL if you are in a different deployment
• The sources.json file url can be updated to point to your own sources.json file, or you can update the template to use the Packer File Provisioner to upload your sources.json file, and any other files

Step 3 – Build Your Image

Use the command

./packer build -var-file=packer_variables.json -var 'sumo_access_id=<sumo_id>' -var 'sumo_access_key=<sumo_key>' packer_sumo_template.json

to build your image. You should see the build start and finish like this:

Image Build Start

Image Build Finish

Done! Now that you’ve integrated Packer and Sumo Logic, you can navigate to the AMI section of the EC2 AWS console and find the image for use in Autoscaling Launch Configurations, or just launch the image manually.

Now What? View Streaming Logs and Metrics!

Install the Sumo Logic Applications for Linux and Host Metrics to get pre-built monitoring for your EC2 Instance:

What Else Can Sumo Logic Do?

Sumo Logic collects AWS CloudWatch metrics, CloudTrail audit data, and much more. Sumo Logic also offers integrated Threat Intelligence powered by CrowdStrike, so that you can identify threats in your cloud infrastructure in real time. See below for more documentation:

• AWS CloudTrail
• AWS CloudWatch Metrics
• Integrated Threat Intelligence

What’s Next?

In part 3 of this series (will be linked here when published), I’ll cover how to deploy an Autoscaling Group behind a load balancer in AWS. We will integrate the Sumo Logic collector into each EC2 instance in the fleet, and also log the load balancer access logs to an S3 bucket, then scan that bucket with a Sumo Logic S3 source.

If you have any questions or comments, please reach out via my LinkedIn profile, or via our Sumo Logic public Slack Channel: slack.sumologic.com (@grahamwatts-sumologic).

Thanks for reading!