Blog › Authors › Stefan Zier

Mitigating the Heartbleed Vulnerability

04.10.2014 | Posted by Stefan Zier, Lead Architect

skitched-20140410-094910By now, you have likely read about the security vulnerability known as the Heartbleed bug. It is a vulnerability in the widespread OpenSSL library. It allows stealing the information protected, under normal conditions, by the SSL/TLS encryption used to encrypt traffic on the Internet (including Sumo Logic).

How did we eliminate the threat?

When we were notified about the issue, we quickly discovered that our own customer-facing SSL implementation was vulnerable to the attack — thankfully, the advisory was accompanied by some scripts and tools to test for the vulnerability. Mitigation happened in four steps:

  1. Fix vulnerable servers. As a first step, we needed to make sure to close the information leak. In some cases, that meant working with third party vendors (most notably, Amazon Web Services, who runs our Elastic Load Balancers) to get all servers patched. This step was concluded once we confirmed that all of load balancers on the DNS rotation were no longer vulnerable.

  2. Replace SSL key pairs. Even though we had no reason to believe there was any actual attack against our SSL private keys, it was clear all of them had to be replaced as a precaution. Once we had them deployed out to all the servers and load balancers, we revoked all previous certificates with our CA, GeoTrust. All major browsers perform revocation checks against OCSP responders or CRLs.

  3. Notify customers. Shortly after we resolved the issues, we sent an advisory to all of our customers, recommending a password change. Again, this was a purely precautionary measure, as there is no evidence of any passwords leaking.

  4. Update Collectors. We have added a new feature to our Collectors that will automatically replace the Collector’s credentials. Once we complete testing, we will recommend all customers to upgrade to the new version. We also enabled support for certificate revocation checking, which wasn’t enabled previously.     

How has this affected our customers?

Thus far, we have not seen any signs of unusual activity, nor have we seen any customers lose data due to this bug. Unfortunately, we’ve had to inconvenience our customers with requests to change passwords and update Collectors, but given the gravity of the vulnerability, we felt the inconvenience was justified.

Internal impact

Our intranet is hosted on AWS and our users use OpenVPN to connect to it. The version of OpenVPN we had been running needed to be updated to a version that was released today. Other servers behind OpenVPN also needed updating.

Sumo Logic uses on the order of 60-70 SaaS services internally. Shortly after we resolved our customer facing issues, we performed an assessment of all those SaaS services. We used the scripts to test for the vulnerability combined with DNS lookups. If a service looked like it was hosted with a provider/service that was known to have been vulnerable (such as AWS ELB), we added it to our list. We are now working our way through the list and changing passwords on all affected applications, starting with the most critical ones. Unfortunately, manually changing passwords in all of the affected applications takes time and presents a burden on our internal IT users. We plan to have completed this process by the end of the week. 

Interesting Days

Overall, the past few days were pretty interesting on the internet. Many servers (as many as 66% of SSL servers on the net) are running OpenSSL, and most were affected. Big sites, including Yahoo Mail and many others were affected. The pace of exploitation and spread of the issue were both concerning. Thankfully, Codenomicon, the company that discovered this vulnerability, did an amazing job handling and disclosing it in a pragmatic and responsible fashion. This allowed everybody to fix the issue rapidly and minimize impact on their users.

AWS re:Invent – The future is now

12.05.2012 | Posted by Stefan Zier, Lead Architect

This past week, several of us had the pleasure of attending Amazon Web Service’s inaugural re:Invent conference in Las Vegas. In the weeks leading up to the conference, it wasn’t fully obvious to me just how big this show was going to be (not for lack of information, but mostly because I was focused on other things).

When I picked up my badge in the Venetian and walked through the enormous Expo hall, it struck me: IaaS, the cloud, is no longer a bleeding edge technology used by a few daring early adopters. The economics and flexibilities afforded are too big to be ignored – by anyone.

Attending sessions and talking to customers showed that, more than ever before, application architectures are distributed. The four design principles outlined in Werner Vogels excellent keynote on day 2 of the conference – Werner calls them “The Commandments of 21st Century Architectures” made it obvious that the cloud requires people to build their applications in fundamentally different ways from traditional on-premise applications.

While at the conference, I spent quite some time at the Sumo Logic booth, explaining and demoing our solutions to customers. Most of them run distributed systems in AWS, and it never took more than 2 minutes for them to realize why their lives would be much easier with a log management solution — having a central tool to collect and quickly analyze logs from a large distributed app is essential to troubleshooting, monitoring and optimizing an app.

Once I started to explain how our product is architected, and started relating to the architecture principles Werner outlined, most people understood why Sumo Logic’s product can scale and perform the way it does, unlike some other “cloud washed” solutions on the market.

In addition to having a highly relevant set of attendees for us, the conference also was a great place to find other vendors who exist in the same ecosystem – we’ve begun several good conversations about integrations and partnerships.

The response from people we talked to was overwhelmingly positive. During the conference, we’ve seen a big increase in sign-ups for our Sumo Logic Free product. We will definitely be back next year.

Work Here – Get a free MacBook Pro!

07.31.2012 | Posted by Stefan Zier, Lead Architect

In the last few years, many developers started to favor Mac OS X as a development platform. When you come work for Sumo Logic, we give you a top-of-the line MacBook Pro, and you get to keep it, for good. This post describes the terms of this offer and the rationale behind it.

… Continue Reading

Pragmatic AWS: 3 Tips to enhance the AWS SDK with Scala

07.12.2012 | Posted by Stefan Zier, Lead Architect

At Sumo Logic, most backend code is written in Scala. Scala is a newer JVM (Java Virtual Machine) language created in 2001 by Martin Odersky, who also co-founded our Greylock sister company, TypeSafe. Over the past two years at Sumo Logic, we’ve found Scala to be a great way to use the AWS SDK for Java. In this post, I’ll explain some use cases. 

1. Tags as fields on AWS model objects

Accessing AWS resource tags can be tedious in Java. For example, to get the value of the “Cluster” tag on a given instance, something like this is usually needed: 

   String deployment = null;
   for (Tag tag : instance.getTags()) {
     if (tag.getKey().equals(“Cluster”)) {
       deployment = tag.getValue();
     }
   }

While this isn’t horrible, it certainly doesn’t make code easy to read. Of course, one could turn this into a utility method to improve readability. The set of tags used by an application is usually known and small in number. For this reason, we found it useful to expose tags with an implicit wrapper around the EC2 SDK’s Instance, Volume, etc. classes. With a little Scala magic, the above code can now be written as:

val deployment = instance.cluster

Here is what it takes to make this magic work:

object RichAmazonEC2 {
 implicit def wrapInstance(i: Instance) = new RichEC2Instance(i)
}

class RichEC2Instance(instance: Instance) {
 private def getTagValue(tag: String): String =
   tags.find(_.getKey == tag).map(_.getValue).getOrElse(null)
 
 def cluster = getTagValue(“Cluster”)
}

Whenever this functionality is desired, one just has to import RichAmazonEC2._

2. Work with lists of resources

Scala 2.8.0 included a very powerful new set of collections libraries, which are very useful when manipulating lists of AWS resources. Since the AWS SDK uses Java collections, to make this work, one needs to import collections.JavaConversions._, which transparently “converts” (wraps implicitly) the Java collections. Here are a few examples to showcase why this is powerful: 

Printing a sorted list of instances, by name:
ec2.describeInstances(). // Get list of instances.
 getReservations.                  
 map(_.getInstances).
 flatten.                          // Translate reservations to instances.
 sortBy(_.sortName).               // Sort the list.
 map(i => “%-25s (%s)”.format(i.name, i.getInstanceId)). // Create String.
 foreach(println(_))               // Print the string.

Grouping a list of instances in a deployment by cluster (returns a Map from cluster name to list of instances in the cluster):
ec2.describeInstances().            // Get list of instances.
 filter(_.deployment = “prod”).    // Filter the list to prod deployment.
 groupBy(_.cluster)                // Group by the cluster.

You get the idea – this makes it trivial to build very rich interactions with EC2 resources.

3. Add pagination logic to the AWS SDK

When we first started using AWS, we had a utility class to provide some commonly repeated functionality, such as pagination for S3 buckets and retry logic for calls. Instead of embedding functionality in a separate utility class, implicits allow you to pretend that the functionality you want exists in the AWS SDK. Here is an example that extends the AmazonS3 class to allow listing all objects in a bucket: 

object RichAmazonS3 {
 implicit def wrapAmazonS3(s3: AmazonS3) = new RichAmazonS3(s3)
}

class RichAmazonS3(s3: AmazonS3) {
 def listAllObjects(bucket: String, cadence: Int = 100): Seq[S3ObjectSummary] = {

   var result = List[S3ObjectSummary]()

   def addObjects(objects: ObjectListing) = result ++= objects.getObjectSummaries

   var objects = s3.listObjects(new ListObjectsRequest().withMaxKeys(cadence).withBucketName(bucket))
   addObjects(objects)

   while (objects.isTruncated) {
     objects = s3.listNextBatchOfObjects(objects)
     addObjects(objects)
   }

   result
 }
}

To use this:

val objects = s3.listAllObjects(“mybucket”)

There is, of course a risk of running out of memory, given a large enough number of object summaries, but in many use cases, this is not a big concern.

Summary

Scala enables programmers to implement expressive, rich interactions with AWS and greatly improves readability and developer productivity when using the AWS SDK. It’s been an essential tool to help us succeed with  AWS.

Nine 1Password Power Tips

06.27.2012 | Posted by Stefan Zier, Lead Architect

As I mentioned in my previous blog post, many of us here are power users 1Password. Here are a few tricks we’ve assembled:

1. Sharing passwords with team members

There are those rare occasions where you cannot avoid sharing a password with somebody else on the team. For example, some older applications don’t support multiple users within a single account. For these cases, it is crucial to keep track of whom a password has been shared with. I use 1Password tags to do this. Whenever I share a password with somebody, I tag that entry “shared_with_name” — for example, “shared_with_joan”. This has two benefits:

  • When a person leaves the company, I can generate a Smart Folder to show me all the passwords I’ve shared with them, and change the passwords. Smart Folders are folders in 1Password that dynamically select entries that match a given set of criteria, for example, tags.   
  • When I need to change the password, I have a list of people to send  the new password to.

In order to send the passwords, a good method is to export them from 1Password and send them over a secure link (we use Skype for this). These files are plain text and contain the password in the clear, so care needs to be taken to only store them on encrypted drives and secure wipe them.  

2. Eradicating Bad/Old Passwords

All of us have used passwords on the internet before the widespread availability of password managers. Let’s be honest, we probably reused a small set of passwords broadly. Now that it is the year 2012 and we know better, 1Password can help us systematically eradicate our old passwords.

Here’s one good way to do it:

  • Create a Smart Folder in 1Password that matches against all your “bad” passwords.
  • Make it a habit to let the 1Password browser plugin remember your password, even if it’s one of your widely reused ones.
  • Regularly go through the Smart Folder and empty it out by changing your passwords on those sites.

Passwords that were exposed as part of any security breaches, for example your LinkedIn password, should also be part of the Smart Folder.   

3. Eradicating Weak Passwords

Similar to the previous tip, use a Smart Folder to match on password strength and length. In addition to that, use a special tag (for example, “must_be_weak”) to label passwords that cannot be strong (annoyingly, some sites limit password length and character set). Exclude these from the Smart Folder.  

4. Keyboard Shortcuts

There are a few handy keyboard shortcuts in the browser plugins for 1Password and 1Password itself. Here are some I use on the Mac:

  • In Chrome: Command-L,  “1p”, <tab> – this brings up a dropdown menu with all the Logins in your 1Password, start typing and select. Chrome will then take you to the site and log you in.
  • Most browsers on the Mac: Command-\ – this opens the 1Password browser plugin and fills in the credentials for the current site (after you select with Enter)
  • In 1Password App: Option – while Option is held, 1Password will reveal the password for the current entry.
  • In 1Password App: Command-F – takes you to the search box.     

5. Frequently Used Items

I have two passwords I have to enter into a terminal (i.e. not a browser) many times a day. Rather than looking them up using the search functionality, I’ve created a “Frequently Used” folder, which I leave selected throughout the day, saving precious keystrokes.

6. Password Length and Complexity

Recent research has shown that longer passwords with a standard (A-Z, a-z, 0-9) character set are preferable to short passwords that use additional special characters. In addition, many web sites are pretty fiddly about the passwords they require/allow. When I create a new password, here’s the rough sequence of steps I take:

  1. Does the form talk about any maximum length and/or restrictions? It not, default to the 50 character maximum (slider all the way right) and letters/numbers only.
  2. If there is a maximum length or the site mandates it and it is below 12 add all the special characters.
  3. Set the password, then try it out.

This works great 99% of the time. Whenever I encounter a case where there’s a bug, I add a note into 1Password to remember the quirks on the next password change. Here are two examples:

  • A mortgage servicing company accepted my 50 character password in the password reset form, but silently trimmed the password to 12 characters. I had to discover what my password had been set to through trial and error.
  • A big national discount brokerage (talk to Chuck…) restricts your password length to 8 characters. They don’t allow special characters, either. The only mitigating factor is that they lock out your account after 3 failed logins.

7. Security Questions

Many sites commonly force you to configure a set of “security questions” that are required as additional authentication in certain circumstances, including:

  • You’re trying to reset the password, and they authenticate you using the questions.
  • You’re logging in from a computer they haven’t seen before.

Aside from being tedious, there are a few other annoyances with security questions. Many sites limit you to a predefined set of questions to select from. Those generally fall into two categories:

  • Questions about your family/history (schools, cars, etc), which are easy for third parties to know or find out.
  • Questions about preferences (favorite color, actor, etc), which are generally non-constant or don’t apply (favorite color? why discriminate…)

Long story short: 1Password can help. Simply answer the questions with “secure” (non-guessable) values, then store the questions and answers in the additional fields in 1Password. Bonus: You can simply copy/paste the answer when the site asks next time.

8. Passwords you need to type

There are a few places where a fully random-generated password isn’t practical:

  • Passwords to unlock your laptop/desktop.
  • Passwords you need to enter on a TV screen (Tivo, Roku, AppleTV, etc)
  • Passwords to web sites or applications that won’t let you paste from the clipboard (how annoying is that!)

For those cases, there are two things you can do:

  1. Make up a nice long password from random words (ideally some not from a dictionary, use camel-case spelling, add numbers that aren’t your DOB, etc).
  2. Use 1Passwords “Pronouncable” feature, which renders passwords that are easier to copy by hand.   

I tend to find 1 easier. For good measure I add some expletives to the passwords for poorly designed sites… :)

9. 1Password and the real world

Here are some ideas for non-obvious uses for 1Password. Many of them only make sense if you have a smart phone with the 1Password app synched.

  • A secure note with the procedure to open your safe (left -> 38, right -> 4, etc.)
  • Random-generated PINs codes for your garage door opener, home alarm, car stereo, any exterior doors that use codes, etc.
  • Credit cards, passport numbers, social security numbers, bank account and routing numbers. 1Password has a Wallet section for this type of information.

People habitually re-use the  number combinations (often based on dates) for these types of access controls. Do you really want your friend whom you’ve given the alarm code to also know the PIN for your ATM account?

Security-Gain without Security-Pain

06.21.2012 | Posted by Stefan Zier, Lead Architect

As Joan mentioned, we use SaaS products a lot here at Sumo Logic. On an average day, I log into sites on the internet tens or even hundreds of times, supplying a username and password each time. The proliferation of usernames and passwords creates three issues:

  • Password hygiene. In this day and age, it is reckless  to reuse passwords across sites. At the same time, it is impossible to remember arbitrarily many unique passwords.
  • Strength. With rainbow tables and the tumbling cost of compute power, passwords need to be increasingly long and complex.
  • Efficiency. I shouldn’t have to spend half my day logging into sites.

What we need are tools that:

  • Encourage you to use different passwords everywhere.
  • Are secure, ideally using two factors of authentication.
  • Require the least number of keystrokes or mouse actions to get past login screens.

Here are a few tools we use and like.

1Password

1Password is the password manager most of us use. It stands out from many other password managers in several ways:

  • It is a native Mac application and has excellent integration. There is a version for Windows.
  • Support for iOS and Android.
  • Well-implemented sync via Dropbox, including for iOS.
  • Plugins for the 3 major browsers (Safari, Chrome, Firefox).
  • Keyboard compatible.

One of the major benefits of 1Password is that it’s designed to stay out of your way. To log into a site:

  • Without 1Password, I enter the URL in the address bar, navigate to the login form. Then, I enter my login, then my password. A lot of typing.
  • With 1Password, I enter 1pinto the address bar, start typing the site’s name to select from the list and hit enter. Then, I watch 1Password log me into the site.

Properly used, 1Password can be regarded as a one and a half factor authentication solution. There’s a great discussion on Agile Bits blog. We’ll share some power user tips on 1Password in the near future.

IronKeys

IronKeys are cool toys. They’re USB sticks with “spook-grade” crypto and self-destruction capabilities. We issue every developer an IronKey for the storage of all key files, such as ssh private keys and AWS credential files. Aside from being geek-chic, the IronKeys offer two benefits:

  • The key files are only exposed while the IronKey is plugged in and mounted. Not when people are at Starbucks browsing the web.
  • If an IronKey is ever lost, we can remote-detonate them. The minute they get plugged into a USB port, the software on the IronKey phones home and gets a self destruct signal. This requires an internet connection, but we’ve configured IronKeys to not unlock without one.  

OATH (Google Apps and AWS)

Retired Hardware MFA Tokens

Retired Hardware MFA Tokens

Google’s Two-Step Verification and Amazon Web Service’s MFA both use the OATH open architecture, not to be confused with OAuth. OATH is a software replacement for traditional hardware-based two-factor authentication tokens. 

Google offers open sourced client applications for iOS and Android that serve as the second factor of authentication. This reduces clutter, since you don’t need to carry any hardware tokens. Having the phone be your token also makes it more likely that you have your token with you most of the time.

Google has also taken several steps to remove friction:

  • To set up your phone, you simply scan a QR code form the screen.
  • After the first two factor authentication with your phone, you can check a box “Remember me for 30 days”. The browser cookie then serves as your second factor of authentication.

AWS initially only supported classical hardware MFA tokens. To make matters worse, one MFA token couldn’t be shared across multiple AWS accounts. More recently, they’ve also added support for OATH. In fact, the same Google Authenticator apps work for AWS, as well.

Wrapping up

Traditional two-factor authentication approaches based on hardware tokens are painful to use. OATH, 1Password and IronKeys strengthen security without adding too much pain to people’s lives.

Pragmatic AWS: Principle of Least Privilege with IAM

06.12.2012 | Posted by Stefan Zier, Lead Architect

Lock and Chain - by Martin Magdalene

One of the basic principles in information security is the Principle of Least Privilege. The idea is simple: give every user/process/system the minimal amount of access required to perform its tasks. In this post, I’ll describe how this principle can be applied to applications running in a cluster of EC2 instances that need access to AWS resources. 

What are we protecting?

The AWS Access Key ID and Secret are innocent looking strings. I’ve seen people casually toss them around scripts and bake them into AMIs. When compromised, however, they give the attacker full control over all of our resources in AWS. This goes beyond root access on a single box – it’s “god mode” for your entire AWS world! Needless to say, it is critical to limit both the likelihood of a successful attack and the exposure in case of a successful attack against one part of your application.

Why do we need to expose AWS credentials at all?

Since our applications run on EC2 instances and access other AWS services, such as S3, SQS, SimpleDB, etc, they need AWS credentials to run and perform their functions.

Limiting the likelihood of an attack: Protecting AWS credentials

In an ideal world, we could pass the AWS credentials into applications without ever writing them to disk and encrypt them in application memory. Unfortunately, this would make for a rather fragile system – after a restart, we’d need to pass the credentials into the application again. To enable automated restarts, recovery, etc., most applications store the credentials in a configuration file.

There are many other methods for doing this. Shlomo Swidler compared tradeoffs between different methods for keeping your credentials secure in EC2 instances.

At Sumo Logic, we’ve picked what Shlomo calls the SSH/On Disk method. The concerns around forgetting credentials during AMI creation don’t apply to us. Our AMI creation is fully automated, and AWS credentials never touch those instances. The AWS credentials only come into play after we boot from the AMI. Each application in our stack runs as a separate OS user, and the configuration file holding the AWS credentials for the application can only be read by that user. We also use file system encryption wherever AWS credentials are stored.

To add a twist, we obfuscate the AWS credentials on disk. We encrypt them using a hard-coded, symmetric key. This obfuscation, an additional Defense-in-Depth measure, makes it a little more difficult to get the plain text credentials in the case of instance compromise. It also makes shoulder surfing much more challenging. 

Limiting exposure in case of a successful attack: Restricted access AWS credentials

Chances are that most applications only need a very small subset of the AWS portfolio of services, and only a small subset of resources within them. For example, an application using S3 to store data will likely only need access to a few buckets, and only perform limited set of operations against them.

AWS’s IAM service allows us to set up users with limited permissions, using groups and policies. Using IAM, we can create a separate user for every application in our stack, limiting the policy to the bare minimum of resources/actions required by the application. Fortunately, the actions available in policies directly correspond to AWS API calls, so one can simply analyze which calls an application makes to the AWS API and derive the policy from this list.

For every application-specific user, we create a separate set of AWS credentials and store them in the application’s configuration file.

In Practice – Automate, automate, automate!

If your stack consists of more than one or two applications or instances, the most practical option for configuring IAM users is automation. At Sumo Logic, our deployment tools create a unique set of IAM users. One set of users per deployment and one user per application within the deployment. Each user is assigned a policy that restricts access to only those of the deployments resources that are required for the application.

If the policies changes, the tools update them automatically. The tools also configure per-application OS level users and restrict file permissions for the configuration files that contain the AWS credentials for the IAM user. The configuration files themselves store the AWS credentials as obfuscated strings.

One wrinkle in this scheme is that the AWS credentials created for the IAM users need to be stored somewhere after their initial creation. After the initial creation of the AWS credentials, they can never be retrieved from AWS again. Since many of our instances are short-lived, we needed to make sure we could use the credentials again later. To solve this particular issue, we encrypt the credentials, then store them in SimpleDB. The key used for this encryption does not live in AWS and is well-protected on hardware tokens.   

Summary

It is critical to treat your AWS credentials as secrets and assign point-of-use specific credentials with minimal privileges. IAM and automation are essential enablers to make this practical.  

Update (6/12/2012): AWS released a feature named IAM Roles for EC2 Instances today. It makes temporary a set of AWS credentials available via instance metadata. The credentials are rotated multiple times a day. IAM Roles add a lot of convenience, especially in conjunction with the AWS SDK for Java.

Unfortunately, this approach has an Achilles heel: any user with access to the instance can now execute a simple HTTP request and get a valid set of AWS credentials. To mitigate some of the risk, a local firewall, such as iptables, can be used to restrict HTTP access to a subset of users on the machine.

Comparing the two approaches 

+ User privileges and obfuscation offer a stronger defense in scenarios where a single (non-root) user is compromised.
+ Per-application (not per-instance) AWS credentials are easier to reason about.
- The rotation of IAM keys performed transparently by IAM roles adds security. An attacker has to maintain access to a compromised machine to maintain access to valid credentials.

Best of Both Worlds

AWS’s approach could be improved upon with a small tweak: Authenticate access to the temporary/rotating credentials T in instance metadata using another pair of credentials A. A itself would not have any privileges other than accessing T from within an instance. This approach would be a “best of both worlds”. Access to A could be restricted using the methods described above, but keys would still be rotated on an ongoing basis.   

Pragmatic AWS: Data Destroying Drones

06.05.2012 | Posted by Stefan Zier, Lead Architect

As we evolve our service, we occasionally delete EBS (Elastic Block Store) volumes. This releases the disk space back to AWS to be assigned to another customer. As a security precaution, we have decided to perform a secure wipe of the EBS volumes. In this post, I’ll explain how we implemented the wipe.

Caveats

Wiping EBS volumes may be slightly paranoid and not strictly needed, since AWS guarantees to never return a previous users data via the hypervisor (as mentioned in their security white paper). We also understand that the secure wipe is not perfect. EBS is able to move our data around in the background and leave back blocks that we didn’t wipe. Still, we felt that this additional precaution was worth the bit of extra work and cost – better safe than sorry.

Drones

We wanted to make sure secure wiping did not to have any performance impact on our production deployment. Therefore, we decided that it would be great to perform the secure wipe from a different set of AWS instances — Data Destroying Drones. We also wanted them to be fire-and-forget, so we wouldn’t have to manually check up on them.

To accomplish all this, we built a tool that:

  1. Finds to-be-deleted EBS volumes matching a set of tag values. (we tag the volumes to mark them for wiping).
  2. Launches one t1.micro instance per EBS volume that needs wiping (using an Ubuntu AMI).
  3. Passes a cloud-init script with Volume ID and (IAM limited) AWS credentials into the instance.

The Gory Details

Ubuntu has a mechanism named cloud-init. It accepts a shell script via EC2’s user data, which is passed in as part of the RunInstances API call to EC2. Here is the script we use for the Data Destroying Drones:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
#!/bin/bash
set -e
export INSTANCE_ID=`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`
export VOLUME_ID=v-12345678
export EC2_URL=https://ec2.us-east-1.amazonaws.com
export EC2_ACCESS_KEY=[key id]
export EC2_SECRET_KEY=[key]
 
sudo apt-get install scrub
euca-attach-volume -i $INSTANCE_ID -d /dev/sdj $VOLUME_ID
sudo scrub -b 50M -p dod /dev/sdj > ~/sdj.scrub.log 2>&1
sleep 30
 
euca-detach-volume $VOLUME_ID
euca-delete-volume $VOLUME_ID
halt
view raw destroy_volume.sh hosted with ❤ by GitHub

This script automates the entire process:

  1. Attach the volume.
  2. Perform a  DoD 5220.22-M secure wipe of the volume using scrub.
  3. Detach and delete the volume.
  4. Halt the instance.

The instances are configured to terminate on halt, which results in all involved resources to disappear once the secure wipe completes. The scrub can take hours or even days, depending on the size of the EBS volumes, but the cost for the t1.micro instances makes this a viable option. Even if the process takes 48 hours, it costs less than $1 to wipe the volume.

Summary

Aside from being a fun project the Data Destroying Drones have given us additional peace of mind and confidence that we’ve followed best practice and made a best effort to secure our customers data by not leaving any of it behind in the cloud.

Pragmatic AWS: 4 Ideas for using EC2 Tags

05.15.2012 | Posted by Stefan Zier, Lead Architect

At Sumo Logic, we use Amazon Web Services (AWS) for everything. Our product, as well as all our internal infrastructure live in AWS. In this series of posts, we’ll share some useful practices around using AWS. In the first installment, I’ll outline some useful things we do with tags

1. Organize resources

We’ve decided on a hierarchical way of managing our EC2 (Elastic Compute Cloud) resources:

Deployment
 + Cluster
   + Instance/Node

Within an AWS account, we can have multiple “deployments”. A deployment is a complete, independent copy of our product and uses the same architecture as our production service. Besides production, we use several smaller-scale deployments for development, testing and staging. Each deployment consists of a number of clusters, and each cluster of one or more instances.

Instances and their corresponding EBS (Elastic Block Store) volumes are tagged with Deployment, Cluster and Node tags. As an example, the third frontend node of our production deployment would be tagged like so:

Deployment=prod
Cluster=frontend
NodeNumber=3

There is also a direct mapping to DNS names. The DNS name for this node would be prod-frontend-3.

Combined with the filtering features in AWS Console (you can make any tag a column in the resource listings), this makes it very easy to navigate to a particular set of resources.

2. Display Instance Status

Tags can also be used as an easy way to display status information in the AWS console. Simply update a tag with the current status, whenever it changes.

The code that deploys our instances into EC2 updates a DeployStatus tag whenever it progresses from one step to another. For example, it could read:

2012-05-10 17:53 Installing Cassandra

This allows you to see what’s going on with instances at a glance.

3. Remember EBS Volume Devices

For EC2 instances that have multiple EBS volumes, when they need to be attached, our tools need to know which volume gets mapped to which device on the instance.

When we first create a volume, for example /dev/sdj, we create add a DeviceName tag to the volume with a value of /dev/sdj to track where it needs to be attached. Next time we attach the volume, we know it’s “proper place”.

4. Attribute and remind people of costs

All our developers are empowered to create their own AWS resources. This is a huge benefit for full-scale testing, performance evaluations, and many other use cases. Since AWS is not a charity, however, we need to manage costs tightly. In order to do this, we tag all AWS resources with an Owner tag (either by hand, or via our automated deployment tool).

To consume this tag, we have a cron job that runs daily and emails users who have active resources in AWS to remind them to shut down what they no longer require.

The subject line of the email reads “[AWS] Your current burn rate is $725.91/month!”. The body of the email contains a table with a more detailed cost breakdown. In addition, there is also a rollup email that goes out to the entire development team.

 

Summary

EC2 tags are extremely useful tools to track state, organize resources and store relationships between resources like instances and EBS volumes. There are a myriad more ways to use them. I hope these tips have been helpful.

Sumo on Sumo, Part 2: User Signups

05.01.2012 | Posted by Stefan Zier, Lead Architect

In part 1, we mentioned that we’re big on “dogfooding”. In this short post we’ll run you through a very simple use case we solve with our Sumo Logic product: When we launched in January, everybody here was extremely excited, and we wanted to know who signed up for our demo. Solving this problem with the product was a matter of 2 minutes.

The component responsible for account creation logs a line like the one below whenever somebody signs up for the demo: 

2012-04-13 10:31:58,917 -0700 INFO  [hostId=prod-bill-1] [module=BILL] [logger=bill.signup.DemoAccountRequestHandler] [thread=btpool0-19] [10.10.10.10] Successful request: Stefan Zier (stefan+blog@sumologic.com) https://www.sumologic.com/free-trial/?email=stefan%2Bblog%40sumologic.com&firstName=Stefan&lastName=Zier&organizationName=Sumo&key=nnnnnnnnnnnnnnnnnn

Looking carefully, this has all the information we’re interested in. Let me walk you through the process of building up the search, refining it iteratively. First, let’s find the lines we’re interested in by picking two keywords we think uniquely identify this type of message:

DemoAccountRequestHandler Successful

Ok, this gives us the lines. Now, let’s parse out the first, last name and email:

DemoAccountRequestHandler Successful | parse “request: * (*)” as name, email

Now we have the data we care about, all parsed out. Note the simplified regular expression in the parse section of the query. Simply find a good anchor (“request:” in this case) and put * for the values you want parsed out. Looking at the results, we see that our own sign ups for testing and QA are included. Let’s get rid of them.

DemoAccountRequestHandler Successful !sumologic.com | parse “request: * (*)” as name, email  

The only task that remains is to schedule it. Let’s say we want the results sent to employees every hour. We only want an email when there are results, not at 4am, when people are asleep. We save the search, schedule it, and configure the conditions for the schedule, as well as the destination of the notification:

That’s all – everybody at Sumo Logic now gets an hourly summary of demo signups! Here is what the email looks like (ok, I cheated and removed the “!sumologic.com” for this demo):

Twitter