Nine 1Password Power Tips

06.27.2012 | Posted by Stefan Zier, Cloud Infrastructure Architect

As I mentioned in my previous blog post, many of us here are power users 1Password. Here are a few tricks we’ve assembled:

1. Sharing passwords with team members

There are those rare occasions where you cannot avoid sharing a password with somebody else on the team. For example, some older applications don’t support multiple users within a single account. For these cases, it is crucial to keep track of whom a password has been shared with. I use 1Password tags to do this. Whenever I share a password with somebody, I tag that entry “shared_with_name” — for example, “shared_with_joan”. This has two benefits:

  • When a person leaves the company, I can generate a Smart Folder to show me all the passwords I’ve shared with them, and change the passwords. Smart Folders are folders in 1Password that dynamically select entries that match a given set of criteria, for example, tags.   
  • When I need to change the password, I have a list of people to send  the new password to.

In order to send the passwords, a good method is to export them from 1Password and send them over a secure link (we use Skype for this). These files are plain text and contain the password in the clear, so care needs to be taken to only store them on encrypted drives and secure wipe them.  

2. Eradicating Bad/Old Passwords

All of us have used passwords on the internet before the widespread availability of password managers. Let’s be honest, we probably reused a small set of passwords broadly. Now that it is the year 2012 and we know better, 1Password can help us systematically eradicate our old passwords.

Here’s one good way to do it:

  • Create a Smart Folder in 1Password that matches against all your “bad” passwords.
  • Make it a habit to let the 1Password browser plugin remember your password, even if it’s one of your widely reused ones.
  • Regularly go through the Smart Folder and empty it out by changing your passwords on those sites.

Passwords that were exposed as part of any security breaches, for example your LinkedIn password, should also be part of the Smart Folder.   

3. Eradicating Weak Passwords

Similar to the previous tip, use a Smart Folder to match on password strength and length. In addition to that, use a special tag (for example, “must_be_weak”) to label passwords that cannot be strong (annoyingly, some sites limit password length and character set). Exclude these from the Smart Folder.  

4. Keyboard Shortcuts

There are a few handy keyboard shortcuts in the browser plugins for 1Password and 1Password itself. Here are some I use on the Mac:

  • In Chrome: Command-L,  “1p”, <tab> – this brings up a dropdown menu with all the Logins in your 1Password, start typing and select. Chrome will then take you to the site and log you in.
  • Most browsers on the Mac: Command-\ – this opens the 1Password browser plugin and fills in the credentials for the current site (after you select with Enter)
  • In 1Password App: Option – while Option is held, 1Password will reveal the password for the current entry.
  • In 1Password App: Command-F – takes you to the search box.     

5. Frequently Used Items

I have two passwords I have to enter into a terminal (i.e. not a browser) many times a day. Rather than looking them up using the search functionality, I’ve created a “Frequently Used” folder, which I leave selected throughout the day, saving precious keystrokes.

6. Password Length and Complexity

Recent research has shown that longer passwords with a standard (A-Z, a-z, 0-9) character set are preferable to short passwords that use additional special characters. In addition, many web sites are pretty fiddly about the passwords they require/allow. When I create a new password, here’s the rough sequence of steps I take:

  1. Does the form talk about any maximum length and/or restrictions? It not, default to the 50 character maximum (slider all the way right) and letters/numbers only.
  2. If there is a maximum length or the site mandates it and it is below 12 add all the special characters.
  3. Set the password, then try it out.

This works great 99% of the time. Whenever I encounter a case where there’s a bug, I add a note into 1Password to remember the quirks on the next password change. Here are two examples:

  • A mortgage servicing company accepted my 50 character password in the password reset form, but silently trimmed the password to 12 characters. I had to discover what my password had been set to through trial and error.
  • A big national discount brokerage (talk to Chuck…) restricts your password length to 8 characters. They don’t allow special characters, either. The only mitigating factor is that they lock out your account after 3 failed logins.

7. Security Questions

Many sites commonly force you to configure a set of “security questions” that are required as additional authentication in certain circumstances, including:

  • You’re trying to reset the password, and they authenticate you using the questions.
  • You’re logging in from a computer they haven’t seen before.

Aside from being tedious, there are a few other annoyances with security questions. Many sites limit you to a predefined set of questions to select from. Those generally fall into two categories:

  • Questions about your family/history (schools, cars, etc), which are easy for third parties to know or find out.
  • Questions about preferences (favorite color, actor, etc), which are generally non-constant or don’t apply (favorite color? why discriminate…)

Long story short: 1Password can help. Simply answer the questions with “secure” (non-guessable) values, then store the questions and answers in the additional fields in 1Password. Bonus: You can simply copy/paste the answer when the site asks next time.

8. Passwords you need to type

There are a few places where a fully random-generated password isn’t practical:

  • Passwords to unlock your laptop/desktop.
  • Passwords you need to enter on a TV screen (Tivo, Roku, AppleTV, etc)
  • Passwords to web sites or applications that won’t let you paste from the clipboard (how annoying is that!)

For those cases, there are two things you can do:

  1. Make up a nice long password from random words (ideally some not from a dictionary, use camel-case spelling, add numbers that aren’t your DOB, etc).
  2. Use 1Passwords “Pronouncable” feature, which renders passwords that are easier to copy by hand.   

I tend to find 1 easier. For good measure I add some expletives to the passwords for poorly designed sites… :)

9. 1Password and the real world

Here are some ideas for non-obvious uses for 1Password. Many of them only make sense if you have a smart phone with the 1Password app synched.

  • A secure note with the procedure to open your safe (left -> 38, right -> 4, etc.)
  • Random-generated PINs codes for your garage door opener, home alarm, car stereo, any exterior doors that use codes, etc.
  • Credit cards, passport numbers, social security numbers, bank account and routing numbers. 1Password has a Wallet section for this type of information.

People habitually re-use the  number combinations (often based on dates) for these types of access controls. Do you really want your friend whom you’ve given the alarm code to also know the PIN for your ATM account?

IT Insights for All: Sumo Logic Launches Free Machine Data Search and Analytics Service

06.25.2012 | Posted by Vance Loiselle, CEO

 

 What is Free Machine Data Search and Analytics? 

Today we announce Sumo Logic Free, the first and only free, enterprise-class machine data search and analytics service. The free version of Sumo Logic’s cloud-based service offers full functionality, including:

  • Loading, indexing and archival of machine data
  • Distributed search
  • Real-time analytics
  • Proactive monitoring and alerting
  • Reporting and visualization
  • Role-based access controls

Our cloud-based approach eliminates the need for expensive premise-based solutions that require upfront investments in hardware and software as well the headache of constant management and upgrades .  The Sumo Logic service delivers a petabyte-scale platform that provides companies with access to valuable operational insights from their machine data—all in real time.

 

Why Are Companies Embracing Machine Data?

The volume, velocity and variety of machine data  (logs, events, configuration changes, clickstream, etc.) being generated by applications, networks, servers, and mobile devices is overwhelming IT organizations.  Our goal is to give every enterprise a way to search and analyze this machine data to troubleshoot application problems, proactively monitor performance and availability, and gain valuable operational and customer insights. 

For those companies just initiating the search for the right approach to handling their Big Data, we provide a purpose-built, scalable Big Data service to harness all the great information IT organizations have at their disposal.  We leverage our patent-pending Elastic Log Processing ™ platform to intelligently scale and tune the service as the enterprise grows, so IT organizations can focus on delivering value to the business.

Free Service =  Win – Win. 

We know that once customers start using our service, they’ll quickly see the power and value of the insights they can glean, and will want to use it for more use cases and with more data.  In addition, as more customers use our service in interesting and strategic ways, we’ll be able to apply their insights for the benefits of future customers and for future product development.   

 

Security-Gain without Security-Pain

06.21.2012 | Posted by Stefan Zier, Cloud Infrastructure Architect

As Joan mentioned, we use SaaS products a lot here at Sumo Logic. On an average day, I log into sites on the internet tens or even hundreds of times, supplying a username and password each time. The proliferation of usernames and passwords creates three issues:

  • Password hygiene. In this day and age, it is reckless  to reuse passwords across sites. At the same time, it is impossible to remember arbitrarily many unique passwords.
  • Strength. With rainbow tables and the tumbling cost of compute power, passwords need to be increasingly long and complex.
  • Efficiency. I shouldn’t have to spend half my day logging into sites.

What we need are tools that:

  • Encourage you to use different passwords everywhere.
  • Are secure, ideally using two factors of authentication.
  • Require the least number of keystrokes or mouse actions to get past login screens.

Here are a few tools we use and like.

1Password

1Password is the password manager most of us use. It stands out from many other password managers in several ways:

  • It is a native Mac application and has excellent integration. There is a version for Windows.
  • Support for iOS and Android.
  • Well-implemented sync via Dropbox, including for iOS.
  • Plugins for the 3 major browsers (Safari, Chrome, Firefox).
  • Keyboard compatible.

One of the major benefits of 1Password is that it’s designed to stay out of your way. To log into a site:

  • Without 1Password, I enter the URL in the address bar, navigate to the login form. Then, I enter my login, then my password. A lot of typing.
  • With 1Password, I enter 1pinto the address bar, start typing the site’s name to select from the list and hit enter. Then, I watch 1Password log me into the site.

Properly used, 1Password can be regarded as a one and a half factor authentication solution. There’s a great discussion on Agile Bits blog. We’ll share some power user tips on 1Password in the near future.

IronKeys

IronKeys are cool toys. They’re USB sticks with “spook-grade” crypto and self-destruction capabilities. We issue every developer an IronKey for the storage of all key files, such as ssh private keys and AWS credential files. Aside from being geek-chic, the IronKeys offer two benefits:

  • The key files are only exposed while the IronKey is plugged in and mounted. Not when people are at Starbucks browsing the web.
  • If an IronKey is ever lost, we can remote-detonate them. The minute they get plugged into a USB port, the software on the IronKey phones home and gets a self destruct signal. This requires an internet connection, but we’ve configured IronKeys to not unlock without one.  

OATH (Google Apps and AWS)

Retired Hardware MFA Tokens

Retired Hardware MFA Tokens

Google’s Two-Step Verification and Amazon Web Service’s MFA both use the OATH open architecture, not to be confused with OAuth. OATH is a software replacement for traditional hardware-based two-factor authentication tokens. 

Google offers open sourced client applications for iOS and Android that serve as the second factor of authentication. This reduces clutter, since you don’t need to carry any hardware tokens. Having the phone be your token also makes it more likely that you have your token with you most of the time.

Google has also taken several steps to remove friction:

  • To set up your phone, you simply scan a QR code form the screen.
  • After the first two factor authentication with your phone, you can check a box “Remember me for 30 days”. The browser cookie then serves as your second factor of authentication.

AWS initially only supported classical hardware MFA tokens. To make matters worse, one MFA token couldn’t be shared across multiple AWS accounts. More recently, they’ve also added support for OATH. In fact, the same Google Authenticator apps work for AWS, as well.

Wrapping up

Traditional two-factor authentication approaches based on hardware tokens are painful to use. OATH, 1Password and IronKeys strengthen security without adding too much pain to people’s lives.

Cry “Havoc!” and let slip the Sumos of war

06.19.2012 | Posted by Megha Bangalore, Software Engineer

While there is no outward sign of it, Sumo Towers (605 Castro) is not a building for the faint of heart.

The ground floor is where we devs have relegated all our ‘evil, as they can pull off wearing a suit without looking like poseurs’ salespeople and marketing staff. It lulls you into a false sense of security – surely, you tell yourself, this is just an office like any other. The door to Amnesty is closed, after all, so how are you to know that it is our own version of Las Vegas? What happens in Amnesty…

The stairwell leading up to the second floor – The Realm of the Devs – does not have any signs saying “Abandon Hope, all ye who enter here” or even “Beware the leopard,” but just the gentle admonishment, “Employees Only.”

It opens onto a bright and open area, with many desks, each covered in monitors – where there had initially been dark twisty passages, and poor lighting, there is now a large room, with delicate (and in fact slightly load-bearing) pillars, painted with strips of steel blue, grey and white. The result of an intense period of bashing through walls, which this particular sumo is saddened to have missed – after all, when is swinging around a sledgehammer not fun?

There is an almost overwhelming feeling of ‘quiet.’ The soft sounds of the clacking of keyboard keys, the hum of music seeping through oversized headphones.

This quiet has misled many a visitor - after all, it is only occasionally punctuated with violent thrashings as the writer is either struck by a blinding flash of inspiration or finally, woefully, temporarily beaten down, expressing a wordless Hulk-ism with the well-known and recognized Keyboard SMASH.

But of course, all in the room know that the keyboard smash is a call to arms! Foam bats are produced, from their easily accessible but very discrete spots, lined up against innocent-seeming shelves and desk frames.

The artillery units have been gathering their ammo since the last battle – in the form of railing angry birds or squirrels, or soft squishy sumos – and their eyes dart around, arms launching missiles with barely a pause and which, after a few years of practice, often strike true.

The rubber band guns, bought on a whim at a crafts festival on Castro, have devolved into diversions versus actual weapons – the rubber bands long since lost to the chaos of previous wars, they still cause the rampaging hordes to pause in their slaughter, which is often enough to turn the tide of any war.

As with any hands on battle, there are the berserkers – huge hulking wielders of destruction. Fortunately for sumo, and for the monitors and coffee mugs, there is only one Panda. Sumo Towers quite literally trembles at his commands (and jumps) and none are safe when he is released – friend and foe alike are subject to his violent whims, and allegiances last no longer than the time it takes to acknowledge them.

The battle rages on – has it really only been a few minutes?? – and the few remaining innocent bystanders get to a half-crouch out of their seats, and indecision flows over their faces… To enter the fray, or hope that the collateral damage will exclude them. Too late, they discover what all the combatants already knew – When Sumos War, Nowhere is Safe.

This battle is made safer in the knowledge that David, our Platform team manager, has left his sigil of authority at home – as is appropriate for wrangling such a motley crew, on his promotion he was presented with a bright pink riding crop – its purpose  (either punishment or motivation) truly depends on the particular quirks of developer it’s wielded against.

True personalities are revealed – the quiet and gentle Yan shows his inner not-so-secret maniacal genius; Krimy is normally a quiet and gentle UI dev – but when the battle heats she is the first to jump right into the thick of it, releasing angry birds with equally angry battle cries; Joan attacks with an aggressive rapier-like bat style, squawking to punctuate each successful hit; Yong plays the puppet master, the devil on a shoulder, whispering evil suggestions to all who are weak to his words. This author herself, who is obviously of a generally delicate and shy character, will rally to defend herself against the injustices of the ‘evil jerks’ that cannot resist taunting her with their longer reach.Stealthily Hidden Bat

And all at once, with a synchronization that belies the close familiarity of these particular warriors, weapons are lowered. A few manic giggles escape, and there is a brief pause as everyone silently acknowledges that ‘I just won,’ and compares battle scars.

The quiet that follows is similar to that from before the fight in only the most superficial ways – the focus which had been slowly drifting returns, and the programmers-turned-warriors revert to their more usual selves, and all that can be heard is the soft clacking of keyboard keys, with renewed purpose and enthusiasm.

Some of Our Essential Service Providers

06.15.2012 | Posted by Joan Pepin, Director of Security

As I mentioned in one of my previous posts, here at Sumo Logic we believe cloud-based services provide excellent value due to their ease of setup, convenience and scalability, and we leverage them extensively to provide internal services that would be far more time, labor and cash intensive to manage ourselves. Today I’m going to talk about some of the services we use for collaboration, operations and I/T, why we use them, and how they simplify our lives. 

Campfire

Campfire is a huge part of our productivity and culture at Sumo Logic. While I would lump this and Skype together under something like “Managed Corporate Messaging” they fill two very different niches in our environment.

Campfire from 37 Signals is a fantastic tool for group conversations. Using the Campfire service, we have set up multiple chat rooms for various types of issues, including Production Issues, Development Issues, Sales/Customer-Support Issues, and of course, a free-for-all chat-room where we try to make one another spontaneously erupt into chaotic LOLs.

These group-chats provide a critical space where we can work together to troubleshoot and solve problems cooperatively. Campfire makes it very easy to upload pictures and share large amounts of information in real-time with co-workers who can be anywhere. The conversations are all archived for later reference, which allows us to use the Production Incidents room as a 24×7 conference call and canonical forum of record for anything happening to production systems. Our Production on-call devs are expected to echo their actions into the Production channel and keep up with events there as they transpire.

Campfire also has a cool feature which allows you to start a voice conference with participants if needed, which is a great option in certain situations. These calls can also be archived for later reference. One down side to the text and audio archives is that they are not easily searchable so it helps to know approximately when something happened, and we have found it necessary to consult other records to determine where to look.

Skype

Skype is, of course, the very popular IM and VOIP service that was purchased by Microsoft a while back. We use Skype extensively for 1:1 chatting and easy and secure file-transfers throughout the company. We also make extensive use of the wide array of available emoticons. (Stefan Zier is a particularly prolific and artistic user of these.)


We also use Skype video chat for interviews and to collaborate with team members abroad. We have a conference room with a TV and Skype camera just for this application.

Cloudkick

Running a large-scale cloud-based service requires a lot of operational awareness. One of the ways we achieve this is through Cloudkick. Cloudkick was recently acquired by Rackspace and is evolving into Rackspace Cloud Monitoring. We are still on the legacy Cloudkick service, which we have come to use heavily.
We automatically install Cloudkick agents on all of our production instances and use them to collect a wide array of status codes from the O/S and through JMX as well as by running our own custom scripts which we use to check for the existence of critical processes and to detect if things like HPROF files exist.

The Cloudkick website has a “show only failures” mode which we call the “What’s Wrong? Page”. This is a very helpful tool that allows our EverybodyOps team to quickly assess issues with our production environment.

PagerDuty

Of course, we also need to be proactively alerted to failures and crossed thresholds that could indicate trouble, and for this we rely on PagerDuty. (Affectionately known as P. Diddy to many of us, nickname coined by Christian). PagerDuty is another great tool which allows us to maximize the benefits of our EverybodyOps culture.

Accidental Escalation!

Within PagerDuty we have a number of on-call rotations. One for our Production Primary role and one for the Secondary role, as well as another role for monitoring test failures and a lesser-known role for those of us who monitor the temperature in the one small server room we do have. P. Diddy allows us easily cover for each other using exceptions or by simply switching the Primary and Secondary roles on the fly if the Primary needs to go AFK for a while.

P. Diddy allows each user to set their own personal escalation policy which can include texting, calling, and emailing with a configurable number of re-tries and timeouts. Another nice touch is that the rotation calendars can be imported into our personal calendars to remind us of when we are up next. This all makes the on-call rotation run pretty flawlessly from an administrative perspective with no gnarly configuration and management on our end.

I must admit, I do have a personal habit of “Joaning” my secondary when I am on call… To properly “Joan” your secondary you accidentally escalate an alert to them that you meant to resolve, (I blame the comma after “Resolv”!)

Google Apps

Like many companies of all sizes we rely on Google for our email service. While some Sumos (like myself and Stefan) use mail clients to read our email, most Sumos are happy with the standard web interface from Google. We also heavily use internal groups for team communications.

We also make good use of Google Docs for document authoring and sharing (this blog post was written and communally edited using Google Docs, in fact, due to the impressive real-time collaboration, Stefan Zier is watching me add this bit in order to resolve his comment right now!)  We use Google Calendar for our scheduling needs (and calendar-stalking exercises!)

We also use Google Analytics to obsess over you.

Also, as Sumo Logic’s Director of Security, (which makes me partially responsible for managing the users and groups in Google Apps) I appreciate the richness of their security settings and especially the two-factor authentication and mobile device policy management.

There’s more!

These are just some of our SaaS providers. In an upcoming post I’ll talk more about some of the services that help us support and bill our customers and test and develop our product.

We have found all of these providers deliver valuable and even crucial services that it would be far more expensive and time consuming for us to manage ourselves. We hope you may find some of them helpful too!

Pragmatic AWS: Principle of Least Privilege with IAM

06.12.2012 | Posted by Stefan Zier, Cloud Infrastructure Architect

Lock and Chain - by Martin Magdalene

One of the basic principles in information security is the Principle of Least Privilege. The idea is simple: give every user/process/system the minimal amount of access required to perform its tasks. In this post, I’ll describe how this principle can be applied to applications running in a cluster of EC2 instances that need access to AWS resources. 

What are we protecting?

The AWS Access Key ID and Secret are innocent looking strings. I’ve seen people casually toss them around scripts and bake them into AMIs. When compromised, however, they give the attacker full control over all of our resources in AWS. This goes beyond root access on a single box – it’s “god mode” for your entire AWS world! Needless to say, it is critical to limit both the likelihood of a successful attack and the exposure in case of a successful attack against one part of your application.

Why do we need to expose AWS credentials at all?

Since our applications run on EC2 instances and access other AWS services, such as S3, SQS, SimpleDB, etc, they need AWS credentials to run and perform their functions.

Limiting the likelihood of an attack: Protecting AWS credentials

In an ideal world, we could pass the AWS credentials into applications without ever writing them to disk and encrypt them in application memory. Unfortunately, this would make for a rather fragile system – after a restart, we’d need to pass the credentials into the application again. To enable automated restarts, recovery, etc., most applications store the credentials in a configuration file.

There are many other methods for doing this. Shlomo Swidler compared tradeoffs between different methods for keeping your credentials secure in EC2 instances.

At Sumo Logic, we’ve picked what Shlomo calls the SSH/On Disk method. The concerns around forgetting credentials during AMI creation don’t apply to us. Our AMI creation is fully automated, and AWS credentials never touch those instances. The AWS credentials only come into play after we boot from the AMI. Each application in our stack runs as a separate OS user, and the configuration file holding the AWS credentials for the application can only be read by that user. We also use file system encryption wherever AWS credentials are stored.

To add a twist, we obfuscate the AWS credentials on disk. We encrypt them using a hard-coded, symmetric key. This obfuscation, an additional Defense-in-Depth measure, makes it a little more difficult to get the plain text credentials in the case of instance compromise. It also makes shoulder surfing much more challenging. 

Limiting exposure in case of a successful attack: Restricted access AWS credentials

Chances are that most applications only need a very small subset of the AWS portfolio of services, and only a small subset of resources within them. For example, an application using S3 to store data will likely only need access to a few buckets, and only perform limited set of operations against them.

AWS’s IAM service allows us to set up users with limited permissions, using groups and policies. Using IAM, we can create a separate user for every application in our stack, limiting the policy to the bare minimum of resources/actions required by the application. Fortunately, the actions available in policies directly correspond to AWS API calls, so one can simply analyze which calls an application makes to the AWS API and derive the policy from this list.

For every application-specific user, we create a separate set of AWS credentials and store them in the application’s configuration file.

In Practice – Automate, automate, automate!

If your stack consists of more than one or two applications or instances, the most practical option for configuring IAM users is automation. At Sumo Logic, our deployment tools create a unique set of IAM users. One set of users per deployment and one user per application within the deployment. Each user is assigned a policy that restricts access to only those of the deployments resources that are required for the application.

If the policies changes, the tools update them automatically. The tools also configure per-application OS level users and restrict file permissions for the configuration files that contain the AWS credentials for the IAM user. The configuration files themselves store the AWS credentials as obfuscated strings.

One wrinkle in this scheme is that the AWS credentials created for the IAM users need to be stored somewhere after their initial creation. After the initial creation of the AWS credentials, they can never be retrieved from AWS again. Since many of our instances are short-lived, we needed to make sure we could use the credentials again later. To solve this particular issue, we encrypt the credentials, then store them in SimpleDB. The key used for this encryption does not live in AWS and is well-protected on hardware tokens.   

Summary

It is critical to treat your AWS credentials as secrets and assign point-of-use specific credentials with minimal privileges. IAM and automation are essential enablers to make this practical.  

Update (6/12/2012): AWS released a feature named IAM Roles for EC2 Instances today. It makes temporary a set of AWS credentials available via instance metadata. The credentials are rotated multiple times a day. IAM Roles add a lot of convenience, especially in conjunction with the AWS SDK for Java.

Unfortunately, this approach has an Achilles heel: any user with access to the instance can now execute a simple HTTP request and get a valid set of AWS credentials. To mitigate some of the risk, a local firewall, such as iptables, can be used to restrict HTTP access to a subset of users on the machine.

Comparing the two approaches 

+ User privileges and obfuscation offer a stronger defense in scenarios where a single (non-root) user is compromised.
+ Per-application (not per-instance) AWS credentials are easier to reason about.
- The rotation of IAM keys performed transparently by IAM roles adds security. An attacker has to maintain access to a compromised machine to maintain access to valid credentials.

Best of Both Worlds

AWS’s approach could be improved upon with a small tweak: Authenticate access to the temporary/rotating credentials T in instance metadata using another pair of credentials A. A itself would not have any privileges other than accessing T from within an instance. This approach would be a “best of both worlds”. Access to A could be restricted using the methods described above, but keys would still be rotated on an ongoing basis.   

The Real Interview Questions

06.08.2012 | Posted by zack

Forbes.com recently published an article with a catchy headline that stated recruiters agree the interview process boils down to three questions:

  1. Can you do the job?

  2. Will you love the job?

  3. Can we tolerate working with you?

A lot of people have been weighing in on this article and most agree that it really is that simple. As the Head of Recruitment for Sumo Logic, I strongly believe that we have constructed an interview process for technical positions that answers to each of these questions, implicitly and explicitly.

  1. Can You Do The Job?

Here at Sumo, technology is a key differentiator for us. Engineers here view coding as an art form. Clean, elegant code is the best way to prove to us in an interview that you can do the job. Raw intelligence and horsepower is necessary but not sufficient. We look for candidates that can perceive the larger context of decisions and trade-offs inherent in full system design.

Communication skills are essential in every job. If you can’t explain the projects you have been working on in your current and previous job, it is difficult for an interviewer to understand your contributions and value of the project. It is key for an engineer to be able to explain his previous projects during our interview process.

Emotional Intelligence (EI) has been a term I have been hearing more and more over the last few years with regards to key hires. EI is the “ability to identify, assess, and control the emotions of oneself, of others, and of groups.” Hiring managers have been taking into account a candidate’s ability to be empathetic to their colleagues around them as candidates with a high EI tend to have a higher chance of fitting in with the culture and being a positive addition for our team.

In addition to performance in the interview, we like to see a track record of success. Success in previous role with relevant domain experience is one way to show this. An excellent academic record or great open source projects are a few others. We ensure that we draw on multiple data points to hire the best candidate, reduce attrition and not exclude candidates that do not have the “typical” background.

  1. Will You Love The Job?

We give equity to our employees and this gives you the ability to own a portion of the company. We want to keep ownership of the company exclusive to people who know that Sumo Logic is going to be a wildly successful company and willing to work hard to get it there. From employees to board members, we want our whole team to be striving for a multi-billion dollar exit.

We currently run what some would describe as a EverybodyOps agile development process. We use a rotating PagerDuty scheme with a primary and secondary on call for a week long rotation. We use a a few tools, including Sumo Logic, to monitor and troubleshoot the service. This has been an opt-in system and every engineer that has joined our team has opted in. 

  1. Can We Tolerate Working With You

Once you hang around the Sumo office long enough, you will realize that you could be in danger of being attacked by a Panda wielding a foam bat, Angry Birds, or flying stuffed squirrels while a public address system connected to our continuous integration server will intermittently alert you of who most recently broke the build (and on-demand funny .wav files). You will find people playing board games during lunch and video games and table tennis after.

We enjoy each other’s company and encourage one another to excel. In between the breakout sessions of fun, the team has been known to be dedicated to their work.  We believe in allowing flexible work hours… no one monitors you as long as your work is getting done although- it is still relatively common to walk into the office and see a few people burning the midnight oil.      

We work hard because we love our jobs and we are comfortable in our environment. Every day, lunch is catered. This means, we have to want to sit down next to you at lunch. Individuality is encouraged as we have a bit of a zany culture and we love to include everyone that joins the team (read: inducted into the Sumo family) to join us in our unofficial impromptu events like movie night, wine tasting, coffee breaks, and going to see concerts.

Our Hiring Process:

At Sumo Logic, our hiring process starts with an informational phone call with me. I help answer any questions about the role, organization etc. that I can.

When we are bringing a candidate on-site for an interview, we typically pair up two engineers at a time. This allows the interviewers to get multiple data points on each question and ease gaps in the in conversation.  We have multiple rounds of the pair interview to gain multiple data points and allow each interviewer the voice their opinion. The hiring manager makes the final decision but the team’s feedback is heavily considered.

When an engineer interviews, the questions are most commonly “here’s an input, here’s an output, write the function”. As you go through the process, you will also be asked open ended design, allowing the engineer to dig in as deep as they would like. We also ask about testing and full system design.

Of course, once we have answered the questions with three concrete Yes’s, the process flips as noted by commenter Beth Harte on the article. The candidate must evaluate us and answer three questions about the company:

  1. Will you allow me to do my job? (Trust in letting someone do their job once hired)
  2. Will you love that I love doing my job? (Respect that there is more to a job than a paycheck)
  3. Can I tolerate working for you? (Corporate culture affects both parties)

Looking for a job:

If you are a candidate that is looking for a job, your personal and professional networks are always going to be your best option. Many companies, including Sumo Logic, have an internal employee referral program and the best hires often come from that program. As a candidate, if you have any connection with a company, their customers or vendors, I suggest you leverage them to get your foot in the door.

Other options you can try to get your foot in the door are attending the company sponsored meetups or listen to talks given by employees and network with them at those events. Companies appreciate when a candidate shows genuine interest in the product, market, and technical challenges they are solving.

If you are working with a contingency recruiter, you’re simply going to have to prove yourself to be best candidate in the pool during the interview process.  Recruiters make typically 20% – 25% of a candidate’s base salary, which associates you to a $20k+ price tag the day you start working. If you’re that candidate, you don’t need to be merely as good as the rest of the candidate pool, but a cut above the rest.

Similarly, if you would like to work at Sumo Logic, we would love to hear from you. We have our jobs posted and I read every application that comes in. Feel free to send me an email directly at zack@sumologic.com.

Pragmatic AWS: Data Destroying Drones

06.05.2012 | Posted by Stefan Zier, Cloud Infrastructure Architect

As we evolve our service, we occasionally delete EBS (Elastic Block Store) volumes. This releases the disk space back to AWS to be assigned to another customer. As a security precaution, we have decided to perform a secure wipe of the EBS volumes. In this post, I’ll explain how we implemented the wipe.

Caveats

Wiping EBS volumes may be slightly paranoid and not strictly needed, since AWS guarantees to never return a previous users data via the hypervisor (as mentioned in their security white paper). We also understand that the secure wipe is not perfect. EBS is able to move our data around in the background and leave back blocks that we didn’t wipe. Still, we felt that this additional precaution was worth the bit of extra work and cost – better safe than sorry.

Drones

We wanted to make sure secure wiping did not to have any performance impact on our production deployment. Therefore, we decided that it would be great to perform the secure wipe from a different set of AWS instances — Data Destroying Drones. We also wanted them to be fire-and-forget, so we wouldn’t have to manually check up on them.

To accomplish all this, we built a tool that:

  1. Finds to-be-deleted EBS volumes matching a set of tag values. (we tag the volumes to mark them for wiping).
  2. Launches one t1.micro instance per EBS volume that needs wiping (using an Ubuntu AMI).
  3. Passes a cloud-init script with Volume ID and (IAM limited) AWS credentials into the instance.

The Gory Details

Ubuntu has a mechanism named cloud-init. It accepts a shell script via EC2’s user data, which is passed in as part of the RunInstances API call to EC2. Here is the script we use for the Data Destroying Drones:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
#!/bin/bash
set -e
export INSTANCE_ID=`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`
export VOLUME_ID=v-12345678
export EC2_URL=https://ec2.us-east-1.amazonaws.com
export EC2_ACCESS_KEY=[key id]
export EC2_SECRET_KEY=[key]
 
sudo apt-get install scrub
euca-attach-volume -i $INSTANCE_ID -d /dev/sdj $VOLUME_ID
sudo scrub -b 50M -p dod /dev/sdj > ~/sdj.scrub.log 2>&1
sleep 30
 
euca-detach-volume $VOLUME_ID
euca-delete-volume $VOLUME_ID
halt

This script automates the entire process:

  1. Attach the volume.
  2. Perform a  DoD 5220.22-M secure wipe of the volume using scrub.
  3. Detach and delete the volume.
  4. Halt the instance.

The instances are configured to terminate on halt, which results in all involved resources to disappear once the secure wipe completes. The scrub can take hours or even days, depending on the size of the EBS volumes, but the cost for the t1.micro instances makes this a viable option. Even if the process takes 48 hours, it costs less than $1 to wipe the volume.

Summary

Aside from being a fun project the Data Destroying Drones have given us additional peace of mind and confidence that we’ve followed best practice and made a best effort to secure our customers data by not leaving any of it behind in the cloud.

Twitter