For the last two years, Sumo Logic has been (quietly) building a secure, massively scalable, multi-tenant data management and analytics platform in the cloud. For us at Sumo Logic, the Cloud is a concept we believe in and have internalized deeply into our culture, our processes and infrastructure. In our office we only own two equipment racks, and they are less than half full. The boxes there are our back-up server, some security gear, a VOIP box, and a single small server to provide network and AAA services for the LAN. We have ‘dogfooded’ not only our own product here (we make extensive use of our own product for troubleshooting and operations, see Stefan Zier’s series “Sumo on Sumo”), but the entire idea of the cloud itself. From our email and build environment, through our CRM and our product itself, we live in the Cloud. Through adopting best practices and developing some of our own we operate there in a way that is designed to be secure, and I’d like to share some of the insights we’ve picked up along the way.
Of course, the “Cloud” is a nebulous term 😉 and here at Sumo Logic we use several different types of cloud-based services, which mostly break down into two categories; SaaS and IaaS. On the SaaS side, we have our email and CRM, testing, support and billing and as well as a number of services we use to monitor and alert on our service availability, and on the IaaS side, we use AWS to host our build environment and its associated bug-tracker, wiki and code-repository. One of the many advantages to this model is exemplified by our build environment (Hudson). Hosting this in EC2 provides us with great flexibility in bringing up new build-slaves at peak times, such as before a major release or branch.
In general, the SaaS providers we use provide excellent security features. For instance, we mandate the use of two-factor authentication and strong passwords for access to Sumo Logic email, and our provider has a rich variety of security controls and features such as the two-factor authentication that we can (and do) leverage. This is much simpler than keeping this level of security would be if we ran the whole mess ourselves 🙂
On the IaaS side, Stefan Zier has done an amazing job of setting us up in AWS. One example of how IaaS features can be leveraged is the way in which he handled access to our AWS hosted resources. In addition to username and password authentication to our cloud based services (more on that later) Amazon “security groups” are used to limit network-level access to these services to only certain IP addresses on a whitelist. In order to handle the automation of that whitelist, we make use of a dynamic DNS provider that assigns hostnames to authorized systems. Stefan wrote a program which polls for the addresses of those authorized hosts and updates the corresponding security group in AWS. We plan on getting this set up on AWS’ Virtual Private Cloud sometime soon, which will allow us to layer a VPN on top of this already very secure solution.
Another layer of protection we incorporate is anonymity. All of our cloud-based company infrastructure is attached to a domain which is not connected to Sumo Logic in any way. Similarly, we have used anonymized labels for our private git repositories, etc. The public cloud allows for some obscurity and anonymity, and we leverage that.
Of course, there is still work involved in keeping things secure. Living in the cloud means having a lot of accounts. A LOT of them. Our process for on-boarding and off-boarding employees requires the creation or deletion of a very large number of accounts and the adding or removing of a lot of tags, groups, lists and checkboxes. Having solid documented procedures for this is the only way to keep it straight and running smoothly. We also have to host our own LDAP server for AAA to some of our tools, and we also use this for our VPN authentication. So we have to manage that, and it is a pain. Centralized AAA and policy/group management services exist for cloud-based services, and we’ve looked at some. Unfortunately, none of them also supported hosting or managing an LDAP instance for us, and keeping that synced up and tied-into the rest of the mess would be a killer feature. We certainly feel there is a gap in the market here that we wish somebody would fill.
From an end-user perspective, there are a lot of accounts to keep straight and a lot of passwords to remember. In order to make this both secure and manageable, we provide (and mandate the use of) a password management tool that runs both Mac and Windows (and has a useable web-interface for Linux and others) and also runs on Android and iPhone. It uses a cloud-based file-storage service to sync its encrypted password database between devices. This allows us to mandate that users have extremely strong passwords that are different for every account and it gives our users the tools to actually comply with that rule 🙂
Of course, building a secure cloud-based service ourselves requires a lot of thought and engineering well beyond just leveraging our provider’s consoles. We have done a lot of thought about how to build a secure service leveraging IaaS and we have written a paper about some of the design principles and practices we employ. If you are interested you can download it here.
