3 Ways to make your Data work for you - on AWS and everywhere else

This is an exciting time to be in the Enterprise software business. Many of the predictions we heard in the late 90s and early 2000s are finally coming true. First, we are finally moving from the world of the server artist (pronounced “Arteeste”) lovingly caring for his server, to a world where any self-respecting operations engineer is capable of runnings hundreds of systems on a bad day. That transformation has come on the back of organizations finally embracing things like configuration automation and standardization on top of powerful cloud platforms like Amazon Web Services (AWS). Container technologies like Docker have only accelerated that transition.

Second, the evolution of the DevOps movement is changing the ways operations teams approach their jobs. Now they tend towards small, collaborative teams swarming problems with no respect for technology silos. Developers are also being held accountable now for the uptime of their software, so they are putting more of the operational machinery in code and changing the data from their apps to suit their needs.

“So, what now?” Thanks for asking. Now that we are mechanizing our software and operations, we need to spend more thinking about what our applications and platforms spit out - the machine data. Here are three ways to up your machine data game:

Embrace the Format

Why are we still producing logs that look like they were written by a drunk monkey? If you are planning on getting any value out of those logs at scale you have to put them in a format that makes them easier to analyze. JSON is really the best choice, but even embracing simple key-value pairs will make a huge difference.

Embrace the Content

Imagination is great for jam sessions with your college buddies, but horrible for operational logging. The more your data looks the same, the easier it is to analyze, and to understand at 2 AM when your application is down. And how do you do that at scale? One way is to embrace standardized metrics and logging libraries (Log4J, NLog, etc.). Another way is to use your platform - well, that’s my third point.

Embrace the Platform

The primary reason why AWS is so successful is not the nebulous “cloud” concept. AWS is powerful because it has standardized and commoditized operational details away. Need a load-balancer? No need to order a $50k box with blinking lights. Press a button in your AWS console. Need another app server, or 10 or 100.? No problem. Now AWS is doing the same thing with data. It is simple to generate streams of useful data with CloudTrail, Cloudwatch, AWS Kinesis, etc. If you use the standardized services like ELB, RDS, or S3, you get even more metrics and logs out of the box. So - embrace the data, embrace the platform.

So, what’s the wrap? Just remember to go all in! Half-measures don’t help you scale to that next level. Flexibility is great for that yoga class you just enrolled in, but standardization will make life easier when the proverbial stuff hits the fan at oh-dark:30 in the morning. Use the tools at hand to squeeze unnecessary mess out of your data and focus on the fun part - making pretty pictures for your boss with that newly massaged data. Box Chart, anyone?