Ben Newton, Corporate Sales Engineering Manager

Harder, Better, Faster, Stronger – Machine Data Analytics and DevOps

03.28.2013 | Posted by Ben Newton, Corporate Sales Engineering Manager

Work It Harder, Make It Better

Do It Faster, Makes Us Stronger

More Than Ever Hour After

Our Work Is Never Over

     Daft Punk – “Harder, Better, Faster, Stronger”

 

When trying to explain the essence of DevOps to colleagues last week, I found myself unwittingly quoting the kings of electronica, the French duo Daft Punk (and Kanye West, who sampled the song in “Stronger”). So often, I find the “spirit” of DevOps being reduced to mere automation, the takeover of Ops by Dev (or vice versa), or other over-simplications. This is natural for any new, potentially over-hyped, trend. But how do we capture the DevOps “essence” – programmable architecture, agile development, and lean methodology – in a few words? It seems like the short lyrics really sum up the essence of the flexible, agile, constantly improving ideal of a DevOps “team”, and the continuous improvement aspects of lean and agile methodology.

So, what does this have to do with machine data analytics and Sumo Logic? Part of the DevOps revolution is a deep and wrenching re-evaluation of the state of IT Operations tools. As the pace of technological change and ferocity of competition keep increasing for any company daring to make money on the Internet (which is almost everybody at this point), the IT departments are facing a difficult problem. Do they try to adapt the process-heavy, tops-down approaches as exemplified by ITIL, or do they embrace a state of constant change that is DevOps?  In the DevOps model, the explosion of creativity that comes with unleashing your development and operations teams to innovate quickly overwhelms traditional, static tools. More fundamentally, the continuous improvement model of agile development and DevOps is only as good as the metrics used to measure success. So, the most successful DevOps teams are incredibly data hungry. And this is where machine data analytics, and Sumo Logic in particular, really comes into its own, and is fundamentally in tune with the DevOps approach.

 

1.  Let the data speak for itself

Unlike the management tools of the past, Sumo Logic makes only basic assumptions about the data being consumed (time stamped, text-based, etc.). The important patterns are determined by the data itself, and not by pre-judging what patterns are relevant, and which are not. This means that as the application rapidly changes, Sumo Logic can detect new patterns – both good and ill – that would escape the inflexible tools of the past.

2.  Continuous reinterpretation

Sumo Logic never tries to force the machine data into tired old buckets that are forever out of date. The data is stored raw so that it can continually be reinterpreted and re-parsed to reveal new meaning. Fast moving DevOps teams can’t wait for the stodgy software vendor to change their code or send their consultant onsite. They need it now.

3. Any metric you want, any time you want it

The power of the new DevOps approach to management is that the people that know the app the best, the developers, are producing the metrics needed to keep the app humming. This seems obvious in retrospect, yet very few performance management vendors support this kind of flexibility. It is much easier for developers to throw more data at Sumo Logic by outputting more data to the logs than to integrate with management tools. The extra insight that this detailed, highly specific data can provide into your customers’ experience and the operation of your applications is truly groundbreaking. 

4. Set the data free

Free-flow of data is the new norm, and mash-ups provide the most useful metrics. Specifically, pulling business data from outside of the machine data context allows you to put it in the proper perspective. We do this extensively at Sumo Logic with our own APIs, and it allows us to view our customers as more than nameless organization ID numbers. DevOps is driven by the need to keep customers happy.

5. Develop DevOps applications, not DevOps tools

The IT Software industry has fundamentally failed its customers. In general, IT software is badly written, buggy, hard to use, costly to maintain, and inflexible. Is it any wonder that the top DevOps shops overwhelmingly use open source tools and write much of the logic themselves?! Sumo Logic allows DevOps teams the flexibility and access to get the data they need when they need it, without forcing them into a paradigm that has no relevance for them. And why should DevOps teams even be managing the tools they use? It is no longer acceptable to spend months with vendor consultants, and then maintain extra staff and hardware to run a tool. DevOps teams should be able to do what they are good at – developing, releasing, and operating their apps, while the vendors should take the burden of tool management off their shoulders.

 

The IT industry is changing fast, and DevOps teams need tools that can keep up with the pace – and make their job easier, not more difficult. Sumo Logic is excited to be in the forefront of that trend. Sign up for Sumo Logic Free and prove it out for yourself.

Ben Newton, Corporate Sales Engineering Manager

Finding Needles in the the Machine Data Haystack – LogReduce in the Wild

03.19.2013 | Posted by Ben Newton, Corporate Sales Engineering Manager

Making Sense of Data with Log ReduceAs with any new, innovative feature in a product, it is one thing to say it is helpful for customers – it is quite another to see it in action in the wild. Case in point, I had a great discussion with a customer about using LogReduce™ in their environment. LogReduce is a groundbreaking tool for uncovering the unknown in machine data, and sifting through the inevitable noise in the sea of log data our customers put in Sumo Logic. The customer in question had some great use cases for LogReduce that I would like to share.

Daily Summaries

With massive amounts of log data flowing through modern data centers, it is very difficult to get a bird’s eye view of what is happening. More importantly, the kind of summary that provides actionable data about the day’s events is elusive at best. In our customer example, they have been using LogReduce to provide exactly that type of daily, high-level overview of the previous day’s log data. How does it work? Instead of using obvious characteristics to group log data like the source (e.g. Window’s Events) or host (e.g. server01 in data center A), LogReduce uses “fuzzy logic” to look for patterns across all of your machine data at once – letting the data itself dictate the summary. Log data with the same patterns, or signatures, are grouped together – meaning that new patterns in the data will immediately stand out, and the noise will be condensed to a manageable level.

Our customer is also able to supply context to the LogReduce results – adjusting and extending signatures, and adjusting relevance as necessary. In particular, by adjusting the signatures that LogReduce finds, the customer is to “teach” LogReduce to provide the best results in the most relevant way. This allows them to separate the critical errors out, while still acknowledging the background noise of known messages. The end-result is a daily summary that is both more relevant because of the user-supplied, business context as well as being flexible enough to find important, new patterns.

Discovering the Unknown

And finding those new patterns is the essential essence of Big Data analytics. A machine-data analytics tool should be able to find unknown patterns, not simply reinforce the well-known ones. In this use case, our customer already has alerting established for known, critical errors. The LogReduce summary provides a way to identify, and proactively address, new, unknown errors. In particular, by using LogReduce’s baseline and compare functionality, Sumo Logic customers can establish a known state for log data and then easily identify anomalies by comparing the current state to the known, baselined state.

In summary, LogReduce provides the essence of Big Machine Data analytics to our customers – reducing the the constant noise of today’s datacenter, while finding those needles in the proverbial haystack. This is good news for customers who want to leverage the true value of their machine data without the huge investments in the time and expertise required in the past.

Praveen Rangnath, Former Head of Product Marketing

Show Me the VPN Logs!!!

03.07.2013 | Posted by Praveen Rangnath, Former Head of Product Marketing

             Show Me the Money!!!                              Show Me the VPN Logs!!!

Move over Tesla automobile logs, it’s time for Yahoo VPN logs to get their moment in the sun!  

Just as soon as log data dropped out of the headlines they came right back, as Yahoo CEO Marissa Mayer announced a ban on telecommuting – with the decision reportedly driven by analysis of the company’s VPN log data. 

From the VPN data, it’s said that the Yahoo CEO determined too many remote workers were not pulling their weight, as evidenced by their lack of connecting to the VPN and accessing Yahoo’s IT systems.  Certainly, VPN logs don’t tell the entire story around telecommuter productivity, but they are an important data point, and the information contained in those logs certainly was compelling for Ms. Mayer. 

There is of course a bigger picture to this, and it starts with the fact that this is not the first time VPN logs are in the news.  (Not even the first time this year!).  See this blog post from the Verizon RISK team, where they helped their client identify a developer who took global wage arbitrage to an extreme; he collected his six-figure paycheck in the USA and then outsourced his own job to a Chinese consulting firm, paying that firm a fraction of his salary to do his job for him!   

How did he do this?  Simple: He FedEx’d his RSA token to China.  How did he get caught?  Simple: They found him sitting in his office while the VPN logs showed him in China. 

Busted. 

All thanks to the logs. 

At the highest level, what do the Tesla, Yahoo, and wage arbitrage stories tell us?  Simply put, log data is immensely valuable, it’s increasingly becoming front and center, and it’s not going away anytime soon. 

We at Sumo Logic couldn’t be happier, as this is further public recognition of the value hidden in machine data (the biggest component of which is log data).  We’ve said it many times, log data holds the absolute and authoritative record of all the events that occurred.  That’s true for automobile logs, server logs, application logs, device logs, and yes Mr. Developer who outsourced his job to China… VPN logs.   

 

Sanjay Sarathy, CMO

The Marriage of Machine Data and Customer Service

03.06.2013 | Posted by Sanjay Sarathy, CMO

Last week we announced how Atchik uses Sumo Logic and our ability to easily analyze machine data to reshape its customer service function.  In fact, there are a variety of ways in which customer service organizations can become best friends with your log management infrastructure to improve your customers’ perception of your product or service.  Specifically, companies can use a log management service to:

  • Pinpoint exactly what the customer did during the course of a transaction or interaction with an application or service, as opposed to relying purely on email threads or phone logs.  This root cause analysis can help in understanding bottlenecks that the customer complained about and, just as importantly, provide guidance to the development team on how customers are using the product or service.  Actually it’s a great reason for the app development teams to use the service as well, but that’s the subject of another post.  
  • Easily correlate that application activity with the impact on other infrastructure elements that affect the consumer experience.  Unfortunately, many companies today only focus on a single application view of the customer experience when, given how integrated applications and services are today, it’s critical to get a full picture of all the different ways in which the customer is affected.
  • Proactively address potential customer-facing issues *before* they hit by receiving real-time alerts when application anomalies are diagnosed by the log management solution
  • Create customer dashboards and reports that provide real-time insights into the customer activity you care most about tracking  

We use Sumo Logic internally to support every function in the organization from application development to QA to customer service and even marketing.  Our co-founder and VP of Engineering, Kumar Saurabh, is hosting a webinar on March 26th to talk about “Sumo and Sumo”. We invite you to attend.

 


Twitter