Sanjay Sarathy, CMO

Differentiators

05.27.2014 | Posted by Sanjay Sarathy, CMO

As our growth has accelerated over the past few quarters, we’ve gained additional insights into what customers care about and why they choose us for machine data analytics.  In addition, our integrations and partnerships with Akamai, Amazon Web Services and ServiceNow have provided even more context around what customers investing in cloud services want and need.  I thought it would be instructive to share one perspective on what we’ve learned. 

  • Our cloud-native strategy is an asset, not just because of traditional TCO and elasticity reasons but because the fundamental cost of running a high-volume, cloud-based log management service that automatically detects patterns and anomalies is prohibitively expensive for customers choosing an on-premise alternative.  It goes back to a central point that many customers bring up with us – “we want to be users of the system, not administrators of it.”  
  • Our customers really care about Service Level Agreements.  Traditional SLAs focus on uptime/availability.  This is essential, but not always sufficient.  We’ve found that as a cloud provider in this space it’s also necessary to provide a SLA for query performance. Why? It’s quite simple. Query performance is essential to delivering on the promise of time-to-value, not just around initial setup, but also around ongoing operations.
  • My colleagues have previously discussed the rationale behind LogReduce and Anomaly Detection. One of the tenets of our product strategy is that the rate of growth of machine data has far outpaced the ability for human rules to automatically capture all insights in your logs. We thus need to combine machine learning with human knowledge to uncover both known and unknown events in machine data. This combination and the reason we invest so much in data science is the underpinning of our analytics strategy.  
  • Log data is “inherently” chatty and volumes spike when issues arise or seasonality goes beyond the norm. It’s during these periods that the need to instantly burst capacity to meet customer demand is critical. An on-premise environment cannot by definition get this done without having expensive spare capacity sitting around, a situation most organizations don’t typically provision for. It’s why we’ve incorporated elastic bursting to over 5x of your regular volume as part of our regular service.

These and other differentiators are a significant reason why we’ve grown by 500% over the past year.  We decided to take these differentiators and our other capabilities and make this part of our website.  Enjoy the read and understand where we’re focusing our R&D efforts to create a valuable machine data analytics service.  

Vance Loiselle, CEO

Sequoia Joins The Team and Eight Lessons of a First Time CEO

05.20.2014 | Posted by Vance Loiselle, CEO

I originally envisioned this blog as a way to discuss our recent $30 million funding, led by our latest investor, Sequoia Capital, with full participation from Greylock, Sutter Hill and Accel. I’ve been incredibly impressed with the whole Sequoia team and look forward to our partnership with Pat. Yet despite 300 enterprise customers (9 of the Global 500), lots of recent success against our large competitor, Splunk, and other interesting momentum metrics, I’d rather talk about the ride and lessons learned from my first two years as a CEO.

  1. It’s Lonely. Accept It and Move On.  My mentor, former boss and CEO of my previous company told me this, years ago. But at the time, it applied to him and not me (in hindsight I realize I did not offer much help). But, like being a first time parent, you really can’t fathom it until you face it yourself. I’m sure there’s some psychology about how certain people deal with it and others don’t. I’m constantly thinking about the implications of tactical and strategic decisions. I’ve learned that if you’re too comfortable, you’re not pushing hard enough. The best advice I can give is to find a Board member you can trust, and use him or her as a sounding board early and often.
  2. Trust Your Gut.  There have been many occasions when I have been given good advice on key decisions. One problem with good advice, is you can get too much of it, and it isn’t always aligned. The best leader I ever met, and another long-time mentor, would always ask, ‘what is your gut telling you?’ More often than not, your gut is right. The nice thing about following your instincts, the only person to blame if it goes awry is yourself.
  3. Act Like It’s Your Money. I grew up in Maine, where $100,000 can still buy a pretty nice house. When I first moved to California from Boston it took me some time to get accustomed to the labor costs and other expenses. The mentality in most of the top startups in Silicon Valley is “don’t worry, you can always raise OPM (other people’s money)”. Though I understand the need to invest ahead of the curve, especially in a SaaS-based business like ours, I also believe too much funding can cause a lack of discipline. People just expect they can hire or spend their way around a problem.
  4. Don’t Be Arrogant. Just saying it almost disqualifies you. Trust me, I have come across all kinds. Backed by arguably the four best Venture Capital firms in the business, I have had plenty of opportunities to meet other CEOs, founders and execs.  Some are incredible people and leaders. Some, however, act like they and their company are way too valuable and important to treat everyone with respect. Life is too short not to believe in karma.
  5. Listen Carefully. If a sales rep is having trouble closing deals, put yourself in his shoes and figure out what help he needs. If the engineering team is not meeting objectives fast enough, find out if they really understand the customer requirements. Often the smallest tweaks in communication or expectations can drastically change the results. Lastly, listen to your customer(s). It is very easy to write off a loss or a stalled relationship to some process breakdown, but customers buy from people they trust.  Customers trust people who listen.
  6. It’s a People Business. Software will eat the world, but humans still make the decisions. We’re building a culture that values openness and rapid decision-making while aligning our corporate mission with individual responsibilities. This balance is a constant work in process and I understand that getting this balance right is a key to successfully scaling the Sumo Logic business.
  7. Find the Right VCs at the Right Time. I can’t take any credit for getting Greylock or Sutter Hill to invest in our A and B rounds, respectively.  But I do have them to thank for hiring me and helping me. We partnered with Accel in November of 2012 and now Sequoia has led this recent investment. Do not underestimate the value of getting high quality VCs. Their access to customers, top talent, and strategic partners is invaluable.  Not to mention the guidance they give in Board meetings and at times of key decisions. The only advice I can give here is:  1) know your business cold, 2) execute your plan and 3) raise money when you have wind at your back.  Venture Capitalists make a living on picking the right markets with the right teams with the right momentum. Markets can swing (check Splunk’s stock price in last 3 months) and momentum can swing (watch the Bruins in the Stanley Cup – never mind they lost to the Canadiens).
  8. Believe. It may be cliché, but you have to believe in the mission. If you haven’t watched Twelve O’Clock High, watch it. It’s not politically correct, but it speaks volumes about how to lead and manage. You may choose the wrong strategy or tactics at times. But you’ll never know if you don’t have conviction about the goals.

OK, so I’m no Jack Welch or Steve Jobs, and many of these lessons are common sense. But no matter how much you think you know, there is way more that you don’t. Hopefully one person will be a little better informed or prepared by my own experience.

 

Jacek Migdal

Building Scala at Scale

05.12.2014 | Posted by Jacek Migdal

The Scala compiler can be brutally slow. The community has a love-hate relationship with it. Love means “Yes, scalac is slow”. Hate means, “Scala — 1★ Would Not Program Again”. It’s hard to go a week without reading another rant about the Scala compiler.

Moreover, one of the Typesafe co-founders left the company shouting, “The Scala compiler will never be fast” (17:53). Even Scala inventor Martin Odersky provides a list of fundamental reasons why compiling is slow.

At Sumo Logic, we happily build over 600K lines of Scala code[1] with Maven and find this setup productive. Based on the public perception of the Scala build process, this seems about as plausible as a UFO landing on the roof of our building. Here’s how we do it:

Many modules

At Sumo Logic, we have more than 120 modules. Each has its own source directory, unit tests, and dependencies. As a result, each of them is reasonably small and well defined. Usually, you just need to modify one or a few of them, which means that you can just build them and fetch binaries of dependencies[2].

Using this method is a huge win in build time and also makes the IDE and test suites run more quickly. Fewer elements are always easier to handle.

We keep all modules in single GitHub repository. Though we have experimented with a separate repository for each project, keeping track of version dependencies was too complicated.

Parallelism on module level

Although Moore’s law is still at work, single cores have not become much faster since 2004. The Scala compiler has some parallelism, but it’s nowhere close to saturating eight cores[3] in our use case.

Enabling parallel builds in Maven 3 helped a lot. At first, it caused a lot of non-deterministic failures, but it turns out that always forking the Java compiler fixed most of the problems[4]. That allows us to fully saturate all of the CPU cores during most of the build time. Even better, it allows us to overcome other bottlenecks (e.g., fetching dependencies).

Incremental builds with Zinc

Zinc brings features from sbt to other build systems, providing two major gains:

  • It keeps warmed compilers running, which avoids the startup JVM “warm-up tax”.
  • It allows incremental compilation. Usually we don’t compile from a clean state, we just make a simple change to get recompiled. This is a huge gain when doing Test Driven Development.

For a long time we were unable to use Zinc with parallel modules builds. As it turns out, we needed to tell Zinc to fork Java compilers. Luckily, an awesome Typesafe developer, Peter Vlugter, implemented that option and fixed our issue.

Time statistics

The following example shows the typical development workflow of building one module. For this benchmark, we picked the largest one by lines of code (53K LOC).

Example module build time

This next example shows building all modules (674K LOC), the most time consuming task.

 Total build time

Usually we can skip test compilation, bringing build time down to 12 minutes.[5]

Wrapper utility

Still, some engineers were not happy, because:

  • Often they build and test more often than needed.
  • Computers get slow if you saturate the CPU (e.g., video conference becomes sluggish).
  • Passing the correct arguments to Maven is hard.

Educating developers might have helped, but we picked the easier route. We created a simple bash wrapper that:
Runs every Maven process with lower CPU priority (nice -n 15); so the build process doesn’t slow the browser, IDE, or a video conference.

  • Makes sure that Zinc is running. If not, it starts it. 
  • Allows you to compile all the dependencies (downstream) easily for any module.
  • Allows you to compile all the things that depend on a module (upstream).
  • Makes it easy to select the kind of tests to run.

Though it is a simple wrapper, it improves usability a lot. For example, if you fixed a library bug for a module called “stream-pipeline” and would like to build and run unit tests for all modules that depend on it, just use this command:

bin/quick-assemble.sh -tu stream-pipeline

Tricks we learned along the way

  1. Print the longest chain of module dependency by build time.
    That helps identify the “unnecessary or poorly designed dependencies,” which can be removed. This makes the dependency graph much more shallow, which means more parallelism.
  2. Run a build in a loop until it fails.
    As simple as in bash:  while bin/quick-assemble.sh; do :; done.
    Then leave it overnight. This is very helpful for debugging non-deterministic bugs, which are common in a multithreading environment.
  3. Analyze the bottlenecks of build time.
    CPU? IO? Are all cores used? Network speed? The limiting factor can vary during different phases. iStat Menus proved to be really helpful.
  4. Read the Maven documentation.
    Many things in Maven are not intuitive. The “trial and error” approach can be very tedious for this build system. Reading the documentation carefully is a huge time saver.

Summary

Building at scale is usually hard. Scala makes it harder, because relatively slow compiler. You will hit the issues much earlier than in other languages. However, the problems are solvable through general development best practices, especially:

  • Modular code
  • Parallel execution by default
  • Invest time in tooling

Then it just rocks!

 

[1] ( find ./ -name ‘*.scala’ -print0 | xargs -0 cat ) | wc -l
[2] All modules are built and tested by Jenkins and the binaries are stored in Nexus.
[3] The author’s 15-inch Macbook Pro from late 2013 has eight cores.
[4] We have little Java code. Theoretically, Java 1.6 compiler is thread-safe, but it has some concurrency bugs. We decided not to dig into that as forking seems to be an easier solution.
[5] Benchmark methodology:

  • Hardware: MacBook Pro, 15-inch, Late 2013, 2.3 GHz Intel i7, 16 GB RAM.
  • All tests were run three times and median time was selected.
  • Non-incremental Maven goal: clean test-compile.
  • Incremental Maven goal: test-compile. A random change was introduced to trigger some recompilation.
Russell

Why You Should Never Catch Throwable In Scala

05.05.2014 | Posted by Russell

Scala is a subtle beast and you should heed its warnings. Most Scala and Java programmers have heard that catching Throwable, a superclass of all exceptions, is evil and patterns like the following should be avoided:

1 2 3 4 5 6 7
try {
aDangerousFunction()
} catch {
case ex: Throwable => println(ex)
// Or even worse
case ex => println(ex)
}

This pattern is absurdly dangerous. Here’s why:

The Problem

In Java, catching all throwables can do nasty things like preventing the JVM from properly responding to a StackOverflowError or an OutOfMemoryError. Certainly not ideal, but not catastrophic. In Scala, it is much more heinous. Scala uses exceptions to return from nested closures. Consider code like the following:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
def inlineMeAgain[T](f: => T): T = {
f
}
 
def inlineme(f: => Int): Int = {
try {
inlineMeAgain {
return f
}
} catch {
case ex: Throwable => 5
}
}
 
def doStuff {
val res = inlineme {
10
}
println("we got: " + res + ". should be 10")
}
doStuff
view raw scalaclosures.scala hosted with ❤ by GitHub

We use a return statement from within two nested closures. This seems like it may be a bit of an obscure edge case, but it’s certainly possible in practice. In order to handle this, the Scala compiler will throw a NonLocalReturnControl exception. Unfortunately, it is a Throwable, and you’ll catch it. Whoops. That code will print 5, not 10. Certainly not what was expected.

The Solution

While we can say “don’t catch Throwables” until we’re blue in the face, sometimes you really want to make sure that absolutely no exceptions get through. You could include the other exception types everywhere you want to catch Throwable, but that’s cumbersome and error prone. Fortunately, this is actually quite easy to handle, thanks to Scala’s focus on implementing much of the language without magic—the “catch” part of the try-catch is just some sugar over a partial function—we can define partial functions!

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
def safely[T](handler: PartialFunction[Throwable, T]): PartialFunction[Throwable, T] = {
case ex: ControlThrowable => throw ex
// case ex: OutOfMemoryError (Assorted other nasty exceptions you don't want to catch)
//If it's an exception they handle, pass it on
case ex: Throwable if handler.isDefinedAt(ex) => handler(ex)
// If they didn't handle it, rethrow. This line isn't necessary, just for clarity
case ex: Throwable => throw ex
}
 
// Usage:
/*
def doSomething: Unit = {
try {
somethingDangerous
} catch safely {
ex: Throwable => println("AHHH")
}
}
*/

This defines a function “safely”, which takes a partial function and yields another partial function. Now, by simply using catch safely { /* catch block */ } we’re free to catch Throwables (or anything else) safely and restrict the list of all the evil exception types to one place in the code. Glorious.

Twitter