Blog › Culture

Vivek Kaushal

Debugging Amazon SES message delivery using Sumo Logic

10.02.2014 | Posted by Vivek Kaushal

 

We at Sumo Logic use Amazon SES (Simple Email Service) for sending thousands of emails every day for things like search results, alerts, account notifications etc. We need to monitor SES to ensure timely delivery and know when emails bounce.

Amazon SES provides notifications about status of email via Amazon SNS (Simple Notification Service). Amazon SNS allows you to send these notifications to any HTTP endpoint. We ingest these messages using Sumo Logic’s HTTP Source.

Using these logs, we have identified problems like scheduled searches which always send results to an invalid email address; and a Microsoft Office 365 outage when a customer reported having not received the sign up email.

 

Here’s a step by step guide on how to send your Amazon SES notifications to Sumo Logic.

1. Set Up Collector. The first step is to set up a hosted collector in Sumo Logic which can receive logs via HTTP endpoint. While setting up the hosted collector, we recommend providing an informative source category name, like “aws-ses”.  

2. Add HTTP Source. After adding a hosted collector, you need to add a HTTP Source. Once a HTTP Source is added, it will generate a URL which will be used to receive notifications from SNS. The URL looks like https://collectors.sumologic.com/receiver/v1/http/ABCDEFGHIJK.  

3. Create SNS Topic. In order to send notifications from SES to SNS, we need to create a SNS topic. The following picture shows how to create a new SNS topic on the SNS console. We uses “SES-Notifications” as the name of the topic in our example.

4. Create SNS Subscription. SNS allows you to send a notification to multiple HTTP Endpoints by creating multiple subscriptions within a topic. In this step we will create one subscription for the SES-Notifications topic created in step 3 and send notifications to the HTTP endpoint generated in step 2.

5. Confirm Subscription. After a subscription is created, Amazon SNS will send a subscription confirmation message to the endpoint. This subscription confirmation notification can be found in Sumo Logic by searching for: _sourceCategory=<name of the sourceCategory provided in step 1>

For example: _sourceCategory=aws-ses 

Copy the link from the logs and paste it in your browser.

6. Send SES notifications to SNS. Finally configure SES to send notifications to SNS. For this, go to the SES console and select the option of verified senders on the left hand side. In the list of verified email addresses, select the email address for which you want to configure the logs. The page looks like

On the above page, expand the notifications section and click edit notifications. Select the SNS topic you created in step 3.

 

7. Switch message format to raw (Optional). SES sends notifications to SNS in a JSON format. Any notification sent through SNS is by default wrapped into a JSON message. Thus in this case, it creates a nested JSON, resulting in a nearly unreadable message. To remove this problem of nested JSON messages, we highly recommend configuring SNS to use raw message delivery option.

Before setting raw message format

After setting raw message format

 

 

JSON operator was used to easily parse the messages as show in the queries below:

1. Retrieve general information out of messages
_sourceCategory=aws-ses | json “notificationType”, “mail”, “mail.destination”, “mail.destination[0]“, “bounce”, “bounce.bounceType”, “bounce.bounceSubType”, “bounce.bouncedRecipients[0]” nodrop

2. Identify most frequently bounced recipients
_sourceCategory=aws-ses AND !”notificationType\”:\”Delivery” | json “notificationType”, “mail.destination[0]” as type,destination  nodrop | count by destination | sort by _count

Ozan Unlu

Debugging to Customer Hugging – Becoming an SE

09.10.2014 | Posted by Ozan Unlu

“I know app developers, and that’s not you!” It was a statement that I couldn’t really argue with, and it was coming from one of my closest friends. It didn’t matter that I was employed in a career as an app developer at one of the top software companies in the world. It didn’t matter that I was performing well and the tools and applications I coded were being used by hundreds of internal developers. It didn’t even matter that the friend making the conclusion had never written a single line of code in his life, nor had he any idea of my technical ability. The funny thing was, he meant it as a compliment, and so began the biggest career transition of my life.

Coding and logic puzzles were always very intuitive to me, so I always enjoyed solving a variety of technical challenges. Yet, articulation, interpersonal communication and cross-team collaboration were some of my other strong suits I felt weren’t being used in my professional life. My career ambitions to be the biggest success possible combined with my desire to fulfill my potential always had me wondering if there was a role better suited for me where I would be able to leverage both diverse skills sets. Over the years I had many mentors and through all the various conversations and constructive criticism, the same trend was always prevalent. They all thought I could be more successful within a Program Manager or Technical Lead role as it would allow me to take advantage of these strengths that were being under-used in a purely development-focused role. So I made those career moves, but decided to stay within the company. After all, I didn’t want to cast away the experience and knowledge I had gained during my role there, and believed it would propel me in my new roles as they were in a related field. It did; I continued to be successful, and it was certainly a step in the right direction, but needed to be taken further. I had tunnel vision and when I looked at my career, all my choices seemed a little too safe. It was time to take a risk.

I was informed of the Sales Engineering role as it could be the perfect position for me to stretch my wings and use my full potential. The more I looked into it, the better it seemed. I would be a technical expert with deep knowledge of the product while at the same time selling the value of the solution to potential clients. I would be listening to the customer’s needs and educating them on whether or not our product would be the best fit for them. After spending so much time on research and development teams creating software with the same handful of peers every day, the prospect of working with a mixture of clients who were the top engineering minds in the world across a plethora of different technologies was enticing. Just the ability to work with these industry leaders in a variety of different challenges allowed me to solve more technical problems than I was ever able to do as a developer working on a only a handful of projects over the course of a year. I had warmed up to the idea and it was time to commit to something new.

There is one area of the world that people consistently consider the “Mecca of Tech,” and that is the San Francisco / Silicon Valley Bay Area. That was settled. If I was going to go into sales, I had promised myself I would never sell a product in which I didn’t have full confidence, so I needed to find a company with a product I really believed in. Enter Sumo Logic: a fully cloud based data analytics and machine learning solution.

Curious, I created a free account and played around with the product. In a very short time, I could see the impressive power and versatile functionality, the value it could provide to nearly any tech company. Also growing at a tremendous rate, supported by the top investors and sporting a unique combination of relatively low risk and high upside, I couldn’t craft an argument to deter myself from joining the company. I interviewed, and when offered, accepted the job. After committing, what awaited me next felt like a breath of fresh air.

Joining a start up from a large company and transitioning into the sales field from development, I didn’t know what type of culture to expect. What awaited me was a company culture where team members are genuinely and actively supportive, and it was awesome. In the first couple months I learned more about various technologies in the market than I ever knew existed before. I work with customers and drastically improve their systems, processes and consequently their careers. I did not expect to be able to contribute to our internal product development process yet I have our best engineers coming to ask which direction we should take our features. Being able to work with customers and feel like you’re truly helping them while at the same time continuing to design and engineer a product on the cutting edge is the best of both worlds, and the sizable increase in compensation isn’t a bad side effect either. I have no regrets in making the biggest career transition of my life, I’m happier than I’ve ever been and I’m not looking back.

If you want to join Ozan and work as a Sales Engineer at Sumo, click here!

Jim Wilson

Why I Joined Sumo Logic

01.16.2014 | Posted by Jim Wilson

Today I joined Sumo Logic, a cloud-based company that transforms Machine Data into new sources of operations, security, and compliance insights. I left NICE Systems, a market leader and successful organization that had acquired Merced Systems, where I led the Sales Organization for the past 6 years. I had a good position and enjoyed my role, so why leave? And why go to Sumo Logic versus many other options I considered? Many of my friends and colleagues have asked me this, so I wanted to summarize my thinking here.

First, I believe the market that Sumo Logic is trying to disrupt is massive. Sumo Logic, like many companies in Silicon Valley these days, manages Big Data. As Gartner recently noted, the concept of Big Data has now reached the peak of the Hype Cycle. The difference is that Sumo Logic actually does this by generating valuable insights from machine data (primarily log files). As a board member told me, people don’t create Big Data nearly as much as machines do. The emergence in the last 10+ years of cloud solutions, and the proliferation of the Internet and web based technologies in everything we do, in every aspect of business, has created an industry that did not exist 10 years ago. By now it’s a foregone conclusion that cloud technologies and cloud vendors like Amazon Web Services and Workday will ultimately be the solution of choice for all companies, whether they are small mom-and-pop shops or large Global Enterprises. I wanted to join a company that was solving a problem that every company has, and doing it using the most disruptive platform, Software-As-A- Service.

Equally important is my belief that it’s possible to build a better sales team that can make a difference in the traditional Enterprise Sales Process. Sumo Logic competes in a massive market with only one established player, Splunk.  I believe that our capabilities, specifically Machine Data Analytics, are truly differentiated in the market. However, I am also excited to build a sales team that customers and prospects will actually want to work with. Just like technology has evolved (client server, web, cloud) I believe the sales profession needs to as well. Today’s sales organization needs to add value to the sales process, not just get in the way. This means we need to understand more about the product than we describe on the company’s website, be able to explain how our product is different from other choices, and how our service will uniquely solve the complex problems companies face today. I am excited to build an organization that will have a reputation of being knowledgeable about the industry and its ecosystem, will challenge customer thinking while understanding their requirements, and will also be fun to work with. The team at Sumo Logic understands this, and I look forward to delivering on this promise.

Finally, I think Sumo Logic has a great product. I started my sales career at Parametric Technology Corporation (PTC). Selling Pro/ENGINEER was a blast and set the gold standard for great products – everything from watching reactions during demos to hearing loyal customers rave about the innovative work they were doing with the product. I had a similar experience at Groove Networks watching Ray Ozzie and his team build a great product that was ultimately acquired by Microsoft. Sumo Logic seems to be generating that same product buzz. We have some amazing brand names like Netflix, Orange, McGraw-Hill, and Scripps Networks as our customers. These and the other customers we have are generating significant benefits from using our machine data intelligence service. The best measure of a company is the passion of their customer base. The energy and loyalty that our customer base exhibits for the Sumo Logic service is a critical reason why I’m very bullish about the long-term opportunity.

I am fired up to be a part of this organization. The management team and in particular Vance, Mark, and the existing sales team are already off to a great start and have grown sales significantly. I hope to build on their early success, and I will also follow the advice a good friend recently gave me when he heard the news: “You found something good – don’t screw it up!”

I won’t.

Bill Lazar

Open Source in the Sumo Logic UI

12.09.2013 | Posted by Bill Lazar

Startups are well-known for being go fast, release and iterate. Having quality engineers at Sumo Logic is a big part of doing that well enough that customers want our solution, but like many other young tech companies open source libraries and tools are also a key element in our ability to deliver.

As a recent hire into the User Interface development team I was excited to see just which open source software goes into our cloud log management solution. The list is extensive, because so many of our peers are making great stuff available, but a quick look just at the front end codebase shows:

  • jQuery: The big daddy, jQuery is used by millions of web applications and websites to add dynamic behavior and content to otherwise plain pages.
  • Backbone: A lean, subtly powerful framework for building expressive client-side apps, Backbone provides a core set of MV* classes and a foundation for many community-developed extensions.
  • Sass/Compass: Think “programmable CSS” and you’re capturing the essence of Sass while big brother Compass adds an extensive set of reusable cross-browser CSS patterns as well as several handy utilities.
  • D3: A library for manipulating documents based on data, we use D3 to drive many of the beautiful interactive charts that enable our customers to understand the huge volume of data they process in our application.
  • Require.js: Building large applications is much easier to manage when code can be split into small, coherent chunks (files) and Require.js enables apps to do just this. 
  • Code Mirror: This versatile text editor is the basis for Sumo Logic’s powerful search query editors.
  • jQuery Plugins: Many, the more important to us include Select2ToasterqTip, and jQuery’s jQuery UI.

Collectively these libraries–along with their counterparts used in our service layer–make it possible for a small company to rapidly deliver the depth and quality of Sumo Logic in a cost-effective process. Instead of writing essentially boilerplate code to perform mundane tasks our team is able to create application-specific high value code.

In the days before FOSS proliferated the cost per developer or per CPU for each piece of software would have been prohibitive; the economics of Silicon Valley, where two guys in a coffee shop can spin up a Pinterest or Sumo Logic, just wouldn’t have worked.

From Academia to Sumo Logic

05.21.2013 | Posted by Sebastian Mies, Software Engineer

While I was wrapping up my Ph.D. thesis, my girlfriend (now wife) and I decided that we wanted to leave Germany to live and work in a different country.  Prior to my Ph.D., I started off in computer gaming (ported “Turrican 2″ to the PC when I was a kid1).  Following that, I did my MSCS and Ph.D. in distributed systems and computer networks in Karlsruhe, Germany.

I have been working as a Software Engineer at Sumo Logic since October 2012.  At first I was skeptical about how intellectually engaging and challenging a commercial venture in log management could be.  However, after working at Sumo Logic for more than 6 months, I have to admit that I misjudged the academic and engineering challenges of log management.  

Why? I underestimated the problem and potential!

In contrast to academia, where algorithms are tested under controlled and reproducible conditions, we face the full force of unexpected behaviors of a live system here at Sumo Logic. When we turn algorithms into reality, we are responsible for the entire development process, including planning, testing, and implementing the finished component in a production environment.

No other company is approaching Big Data-scale log management like Sumo Logic. As a main differentiator Sumo Logic offers enterprise class log file processing in the Cloud.  Sumo Logic ingests terabytes per day of unstructured log files that need to be processed in real time. In contrast to websites or other content, log files need exact processing; e.g., a needle in the haystack of logs can be comprised of merely 16 characters (out of the terabytes of data ingested and stored). Thus, there are only a few heuristics we can use to increase efficiency. This makes developing new algorithms to process log data challenging and interesting.

Furthermore, all our databases need to answer queries in a timely manner.  Databases with unpredictable latencies on certain queries are not suitable for the problems we are solving.  We mix-and-match between open source technologies and in-house customized solutions for that reason.

In addition, our customers trust us with information of vital importance to them.  Security concerns influence design decisions across many levels, ranging from operating system level for full hard drive encryption, to application level for role-based access control (RBAC). We have to carefully select algorithms to balance performance (encrypted log files can challenge the efficient use of our cloud resources) while continuing to isolate customers, so that one customer’s demands don’t impact the performance of another.

In summary, I am glad I took the opportunity and joined Sumo Logic to turn my academic research into solutions used by customers to process TBs of their critical data in real time.  This experience has brought self-improvement with each challenge, full-stack knowledge, and a sense of engineering not possible in any other environment.

And, by the way, we are hiring. :)

 

[1] http://www.mobygames.com/game/dos/turrican-ii-the-final-fight/reviews/reviewerId,59617/

Ben Newton, Senior Product Manager

Why I joined Sumo Logic and Moved to Silicon Valley

01.28.2013 | Posted by Ben Newton, Senior Product Manager

Entering StartUP

We make hundreds of decisions every day, mostly small ones, that are just part of life’s ebb and flow. And then there are the big decisions that don’t merely create ripples in the flow of your life - they redirect it entirely. The massive, life-defining decisions like marriage and children; the career-defining decisions like choosing your first job after college. I’ve had my share of career-defining decisions – leaving a physics graduate program to chase after the dot com craze, leaving consulting for sales engineering, etc. The thing about this latest decision is that it combines both. I am joining Sumo Logic, leaving behind a safe job in marketing, and moving to Silicon Valley – away from my friends, family, and community. So, why did I do it? 

 

Now is the time for Start-Ups in Enterprise Software. 

Consumer start-ups get all the press, but the enterprise startups are where the real action is. The rash of consolidations in the last five years or so has created an innovation gap that companies like Sumo Logic are primed to exploit.  The perfect storm of cloud computing, SaaS, Big Data, and DevOps/Agile is forcing customers to start looking outside of their comfort zones to find the solutions they need. Sumo Logic brings together all of that innovation in a way that is too good to not be a part of it.

The Enterprise SaaS Revolution is Inevitable.

The SaaS business model, combined with Agile development practices, is completely changing the ways companies buy enterprise software. Gartner sees companies replacing legacy software with SaaS more than ever. The antiquated term-licenses of on-premise software with its massive up-front costs, double digit maintenance charges, and “true-ups” seem positively barbaric by comparison to the flexibility of SaaS. And crucially for me, Sumo Logic is also one of the few true SaaS companies that is delving into the final frontier of the previously untouchable data center. 

Big Data is the “Killer App” for the Cloud.
“Big Data” analytics, using highly parallel-ized architectures like Hadoop or Cassandra, is one of the first innovations in enterprise IT to truly be “born in the cloud”. These new approaches were built to solve problems that just didn’t exist ten, or even five, years ago. The Big Data aspect of Sumo Logic is exciting to me. I am convinced that we are only scratching the surface of what is possible with Sumo Logic’s technology, and I want to be there on the bleeding edge with them.

Management Teams Matter.
When it really comes down to it, I joined Sumo Logic because I have first-hand knowledge of the skills that Sumo Logic’s management team brings to the table. I have complete confidence in Vance Loiselle’s leadership as CEO, and Sumo Logic has an unbeatable combination of know-how and get-it-done people . And clearly some of the top venture capital firms in the world agree with me. This is a winning team, and I like to win!

Silicon Valley is still Nirvana for Geeks and the best place for Start-Ups.
Other cities are catching up, but Silicon Valley is still the best place to start a tech company. The combination of brainpower, money, and critical mass is just hard to beat. On a personal level I have resisted the siren call of San Francisco Bay Area for too long. I am strangely excited to be in a place where I can wear my glasses as a badge of honor, and discuss my love for gadgets and science fiction without shame. Luckily for me, I am blessed with a wife that has embraced my geek needs, and supports me whole heartedly (and a 21-month-old who doesn’t care either way). 

So, here’s to a great adventure with the Sumo Logic team, to a new life in Silicon Valley, and to living on the edge of innovation. 

P.S.  If you want to see what I am so excited about, get a Sumo Logic Free account and check it out. 

Work Here – Get a free MacBook Pro!

07.31.2012 | Posted by Stefan Zier, Chief Architect

In the last few years, many developers started to favor Mac OS X as a development platform. When you come work for Sumo Logic, we give you a top-of-the line MacBook Pro, and you get to keep it, for good. This post describes the terms of this offer and the rationale behind it.

… Continue Reading

David Andrzejewski, Data Sciences Engineer

Connect the dots with the new Trace operator

07.18.2012 | Posted by David Andrzejewski, Data Sciences Engineer

The trace operator is a new “beta” feature in Sumo Logic that allows the user to identify and follow chains of entities across different log messages, which themselves may be distributed across different assemblies, machines, or even datacenters.  Its origins lie in our culture of “dogfooding” and a recent hackathon where engineers had the opportunity to work on cool or itch-scratching projects of their own choosing.

Since the Sumo Logic service itself is a cloud-based distributed system, we often found ourselves investigating behaviors across multiple components of the system.  Following our own logging advice, we use unique IDs to track these events and to make them easily identifiable within our logs.  However, unless the “originating ID” follows activity across every single system component, it was still necessary to perform multiple searches to follow event chains all the way to the end.  To show how trace automates this procedure and makes our lives easier, we’ll walk through a simplified session tracking example.

Session Tracking Example

Say that your product uses a variety of session IDs to track requests as they flow throughout your system.  For example, different components might use a series of 4-digit hexadecimal IDs to process a customer order as shown below.

Now imagine that an error is encountered within the system while processing the accountID causing an internal error log to be generated containing the webID: “PROCESSING FAILED: webID=7F92“.  

Manually connecting the dots

Starting from this information, we could perform a series of searches and manual investigations to uncover the root cause from this set of logs:

  1. User action webID=7F92
  2. Initiating requestID=082A for webID=7F92 …
  3.  … orderID=34C8 received for requestID=082A …
  4. Retrieving userID=11D2 for requestID=082A …
  5. … accountID=1234 access, userID=11D2 …
  6. ERROR accountID=1234 not found! 
    (this error percolates back until the original webID fails)
  7. PROCESSING FAILED: webID=79F92

Note that to arrive at this conclusion we are essentially following a ”chain” of these hex IDs across different components of our system.

Session tracking with trace

The idea of the trace operator is to automate this process, allowing us to jump almost directly from the observed webID (log #1) to the original failure deep within the system (log #6) via the following query:

* | trace “ID=([0-9a-fA-F]{4})” “7F92″ | where _raw matches “*ERROR*”

Let’s deconstruct what’s happening here. First, assume that our * keyword search query runs over the time window of interest, capturing all relevant logs and plenty of irrelevant ones as well.  Next we have the trace operator:

  • The regular expression (with exactly one capturing group) ”ID=([0-9a-fA-F]{4})” tells trace how to identify the individual pieces of the chain we are trying to build, in this case 4-digit hex strings following “ID=“.
  • The final value gives trace the starting point to build a chain from, which for us is the original webID 7F92.
  • trace then scans incoming logs to build the underlying chain based on IDs occurring together in the same log, starting from the user supplied initial value (here 7F92).  

For example, when trace observes this log

Initiating requestID=082A for webID=7F92 …

it uses the regex to identify two IDs: 082A and 7F92.  Since 7F92 is the starting point it is already part of the chain, and since 082A has just co-occurred with 7F92 we add it to the chain as well.  As trace works its way through the logs, any log containing any ID which is part of the chain is passed through, and any other log is simply ignored. For example the following log would not be added, because none of these IDs are connected to the chain we build starting from the webID 7F92:

Initiating requestID=8182 for webID=8384 …

This is how the trace operator filters logs by “connecting the dots” across different log messages.

The smoking gun

Finally, once we’ve used trace to filter down to logs containing IDs which we know to be connected to the failing webID 7F92, we do string matching to filter down to logs containing the substring “ERROR” and discover a failure associated with the accountID.  Note that if we had simply done an “ERROR” keyword search we might be faced with a deluge of other errors not directly connected to the specific issue we were trying to investigate.  Furthermore, without the constructing our chain of IDs, there would be no obvious connection between accountID 1234 and our failure webID 7F92.  Hopefully this example has given you a taste for what you can do with trace – there are certainly many other possible applications.

Megha Bangalore, Software Engineer

Cry “Havoc!” and let slip the Sumos of war

06.19.2012 | Posted by Megha Bangalore, Software Engineer

While there is no outward sign of it, Sumo Towers (605 Castro) is not a building for the faint of heart.

The ground floor is where we devs have relegated all our ‘evil, as they can pull off wearing a suit without looking like poseurs’ salespeople and marketing staff. It lulls you into a false sense of security – surely, you tell yourself, this is just an office like any other. The door to Amnesty is closed, after all, so how are you to know that it is our own version of Las Vegas? What happens in Amnesty…

The stairwell leading up to the second floor – The Realm of the Devs – does not have any signs saying “Abandon Hope, all ye who enter here” or even “Beware the leopard,” but just the gentle admonishment, “Employees Only.”

It opens onto a bright and open area, with many desks, each covered in monitors – where there had initially been dark twisty passages, and poor lighting, there is now a large room, with delicate (and in fact slightly load-bearing) pillars, painted with strips of steel blue, grey and white. The result of an intense period of bashing through walls, which this particular sumo is saddened to have missed – after all, when is swinging around a sledgehammer not fun?

There is an almost overwhelming feeling of ‘quiet.’ The soft sounds of the clacking of keyboard keys, the hum of music seeping through oversized headphones.

This quiet has misled many a visitor - after all, it is only occasionally punctuated with violent thrashings as the writer is either struck by a blinding flash of inspiration or finally, woefully, temporarily beaten down, expressing a wordless Hulk-ism with the well-known and recognized Keyboard SMASH.

But of course, all in the room know that the keyboard smash is a call to arms! Foam bats are produced, from their easily accessible but very discrete spots, lined up against innocent-seeming shelves and desk frames.

The artillery units have been gathering their ammo since the last battle – in the form of railing angry birds or squirrels, or soft squishy sumos – and their eyes dart around, arms launching missiles with barely a pause and which, after a few years of practice, often strike true.

The rubber band guns, bought on a whim at a crafts festival on Castro, have devolved into diversions versus actual weapons – the rubber bands long since lost to the chaos of previous wars, they still cause the rampaging hordes to pause in their slaughter, which is often enough to turn the tide of any war.

As with any hands on battle, there are the berserkers – huge hulking wielders of destruction. Fortunately for sumo, and for the monitors and coffee mugs, there is only one Panda. Sumo Towers quite literally trembles at his commands (and jumps) and none are safe when he is released – friend and foe alike are subject to his violent whims, and allegiances last no longer than the time it takes to acknowledge them.

The battle rages on – has it really only been a few minutes?? – and the few remaining innocent bystanders get to a half-crouch out of their seats, and indecision flows over their faces… To enter the fray, or hope that the collateral damage will exclude them. Too late, they discover what all the combatants already knew – When Sumos War, Nowhere is Safe.

This battle is made safer in the knowledge that David, our Platform team manager, has left his sigil of authority at home – as is appropriate for wrangling such a motley crew, on his promotion he was presented with a bright pink riding crop – its purpose  (either punishment or motivation) truly depends on the particular quirks of developer it’s wielded against.

True personalities are revealed – the quiet and gentle Yan shows his inner not-so-secret maniacal genius; Krimy is normally a quiet and gentle UI dev – but when the battle heats she is the first to jump right into the thick of it, releasing angry birds with equally angry battle cries; Joan attacks with an aggressive rapier-like bat style, squawking to punctuate each successful hit; Yong plays the puppet master, the devil on a shoulder, whispering evil suggestions to all who are weak to his words. This author herself, who is obviously of a generally delicate and shy character, will rally to defend herself against the injustices of the ‘evil jerks’ that cannot resist taunting her with their longer reach.Stealthily Hidden Bat

And all at once, with a synchronization that belies the close familiarity of these particular warriors, weapons are lowered. A few manic giggles escape, and there is a brief pause as everyone silently acknowledges that ‘I just won,’ and compares battle scars.

The quiet that follows is similar to that from before the fight in only the most superficial ways – the focus which had been slowly drifting returns, and the programmers-turned-warriors revert to their more usual selves, and all that can be heard is the soft clacking of keyboard keys, with renewed purpose and enthusiasm.

Some of Our Essential Service Providers

06.15.2012 | Posted by Joan Pepin, VP of Security/CISO

As I mentioned in one of my previous posts, here at Sumo Logic we believe cloud-based services provide excellent value due to their ease of setup, convenience and scalability, and we leverage them extensively to provide internal services that would be far more time, labor and cash intensive to manage ourselves. Today I’m going to talk about some of the services we use for collaboration, operations and I/T, why we use them, and how they simplify our lives. 

Campfire

Campfire is a huge part of our productivity and culture at Sumo Logic. While I would lump this and Skype together under something like “Managed Corporate Messaging” they fill two very different niches in our environment.

Campfire from 37 Signals is a fantastic tool for group conversations. Using the Campfire service, we have set up multiple chat rooms for various types of issues, including Production Issues, Development Issues, Sales/Customer-Support Issues, and of course, a free-for-all chat-room where we try to make one another spontaneously erupt into chaotic LOLs.

These group-chats provide a critical space where we can work together to troubleshoot and solve problems cooperatively. Campfire makes it very easy to upload pictures and share large amounts of information in real-time with co-workers who can be anywhere. The conversations are all archived for later reference, which allows us to use the Production Incidents room as a 24×7 conference call and canonical forum of record for anything happening to production systems. Our Production on-call devs are expected to echo their actions into the Production channel and keep up with events there as they transpire.

Campfire also has a cool feature which allows you to start a voice conference with participants if needed, which is a great option in certain situations. These calls can also be archived for later reference. One down side to the text and audio archives is that they are not easily searchable so it helps to know approximately when something happened, and we have found it necessary to consult other records to determine where to look.

Skype

Skype is, of course, the very popular IM and VOIP service that was purchased by Microsoft a while back. We use Skype extensively for 1:1 chatting and easy and secure file-transfers throughout the company. We also make extensive use of the wide array of available emoticons. (Stefan Zier is a particularly prolific and artistic user of these.)


We also use Skype video chat for interviews and to collaborate with team members abroad. We have a conference room with a TV and Skype camera just for this application.

Cloudkick

Running a large-scale cloud-based service requires a lot of operational awareness. One of the ways we achieve this is through Cloudkick. Cloudkick was recently acquired by Rackspace and is evolving into Rackspace Cloud Monitoring. We are still on the legacy Cloudkick service, which we have come to use heavily.
We automatically install Cloudkick agents on all of our production instances and use them to collect a wide array of status codes from the O/S and through JMX as well as by running our own custom scripts which we use to check for the existence of critical processes and to detect if things like HPROF files exist.

The Cloudkick website has a “show only failures” mode which we call the “What’s Wrong? Page”. This is a very helpful tool that allows our EverybodyOps team to quickly assess issues with our production environment.

PagerDuty

Of course, we also need to be proactively alerted to failures and crossed thresholds that could indicate trouble, and for this we rely on PagerDuty. (Affectionately known as P. Diddy to many of us, nickname coined by Christian). PagerDuty is another great tool which allows us to maximize the benefits of our EverybodyOps culture.

Accidental Escalation!

Within PagerDuty we have a number of on-call rotations. One for our Production Primary role and one for the Secondary role, as well as another role for monitoring test failures and a lesser-known role for those of us who monitor the temperature in the one small server room we do have. P. Diddy allows us easily cover for each other using exceptions or by simply switching the Primary and Secondary roles on the fly if the Primary needs to go AFK for a while.

P. Diddy allows each user to set their own personal escalation policy which can include texting, calling, and emailing with a configurable number of re-tries and timeouts. Another nice touch is that the rotation calendars can be imported into our personal calendars to remind us of when we are up next. This all makes the on-call rotation run pretty flawlessly from an administrative perspective with no gnarly configuration and management on our end.

I must admit, I do have a personal habit of “Joaning” my secondary when I am on call… To properly “Joan” your secondary you accidentally escalate an alert to them that you meant to resolve, (I blame the comma after “Resolv”!)

Google Apps

Like many companies of all sizes we rely on Google for our email service. While some Sumos (like myself and Stefan) use mail clients to read our email, most Sumos are happy with the standard web interface from Google. We also heavily use internal groups for team communications.

We also make good use of Google Docs for document authoring and sharing (this blog post was written and communally edited using Google Docs, in fact, due to the impressive real-time collaboration, Stefan Zier is watching me add this bit in order to resolve his comment right now!)  We use Google Calendar for our scheduling needs (and calendar-stalking exercises!)

We also use Google Analytics to obsess over you.

Also, as Sumo Logic’s Director of Security, (which makes me partially responsible for managing the users and groups in Google Apps) I appreciate the richness of their security settings and especially the two-factor authentication and mobile device policy management.

There’s more!

These are just some of our SaaS providers. In an upcoming post I’ll talk more about some of the services that help us support and bill our customers and test and develop our product.

We have found all of these providers deliver valuable and even crucial services that it would be far more expensive and time consuming for us to manage ourselves. We hope you may find some of them helpful too!

Twitter