Sign up for a live Kubernetes or DevSecOps demo

Click here

Brian Bozzello

Posts by Brian Bozzello

Blog

Transform Graphite Data into Metadata-Rich Metrics using Sumo Logic’s Metrics Rules

Graphite Metrics are one of the most common metrics formats in application monitoring today. Originally designed in 2006 by Chris Davis at Orbitz and open-sourced in 2008, Graphite itself is a monitoring tool now used by many organizations both large and small. It accepts metrics from a wide variety of sources, including popular daemons like collectd and statsd, provided that the metrics are sent in the following simple format: Where metric path is a unique identifier, specified in a dot-delimited format. Implicit in this format is also some logical hierarchy specific to each environment, for example: While this hierarchical format has been widely accepted in the industry for years, it creates challenges for usability and ultimately lengthens the time to troubleshoot application issues. Users need to carefully plan and define these hierarchies ahead of time in order to maintain consistency across systems, scale monitoring effectively in the future and reduce confusion for the end user leveraging these metrics. Fortunately, the industry is evolving towards tag-based metrics to make it easier to design and scale these systems, and Sumo Logic is excited to announce the launch of Metrics Rules to take advantage of this new model immediately. Using Metrics Rules to Bring Graphite Metrics into the New World Sumo Logic built its metrics platform to support metadata-rich metrics, but we also acknowledged that the broader industry and many of our customers have invested heavily in their Graphite architecture and naming schemas over time. Sumo Logic’s Metrics Rules solution now allows users to easily transform these Graphite metrics into the next generation, tag-based metric format, which provides three key benefits: Faster Time to Value: No need to re-instrument application metrics to take advantage of this metadata-rich, multi-dimensional format. Send Graphite-formatted metrics to Sumo immediately and enrich them with tag-based metadata later. Easy Configuration: An intuitive user interface (UI) allows you to validate and edit your transformation rules in real-time, while competitive solutions require carefully defined config files that are difficult to set up and prone to errors. Improved Usability: With rich metadata, use simple key-value pairs to discover, visualize, filter and alert on metrics without knowing the original Graphite-based hierarchy. Using the example above, we can use Metrics Rules to enrich the dot-delimited Graphite names with key-value tags, which will make it easier for us to monitor metrics by our system’s logical groupings in the future: Intuitive Metrics Rules UI for Easy Validation and Edits As Graphite monitoring systems grow, so do the complexities in maintaining these dot-delimited hierarchies across the organization. Some teams may have defined Graphite naming schemes with five different path components (e.g., app.env.host.assembly.metric), while others may have more components or a different hierarchical definition altogether. To make it easier to create tags out of these metrics, the Metrics Rules configuration interface allows you to see a preview of your rules and make sure that you’ve properly captured the different components. Simply specify a match expression (i.e., which metrics the rule will apply to), define variables for each of the extracted fields and then validate that each tag field is extracting the appropriate values. After saving the rule, Sumo Logic will go back in time and tag your metrics with this new metadata so you can take advantage of these rules for prior data points. Improved Discoverability, Filtering and Alerting with Key-Value Tags Once these metrics contain the key-value tags that we’ve applied via Metrics, you can take advantage of several usability features to make finding, visualizing and alerting on your metrics even easier. For example, Sumo Logic’s autocomplete feature makes it easier to find and group metrics based on these key-value tags: Additionally, when using our unified dashboards for logs and metrics, these new tags can be leveraged as filters for modifying visualizations. Selecting a value in one of these filters will append a key-value pair to your query and filter down to the data you’re interested in: Finally, configuring alerts becomes significantly easier when scoping and grouping your metrics with key-value pairs. In the example below, we selected metric=vcpu.user from one of our namespaces, and we’re averaging this across each node in Namespace=csteam. This means that alerts will trigger across each node, and our email and/or webhook notifications will tell us which particular node has breached the threshold: The Bigger Picture Users can now convert legacy Graphite-formatted performance metrics into the metadata-rich metrics with Sumo Logic, both in real-time and after ingestion. This allows customers to increase the usability and accessibility for their analytics users by allowing them to leverage business relevant tags, instead of relying only on obscure, technical tags. Now with the capability to extract business context (metadata) from IT-focused metrics, organizations can use this data to gain actionable insight to inform strategic business decisions. In a broader context, this is significant because as we’ve been seeing from our customers, the hard lines between IT and business are becoming blurred, and there’s a strong emphasis on using data to improve the overall end-user experience. As more organizations continue to leverage machine data analytics to improve their security, IT and business operations, the ability to map machine data insights to actionable, contextual business analytics for IT and non-core-IT users is critical. Learn More Head over to Sumo Logic DocHub for more details on how to configure Metrics Rules on your account. Additionally, see how these rules can even be used for non-Graphite metrics by parsing out values from existing key-value pairs such as _sourceCategory and _sourceHost. Are you at DockerCon 2018 at Moscone Center in San Francisco this week? We’ll be there! Stop by our booth S5 to chat with our experts, get a demo and to learn more! Additional Resources Read the press release on our latest product enhancements unveiled at DockerCon Download the report by 451 Research & Sumo Logic to learn how machine data analytics helps organizations gain an advantage in the analytics economy Check out the Logs-to-Metrics blog Sign up for Sumo Logic for free

June 12, 2018

Blog

Accelerate Data Analytics with Sumo Logic’s Logs-to-Metrics Solution

If you’re building a new application from scratch and are responsible for maintaining its availability and performance, you might wonder whether you should be monitoring logs or metrics. For us, it’s a no-brainer that you’ll want both: metrics are fast and efficient for proactively monitoring the health of your system, while logs are essential for helping to troubleshoot the details of the issue itself to find the root cause. To use a real world analogy, let’s say you go in for an annual check up and the doctor sees you have elevated blood pressure (“the metric”). He then asks you enough questions to discover that you’ve been eating fast food five nights a week (“the logs”), and recommends a diet change to normalize your blood pressure levels (“the fix”). But what if you’re working with an existing application where logs have always been used for monitoring? Or you’re leveraging third-party services that are only sending you logs? These logs may often contain key performance indicators (KPIs) like latency, bytes sent and request time, and Sumo Logic is great for structuring this in a way to create dashboards and alerts. However, to get the performance benefits of metrics, you might consider re-instrumenting your application to output those KPIs as native metrics instead of logs. But we all know how much free time you have to do that. Extract Metrics from Logs for High Performance Analytics Still, you may be wondering: why would I spend time converting all of my log data to metrics? The long and short of it is this: to deliver the best customer experience to your users. And machine data analytics is essential for that. However, according to data we recently released, one of the biggest barriers to adopting a data analytics tool is the lack of real-time analytics to inform operational, security and business decisions. Without it, you’ll suffer from slow analytics and will lose customers in minutes. No one wants that, especially when customers are relying on your tools to help them resolve critical issues. Sumo Logic’s Logs-to-Metrics solution is the answer to that challenge because we make it easy for you to turn logs into metrics that can be then used as valuable KPIs. And since we do the heavy lifting and work with you to create metrics from existing logs, you don’t have to worry about creating them from scratch. Whether your KPIs are embedded in the logs themselves (e.g., latency, request_time) or you’re looking to compute KPIs by counting the logs (e.g., error count, request count), we’ve got you covered. Turning some of your logs into metrics will give you several key benefits: High Performance Analytics: Storing data in a time-series database allows for lightning fast query times, since the data is optimized for speed and efficiency. Thirteen-Month Data Retention: For all metrics, Sumo Logic provides 13-month retention by default, enabling quick long-term trending of critical business and operational KPIs. Flexible and Low Latency Alerting: With metrics, you can leverage Sumo Logic’s real-time metrics alerting engine, which includes intuitive UI configuration, multiple threshold settings, missing data alerts, muting and more. Never Re-Instrument Code Again: Gain all of the benefits of metrics without digging into your code to configure a metrics output. Easy Configuration with Real-Time Validation In order to make this metrics extraction as seamless as possible, we’ve created a fast way for you to validate your rules in real-time. There are three simple steps to pick out your metrics: Specify a Scope: This is the set of logs that contain the metrics you are interested in. Typically, this contains one or more pieces of metadata and some keywords to narrow down the stream of logs. For example, “_sourceCategory=prod/checkout ERROR”. Define a Parse Expression: Use Sumo Logic’s parsing language to extract out the important fields you’ll want to turn into metrics. You can even use regular expressions for more complex log lines. Select Metrics and Dimensions: After successfully parsing your logs, select which fields are metrics and which are dimensions. Metrics will be the actual value you are interested in tracking, while dimensions are the groups you would want to aggregate those values by. For example, if you want to track the number of errors by service and region, “errors” would be a metric while “service” and “region” would be dimensions. In real-time, Sumo Logic will show you a preview of your parse expression to make sure you’ve correctly extracted the right fields. You can also extract multiple metrics and dimensions from a single rule. KPIs as Metrics = 100x Performance over Logs As much as we love the performance of our log analytics at Sumo Logic, we really love the performance of our metrics. Transforming thousands (or millions) of unstructured log messages into structured visualizations on the fly is always possible, but when the data can be stored as a metric in our native time-series database, the resulting query performance can be astounding. In the simple comparison below, it’s pretty easy to see which chart belongs to metrics: Low Latency Monitoring and Highly Flexible Alerting After extracting metrics out of your logs, you can also take advantage of Sumo Logic’s real-time alerting engine, which monitors your metrics in real-time and triggers notifications within seconds of a condition being met. In additional to the low latency, some other benefits include: Multiple Thresholds: Create different alerts based on the severity of the metric. For example, create a warning alert if CPU is above 60 for five minutes, but generate a critical alert if it’s ever above 90. Multiple Notification Destinations: Send your alerts to multiple destinations. For example, create a PagerDuty incident and send an email when the monitor is critical, but just send a Slack message if it’s hit the warning threshold. Missing Data: Get notified when data hasn’t been seen by Sumo Logic, which can be a symptom of misconfiguration or a deeper operational issue. The Bigger Picture Unstructured machine data is not always optimized for the kind of real-time analytics customers need to inform business decisions. With this new release, users can now take advantage of Sumo Logic’s metrics capabilities without re-instrumenting their code by leveraging existing logs for more efficient analytics and insights. In addition to the deep forensics and continuous intelligence provided by logs, customers can take advantage of metrics by easily extracting key performance indicators from unstructured logs, while still retaining those logs for root cause analysis. These metrics can then be used with the Sumo Logic time series engine, providing 10 to 100 times the analytics performance improvements over unstructured log data searches, as well as support long-term trending of metrics. This allows them to move fast and continue to deliver a seamless experience for their end users. Learn More Logs-to-Metrics is now generally available to all Sumo Logic customers. Head over to our documentation to learn more about how to get started. Additional Resources Read the press release on our latest product enhancements unveiled at DockerCon Download the report by 451 Research & Sumo Logic to learn how machine data analytics helps organizations gain an advantage in the analytics economy Check our new Metrics Rules blog Sign up for Sumo Logic for free

June 12, 2018

Blog

Sumo Logic Partners with IP Intelligence Leader Neustar to Meet Growing Customer Needs at Scale

Customers are visiting your website, employees are logging into your systems and countless machines are talking to each other in an effort to deliver the perfect user experience. We’d like to believe that all of these individuals and machines are operating with the best of intentions, but how can we be so sure? One possible answer lies in the connecting device’s IP address and its respective physical location. IP geolocation is the process of determining the location of a device based on its unique IP address. It not only requires knowledge about the physical location of the computer where the IP address is assigned, but also how the device is connecting (e.g., via anonymous proxy, mobile, cable, etc.). This challenge becomes further complicated in an increasingly digital world with proliferating devices and millions of connections being established across the globe daily. That’s why we’re excited to announce that we’ve partnered with Neustar, a leading IP intelligence provider, to deliver one of the most comprehensive and precise geolocation databases in the industry. As a Sumo Logic customer, you can now leverage Neustar’s 20+ years of experience gathering and delivering IP intelligence insights, all at no additional charge. Precision Database + Weekly Updates = Higher Confidence Analytics In the pre-cellphone era (remember that?), everyone had a landline which meant area codes were fairly accurate identifiers of an end-user location. I knew that 516 meant someone was calling from Long Island, New York, while 415 was likely coming from the San Francisco Bay Area. But the invention of the cellphone complicated this matter. I might be receiving a call from someone with a 516 number, but because the caller was using a “mobile” device, he or she could be located anywhere in the U.S. IP addresses are like very complicated cellphone numbers — they can be registered in one place, used in another and then re-assigned to someone else without much notice. Keeping track of this is an enormous task. And over time, malicious actors realized that they could take advantage of this to not only mask their true location, but create false security alerts to distract security teams from identifying and prioritizing legitimate high-risk threats. That’s why partnering with a leader like Neustar, that uses a global data collection network and a team of network geography network analysts, to update their IP GeoPoint database on a daily basis, is key. This accuracy allows security teams to have full visibility into their distributed, global IT environment and when there’s an attempt to compromise a user’s credentials within an application, they can quickly flag any anomalous activity and investigate suspicious logins immediately. Proactive Geo Monitoring and Alerting in Sumo Logic With Neustar’s IP GeoPoint database, you can rest assured that your geolocation results are more trustworthy and reliable than ever before. Using Sumo Logic, you can continue to take advantage of the proactive alerting and dashboarding capabilities to make sense of IP intelligence across your security and operational teams. For example, you’ll have a high confidence in your ability to: Detect Suspicious Logins: alert on login attempts occurring outside of trusted regions. Maintain Regulatory Compliance: see where data is being sent to and downloaded from to keep information geographically isolated. Analyze End-User Behavior: determine where your users are connecting from to better understand product adoption and inform advertising campaigns. With real-time alerts, for example, you can receive an email or Slack notification if a login occurs outside of your regional offices: Configure real-time alerts to get notified when a machine or user is appearing from outside of a specific region. You can also use real-time dashboards to monitor the launch of a new feature, track customer behavior or gain visibility into AWS Console Logins from CloudTrail: Using Sumo Logic’s Applications, you can install out-of-the-box dashboards for instant geographic visibility into AWS Console Logins, for example. The Bigger Picture Born in AWS, Sumo Logic has always held a cloud-first, security-by-design approach and our vision is to create a leading cloud security analytics platform to help our customers overcome the challenges of managing their security posture in the cloud. There is a major gap in the available on-premises security tools for customers that not only need to manage security in the cloud, but also meet rigorous regulatory compliance standards, especially the European Union’s General Data Protection Regulation (GDPR) that went into effect last week on May 25, 2018. Geolocation is key for those needs which is why we’re thrilled to be rolling this out to our customers as part of a bigger strategy to provide visibility and security across the full application stack. Learn More Head over to Sumo Logic DocHub for more details on how to leverage the new database, then schedule some searches and create dashboards to take advantage of the enhanced IP geolocation. Check out our latest press announcement to learn about the additional features and to our cloud security analytics solution, including intelligent investigation workflows, privacy and GDPR dashboards, and enhanced threat intelligence.

Blog

Integrating Machine Data Analytics in New Relic Insights via Sumo Logic Webhooks

When Sumo Logic and New Relic announced a partnership at AWS re:Invent 2016, we immediately started hearing the excitement from our joint customers. The ability to combine the strengths of two leading SaaS services that offer fast time-to-value for monitoring and troubleshooting modern applications would offer a powerful and complete view of digital businesses, from the client down to the infrastructure. Today, we’re pleased to announce another advancement in our partnership: integrated machine data analytics with application and infrastructure performance data in New Relic Insights via a custom New Relic webhook built directly into Sumo Logic. Custom New Relic webhook in Sumo Logic Unlocking Insights from Sumo Logic Scheduled searches in Sumo Logic allow you to monitor and alert on key events occurring in your application and infrastructure. The flexibility of the query language allows you to pull just the information you need while fine tuning the thresholds to trigger only when necessary. Combined with your New Relic APM and New Relic Infrastructure data in New Relic Insights, you’ll now be able to visualize information such as: Events: Service upgrades, exceptions, server restarts, for example Alerts: More than 10 errors seen in 5 minutes, for example, or failed login attempts exceeding 5 in 15 minutes KPIs: Count of errors by host, for example, or top 10 IPs by number of requests Integrating these insights into New Relic provides an integrated context for faster root cause analysis and reduced Mean Time to Resolution (MTTR), all within a single pane of glass. In just three simple steps, you’ll be able to leverage Sumo Logic webhooks to send data to New Relic. Step 1: Configure the New Relic webhook connection In New Relic Insights, you will first need to register an API key that will be used by the Sumo Logic webhook. These keys allow you to securely send custom events into New Relic from different data sources. Type in a short description to keep a record of how this API key will be used, then copy the Endpoint and Key for setup in Sumo Logic. Generate an API Key from New Relic Insights to be used in Sumo Logic In Sumo Logic, create a New Relic webhook connection and insert the Endpoint and Key into the URL and Insert Key fields. The payload field gives you the flexibility to customize the event for viewing in New Relic. In addition to the actual results, you can optionally specify metadata to provide additional context. For example, the name of the Sumo Logic search, a URL to that particular search, a description, and more. This payload can also be customized later when you schedule the search. Variables from your Sumo Logic search can be included in your payload for additional context in New Relic. Step 2: Schedule a search to send custom events After saving your New Relic webhook, you have the option to specify this as the destination for any scheduled search in Sumo Logic. The example below shows a query to look for “Invalid user” in our Linux logs every 15 minutes. To store and visualize this information in New Relic, we simply schedule a search, select the New Relic webhook that we configured in Step 1, and customize the payload with any additional information we want to include. This payload will send each result row from Sumo Logic as an individual event in New Relic. The Sumo Logic query language allows you to transfer meaningful insights from your logs to New Relic Step 3: Visualize events in New Relic Insights Once the scheduled search has been saved and triggered, we can see the data populating in New Relic Insights and use the New Relic Query Language (NRQL) to create the visualizations we need. NRQL’s flexibility lets you tailor the data to your use case, and the visualization options make it seamless to place alongside your own New Relic data. In fact, you might not even notice the difference between the data sources—can you tell which data below is coming from New Relic, and which is coming from Sumo Logic? A unified view: “Source IP’s from Failed Attempts” streams in from Sumo Logic, while “Errors by Class” comes from New Relic The ability to visualize application and infrastructure performance issues alongside insights from your logs reduces the need to pivot between tools, which can speed root cause analysis. If you’ve spotted an issue that requires a deeper analysis of your logs, you can jump right into a linked Sumo Logic dashboard or search to leverage machine learning and advanced analytics capabilities. Learn more Head over to Sumo Logic DocHub for more details on how to configure the New Relic webhook, then schedule some searches to send custom events to New Relic Insights. We’re excited to continue advancing this partnership, and we look forward to sharing more with you in the future. Stay tuned!

June 8, 2017

Blog

Provide Real-Time Insights To Users Without A Sumo Logic Account

You just finished building some beautiful, real-time Sumo Logic dashboards to monitor your infrastructure and application performance and now you want to show them off to your colleagues. But your boss doesn’t have a Sumo Logic account and your ops team wants this information on TVs around the office. Sound like a familiar situation? We’ve got you covered. You can now share your live dashboards in view-only mode with no login required, all while maintaining the security and transparency that your organization requires. We’ll even kick things off with a live dashboard of our own. Share Information with Colleagues and Customers This new feature enables you to share a dashboard so that anyone with the URL can view your dashboard without logging in. It reduces the friction for sharing information even further so that the right people have the right information when they need it. For example: Colleagues: Share operational and business KPIs with colleagues or executives who do not have a Sumo Logic account. Internal TVs: Display real-time information about your infrastructure and application on monitors throughout your building. Customers: Provide SLA performance or other statistics to your customers. Granular Permissions for Administrators Sharing your sensitive information to users without a login is a serious matter. With great power comes great responsibility, and no matter how much you trust your colleagues that use Sumo Logic, you may not want this power being wielded by all of your team members. If you are an administrator, you can decide which users have this permission and educate them on best practices for sharing information within and outside of your organization. By default, this capability is turned off and can only be enabled by administrators on the account. Protect Dashboard URLs with an IP / CIDR Whitelist For those who want even more protection over who can view these dashboards without logging in, you can restrict viewers to only those accessing it from specific IP addresses or CIDRs. This works great when you are placing live dashboards on TVs throughout your building and you want to make sure that this information stays in your building. Similarly, you might want to help your internal ops team troubleshoot a problem quickly without logging in. Send them the URL via email or Slack, for example, and rest assured that the information will remain in the right hands. If you decide to remove an IP address from your whitelist, any users connecting from that IP will no longer be able to view that dashboard. Complete Visibility through Audit Logs As an extra layer of transparency, you can keep track of which dashboards are shared outside of your organization and see which IPs are viewing them through your audit logs. With this information, you can: Configure real-time alerts to get notified anytime a user shares a dashboard Generate daily or weekly reports with a list of users and their shared dashboards Create dashboards of your shared dashboards – see where your dashboards are being viewed from so you can follow up on any suspicious activity. Receive alerts when someone shares a dashboard outside of your organization Use audit logs to see where your dashboards are being viewed from Learn More So go ahead – earn those bonus points with your boss and show off your dashboards today! Check out this webinar for a refresher on creating dashboards, then head over to Sumo Logic DocHub for more information on sharing these to users without an account.

March 10, 2017

Blog

Triggering AWS Lambda Functions from Sumo Logic Alerts