Sumo Logic Illuminate White RGB

October 6–7, 2020 | Illuminate conference—A virtual experience Learn more

Learn more

Andrew Male

Posts by Andrew Male

Blog

A Brief Tutorial to Understanding, Starting, and Using AWS Lambda

As cloud computing continues to become more and more accessible to businesses (and individuals), it can be a full-time job keeping up with all the tools released by the likes of Amazon, Microsoft, and others. One such tool is AWS Lambda, something that stands to revolutionize the accessibility (and affordability) of distributed computing. What Are AWS Lambda Functions? The short answer is that Lambdas are discrete blocks of code which are executed without having to manage the overhead of hardware and software involved in hosting that code. Your code can currently be written in either Python or Node.js, and can be uploaded with project-specific dependencies to facilitate a seemingly endless set of possibilities. Getting Started with AWS Lambda Sometimes the best way to understand a system is to simply start using it, so let’s dispense with the introductions and get right into some code. Since I much more frequently use JavaScript in my life, I’ll be doing my examples here in Node.js. The equivalent scripts in Python will rarely be more complicated, and Amazon provides plenty of examples in both languages to allow you the chance to learn. Once you get through the AWS Console and are creating a new Lambda function, you are first presented with a selection of templates. One of the templates available is a hello-world template in Node.js. Selecting it will bring you to a configuration screen with several options, as well as the following code: There are three important pieces of this code. First, the line of code that creates the function: exports.handler = function(event, context) { This code (and the associated closing bracket) is the heart of any Lambda. The Lambda system relies on having this exported handler to know where it will pass execution when the Lambda has been triggered. Second in line is the event argument for the handler, which receives any input provided by the trigger for the lambda. This example does a good job of showing how the following JSON would be accessible to the code: { “key3”: “value3”, “key2”: “value2”, “key1”: “value1” } The final piece that’s important is the context success/failure call: context.succeed(event.key1); That call lets the Lambda host environment know whether or not the lambda was able to complete its work. You can also include (as shown) additional information with your success/failure indication, which is important if you plan on debugging what you’ve created. Everything else in the code (in this case the console.log() calls) is superfluous as far as the Lambda environment is concerned. How to Use AWS Lambda Functions With the basics out of the way, let’s get into the nitty-gritty of showing how the system can be useful to you. Let’s consider the following situation: You are tasked with sending a notification email to a manager whenever someone visits a specific page on a website. For pre-existing reasons, you aren’t able to send email directly from the web server, so you have to create a system that can be remotely invoked to deliver emails. In the past, maybe you would create a service or cron job that constantly polled a database table for outgoing emails. Maybe you’d even create your own message queue handler to do the work on demand (much better), but you’re still stuck building and maintaining extra infrastructure. Enter Lambdas! Use Case: Our Email Handler Starting from the hello-world example provided by Amazon, I’m going to remove everything except the exported handler. I’ll then include the AWS SDK (thankfully built into the Lambda environment, so I can do this without uploading my code in a ZIP file), building a parameter object for use with my AWS SES (Simple Email Service) account, and attempt sending the email through SES before deciding if the operation was successful. Here is the completed code (and the explanation of the lines below): var AWS = require('aws-sdk'); exports.handler = function(event, context) { var params = { Destination: { ToAddresses: ['manager@domain.com'] }, Message: { Body: { Text: { Data: 'Someone has pinged the website!', Charset: 'utf8' } }, Subject: { Data: 'A Website Ping', Charset: 'utf8' } }, Source: 'noreply@domain.com', SourceArn: 'arn-here' }; new AWS.SES().sendEmail(params, function(err, data) { if (err) { context.fail('Failed to send email (' + err + ')'); } else { context.succeed('The website has been pinged'); } }); }; The first line accomplishes loading the AWS SDK for use later on in the script. We use our typical handler export, and inside the function build our SES parameters and attempt sending through the SDK. The parameter object for SES includes the bare minimum of options for an SES email, including the Destination, Message, Source, and SourceArn definitions. The SourceArn is part of AWS’ Identity and Access Management (IAM) system and indicates which email address the lambda is attempting to use for delivery. Lambda Tutorial Extra Credit Once this much has been done, we technically have a functioning lambda (hit ‘Test’ on the configuration page to be sure). However, getting it to work within our system would require at least creating an API endpoint to enable remote triggering of the lambda via a REST call. Additionally, there are ways to add triggers for other services within AWS, so choosing the best trigger approach will be based on your technology requirements. Other Common Lambda Uses? While utilizing the system for distributed (and thus asynchronous) email notifications/delivery is useful, it’s hardly the end of where Lambdas are useful. Some other ideas for great applications of the feature: On-demand distributed file conversion (change video file encodings, get metadata from uploads, etc) Index updates for services such as ElasticSearch Heavy computations for data analysis And these are just a few of the potential use cases where Lambdas will shine. The Brass Tacks of AWS Lambda Hopefully this has been a quick but educational introduction into the world of AWS Lambdas. We’ve shown how you can perform non-time-sensitive operations outside of the confines of your application with a small amount of effort. Lambdas can accomplish much more—so good luck, and have fun embracing distributed computing. Editor’s Note: A Brief Tutorial to Understanding, Starting, and Using AWS Lambda is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to visit the Sumo Logic Developersfor free tools, API’s and example code that will enable you to monitor and troubleshoot applications from code to production. About the Author Andrew Male (@AndyM84) is a senior engineer at an enterprise development company in Boston, MA. Andrew has been programming from a young age and is entirely self-taught; he has spent time in many corners of the programming world including game/VR work, agency work, and teaching development to students and adults alike. He spends most of his time working on architecture design and pursuing his favorite hobby—physics.

Blog

Using Bintray with Continuous Integration

Developing your company’s pipeline for continuous integration is a daunting and ever-evolving effort. As software becomes more complex, a need to distribute versioned internal libraries will often arise. Tools like NuGet and Maven provide excellent platforms for distribution, but running your own server can be an unnecessary hassle. Enter Bintray.What is Bintray?Bintray bills itself as “…package hosting and download center infrastructure for automated software distribution.” Perhaps a simpler way to think of it is as a way to manage and distribute your versioned products. The system provides integration with repository systems like NuGet, Maven, and Docker, making it easy to find ways of utilizing the service within your build and delivery processes. Paid subscriptions offer you additional features such as private repositories and historical statistics, while an open-source license still offers CDN-based distribution and public repositories that support formats like NuGet, NPM, YUM, and more.An Example: NuGet LibraryI spend a lot of time working in .NET and Visual Studio. The software I help build is extraordinarily complex and is comprised of over 100 different pieces. Not all of those pieces are proprietary, and some are very stable. We’ve found that in these situations, it is often helpful to repackage those components as libraries within NuGet.Here, we’ll pretend we have a library called FileHelper that is both open- source and stable, used throughout our code. We use TeamCity at work, so once the project has been separated from our main solution, we’ll have a build project configured for the library:One of the build steps would be a Powershell script that does various housekeeping before packing the project into a properly formed NuGet package (see this link for instructions). Once the package is built through NuGet commands, I’ll again use the NuGet executable to push my package to Bintray. The following notes should help you get through:Make sure you’re using a version of NuGet that is registered with your system path on Windows.Register your Bintray repository with NuGet on your machine (before trying to run this in a build step in TeamCity) using the steps described by Bintray.After you pack your nupkg file, use the described nuget push command to actually get your version pushed to Bintray’s repository.When you’ve finished all of this, you should see your new version (in this case, mine was the first version) listed on your package’s version section:Other Bintray FeaturesBintray has plenty of useful features if you subscribe. For starters, there is the ability to restrict access to packages using access tokens, which appear to work as though they were download keys. You can also white-label your downloads with your own domain, and you can even determine what countries are able to access your downloads.Like FTP, but BetterTaking care of binary distribution is an important step in the DevOps process, allowing a codebase to be much more flexible in how it handles dependencies and deployments. In the end, Bintray is a solution that can help alleviate several painful components of your development pipeline.Editor’s Note: Using Bintray with Continuous Integration is published by the Sumo Logic DevOps Community. If you’d like to learn more or contribute, visit devops.sumologic.com. Also, be sure to check out the Sumo Logic Developers Open Source page for free tools, API’s and example code that will enable you to monitor and troubleshoot applications from code to production.About the AuthorAndrew Male (@AndyM84) is a senior engineer at an enterprise development company in Boston, MA. Andrew has been programming from a young age and is entirely self-taught; he has spent time in many corners of the programming world including game/VR work, agency work, and teaching development to students and adults alike. He spends most of his time working on architecture design and pursuing his favorite hobby—physics.

July 27, 2016

Blog

Distributed Processing with Azure Functions

As distributed computing becomes more and more available, it’s good to evaluate the different options accessible to you and your organization. At Build 2016, Microsoft announced the upcoming availability of Azure Functions, an event-driven, on-demand compute system similar to AWS Lambda Functions. Azure Functions are still a new feature in Azure, so there are bound to be growing pains while they mature. With that in mind, let’s dig deeper! Building a Twitter Aggregator with Azure Functions I’ve always lived by the mantra that learning is best done by doing. Imagine that we are building a service that generates profile information for users by aggregating their content from different websites/services. We can create a Twitter aggregator as an Azure Function. This Function (which would be part of a group of Functions for aggregating content on-demand) will accept two arguments from a web request: ID and Handle. The ID will represent a mythical unique identifier created by our system elsewhere. The Handle will be the supplied Twitter handle/screen-name of the user who has requested a profile be generated. Once these arguments are supplied, the Function should connect to Twitter and pull down the five most recent tweets for use in the profile. Creating a new Function is incredibly simple. Head over to portal.azure.com/, and search for “Function App” in the marketplace to add a new resource. It is listed underneath “Web + Mobile.” Alternatively you can go to functions.azure.com/ and get a more streamlined creation process. Either way, you are presented with the ability to name your Function and choose things like its physical location and subscription. After that, you are taken straight to an editor where you can begin configuring and coding the Function. I’ll skip through the boring parts and get to the code because there’s a bit of effort involved due to working with Twitter. Here’s a link to the completed Function: I won’t go through every line of code, but there are important elements to discuss. Line #1 is #r “Newtonsoft.Json”, which is a way of referencing assemblies. You can provide new assemblies by taking advantage of the Continuous Integration options and tying your Function to a source control repository with a bin directory that contains pre-compiled DLLs. For more information, see the documentation here. You’ll notice that I’m not creating any new classes within my Function. This is not a requirement, as this is simply a C# “script.” You are free to create classes, but as of the time of this writing, you still cannot create namespaces for organizing your classes. (This can still be achieved by building libraries that your script uses, as referenced in #1). I am making use of the HttpRequestMessage.CreateResponse() method for my Function’s responses, but as I’ll discuss in a moment, it was the “quick and dirty” approach to building a new Function. There are other options that may fit your use cases better. Feel free to explore. I opted to not use any pre-built Twitter libraries for simplicity. I don’t generally recommend this, but it also isn’t terribly complicated for app-only usage, so make a decision based on your use case. Possibilities Everywhere Although I chose to build my Function in C#, Microsoft provides numerous language options including: C# F# Powershell Batch Bash JavaScript PHP Python There’s also a wide range of options for input and output. I set up my Function as a generic webhook, meaning it provides me with an API endpoint to call for triggering the Function. If that’s not your cup of tea, though, maybe one of these satisfies: Storage Blobs (trigger whenever a file is updated/added to an Azure Storage Container) Event Hubs (trigger whenever an event is received from an Azure Event Hub) GitHub WebHook (trigger whenever a GitHub webhook request is received) HttpTrigger (trigger based off of HTTP requests) QueueTrigger (trigger whenever a message is added to an Azure Storage Queue) ServiceBusQueueTrigger (trigger whenever a message is added to a ServiceBus instance) TimerTrigger (trigger on a schedule, similar to scheduled tasks or cron jobs) There is also a full set of options included to direct request output. For my Function, I simply assumed that returning the JSON as a web response was reasonable, but there are likely much better/scalable ways of directing the output into our profiles. See the full documentation for more information. In Closing While Sumo Logic doesn’t yet provide an integration for Azure Functions, you can get visibility into other Microsoft technologies using tools like the Sumo Logic Application for Office 365 (currently in Preview) that offersPre-built dashboards, searches and reports visually highlight key activity across your O365 environment including OneDrive, Exchange Online, SharePoint, and Azure Active Directory. Check out other apps like the App for Windows Directory at sumologic.com. Hopefully this has been a simple introduction to the concept of Azure Functions. The sheer number of things that can be accomplished with them is daunting, but they are fantastic for the world of distributed computing. The documentation and community for Functions is already growing, so don’t be afraid to take them for a spin. Resources Complete Source for Example About the Author Andrew Male (@AndyM84) is a senior engineer at an enterprise development company in Boston, MA. Andrew has been programming from a young age and is entirely self-taught. He has spent time in many corners of the programming world, including game/VR work, agency work, and teaching development to students and adults alike. He spends most of his time working on architecture design and pursuing his favorite hobby—physics.