--- title: "Kauto subscribing cloudwatch log groups to AWS lambda function" page_name: "kAuto Subscribing CloudWatch Log Groups to AWS Lambda Function" type: "blog" slug: "auto-subscribing-log-groups-lambda" published_at: "2018-02-14" modified_at: "2025-05-09" url: "https://www.sumologic.com/blog/auto-subscribing-log-groups-lambda" canonical: "https://www.sumologic.com/blog/auto-subscribing-log-groups-lambda" markdown_url: "https://www.sumologic.com/blog/auto-subscribing-log-groups-lambda.md" lang: "en" excerpt: "Sumo's Log Group Lambda Connector automates the process of creating AWS CloudWatch Log Group subscriptions.This function has multiple use cases like subscribing log groups for Sumo Logic CloudWatch Lambda Function, creating Subscription Filters with Kinesis etc." taxonomy_blog_category: - "AWS" --- [ All blogs ](https://www.sumologic.com/blog "blog")[AWS](https://www.sumologic.com/blog/aws) # kAuto Subscribing CloudWatch Log Groups to AWS Lambda Function [Himanshu Pal](#blog-author-block-146) February 14, 2018 4 min read [AWS](https://www.sumologic.com/blog/aws) ##### Table of contents Serverless computing has enabled developers to develop their next-generation products from ideation to production, without waiting for, or worrying about, infrastructure. [AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html) and [AWS CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html), both services provided by AWS, are one of the important components in serverless architectures. I’ll assume that you know what they are and how to use them and some [jargon](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogsConcepts.html) that accompanies them. It’s no surprise that a large number of our customers are utilizing them and are generating huge amount of logs in Log Groups which they want to ingest into Sumo for monitoring and troubleshooting. Currently customers use AWS console and manually create a subscription filter to subscribe those Log groups to our [Sumo Logic Lambda function for CloudWatch Logs](https://github.com/SumoLogic/sumologic-aws-lambda/tree/master/cloudwatchlogs). But what if they have tons of Log Groups consequently, this small task can soon become a DevOps nightmare. The seriousness of the customer’s plight was instantly recognized and our team developed a Lambda function, [LogGroup Lambda Connector](https://github.com/SumoLogic/sumologic-aws-lambda/tree/master/loggroup-lambda-connector) for automating the process of creating AWS CloudWatch Log Group subscriptions.This solution is generic enough to be used with any lambda function not necessarily Sumo Logic Lambda functions (except for logs generated by function itself).For more information and step by step instructions checkout its [documentation](https://help.sumologic.com/?cid=39393). Ready to do even more with AWS Lambda? See how the [Sumo Logic App for AWS Lambda](https://www.sumologic.com/app-catalog/aws-lambda) drives continuous intelligence with real-time analytics or [request a demo](https://www.sumologic.com/request-demo) to see it in action yourself. ## Configuring and Deploying the LogGroup Lambda Connector using CloudFormation template This lambda function provides following environment variables 1. **LOG\_GROUP\_PATTERN**: This is a javascript regex to filter out Log Groups and only the matched ones will be subscribed to the lambda function. Since the function uses constructor to create a RegExp object, the normal string escape rules (preceding special characters with when included in a string) are necessary. For example 1. Input Pattern: test will result in /test/ and it will match testlogroup, logtestgroup and LogGroupTest 2. **LAMBDA\_ARN**: This specifies ARN of the lambda function. To simplify it we have already provided json snippet and you have to specify FunctionName attribute of your lambda function in the given placeholder. | { “Fn::Join”: \[ “”, \[ “arn:aws:lambda:”, { “Ref” : “AWS::Region” }, “:”, { “Ref” : “AWS::AccountId” }, “:function:<Your Lambda Function Name>” \] \]} | |---| To automate the deployment we have created a cloudformation template([loggroup-lambda-cft.json](https://github.com/SumoLogic/sumologic-aws-lambda/blob/master/loggroup-lambda-connector/loggroup-lambda-cft.json)) which creates following resources: 1. **PermissionForEventsToInvokeLambda**: Permission to CloudTrail events for invoking our lambda function(*SumoLogGroupLambdaConnector*). 2. **SumoLGCnCreateLogGroupTrigger**: CloudTrail Event Rule which triggers our lambda function(*SumoLogGroupLambdaConnector*) on **CreateLogGroup** event. 3. **SumoLGCnLambdaExecutionRole**: IAM Role for our lambda function function which defines permissions to create subscription filter and cloudwatch logs. 4. **SumoLogGroupLambdaConnector**: Lambda function responsible for creating a subscription filter on Log Groups matching filter criteria. After completing above configuration you can upload the [loggroup-lambda-cft.json](https://github.com/SumoLogic/sumologic-aws-lambda/blob/master/loggroup-lambda-connector/loggroup-lambda-cft.json) via AWS console (see [docs](https://help.sumologic.com/Send-Data/Collect-from-Other-Data-Sources/Auto-Subscribe_AWS_Log_Groups_to_a_Lambda_Function#Step_3:__Create_a_stack)) or use the following command in aws cli. `aws cloudformation create-stack --stack-name "teststack" --template-body "file:////loggroup-lambda-cft.json" --region="us-east-2" --capabilities=CAPABILITY_IAM` Now to test our function we will create a log group say **testloggroup** assuming *test* is our log group pattern We assume that our lambda function already exists(say *TestLambda*) whose name is configured in **LAMBDA\_ARN** Now observe after few seconds Subscriptions columns gets populated with subscription filter. Following are some of its use cases: ## Subscription Filters with Sumo Logic CloudWatch Logs Lambda Function Currently CloudWatch Lambda function provides a Log Group (*SumoCWLogGroup*) which has a subscription filter (*SumoCWLogSubsriptionFilter*) associated with it. Now consider these scenarios, currently you have a log group /var/log/syslog from EC2 instance and now you have added /var/log/messages, /var/log/boot.log, /var/log/auth.log(new log groups) in awslogs configuration or you have recently deployed another lambda function and it’s logs are sent to /aws/lambda/newlambda log group. If you wanted to send those logs to Sumo and assuming you knew there pattern(/var/log/\* and /aws/lambda/\*) it’s easy to configure it in **LOG\_GROUP\_PATTERN** and specify *SumoCWLogsLambda* as function attribute in **LAMBDA\_ARN** and you are done, any log groups matching this criteria will be subscribed to Sumo Logic CloudWatch Lambda function and will be ingested to sumo thereby avoiding the likelihood of forgetting to create those subscriptions. One thing to note here is that you must provide CloudWatch logs permission to invoke your lambda function. See example below `aws lambda add-permission --function-name "helloworld" --statement-id "helloworld" --principal "logs.region.amazonaws.com" --action "lambda:InvokeFunction" --source-arn "arn:aws:logs:region:123456789123:log-group:TestLambda:*" --source-account "123456789012"` ## Subscription Filters with Kinesis Though this function is mainly built to be used with lambda functions but you can specify any aws resource arn in **LAMBDA\_ARN** environment variable. For AWS Kinesis update the role with below trust policy. `aws iam update-assume-role-policy --role-name KinesisRoleName --policy-document file://trust_policy.json` Here is the trust\_policy.json | { “Statement”: { “Action”: “sts:AssumeRole”, “Effect”: “Allow”, “Principal”: { “Service”: “logs.*region*.amazonaws.com” } } }} | |---| Associate the role with following permissions policy which defines what actions CloudWatch Logs can do on your account. `aws iam put-role-policy --role-name KinesisRoleName --policy-name Permissions-Policy-For-CWL --policy-document file://PermissionsForCWL.json` Below is the permissions\_for\_cwl.json | { “Statement”: \[ { “Effect”: “Allow”, “Action”: “kinesis:PutRecord”, “Resource”: “arn:aws:kinesis:us-east-2:456227676011:stream/teststream“ }, { “Effect”: “Allow”, “Action”: “iam:PassRole”, “Resource”: “arn:aws:iam::456227676011:role/KinesisRoleName” } \] } | |---| After setting up the environment variables and deploying log group lambda connector function, all the incoming log events from log groups matching the filter pattern are forwarded to your Kinesis stream which can be used for real time event processing. ### Article Tags - [AWS](https://www.sumologic.com/blog/aws) Himanshu Pal Himanshu Pal is an application developer at Sumo Logic where he focuses on building serverless solutions for customers. He has a deep passion for all things technology and these days he is interested in Go, distributed systems and serverless architectures. He has a bachelor’s degree in Information Technology from IIIT Allahabad. In his free time he enjoys trekking, swimming, traveling and watching TV series. [](https://www.sumologic.com/feed "RSS Feed")[](https://twitter.com/intent/tweet?text=kAuto%20Subscribing%20CloudWatch%20Log%20Groups%20to%20AWS%20Lambda%20Function&url=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fauto-subscribing-log-groups-lambda "X")[](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fauto-subscribing-log-groups-lambda "Facebook")[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Fauto-subscribing-log-groups-lambda "Linkedin") [Previous blog How Much Data Comes From The IOT?](https://www.sumologic.com/blog/iot-data-volume)[Next blog 4 Reasons Why I Chose Azure: A Developer’s Perspective](https://www.sumologic.com/blog/why-azure) People who read this also enjoyed [ Eight best practices for a successful cloud migration strategy June 16, 2026 ](https://www.sumologic.com/blog/best-practices-for-cloud-migration-strategy)[ OCSF for Security Hub: Sumo Logic and AWS speaking the same language September 30, 2025 ](https://www.sumologic.com/blog/sumo-logic-aws-ocsf-security-hub)[ Enhance your cloud security visibility with the updated AWS CloudTrail app July 16, 2025 ](https://www.sumologic.com/blog/enhance-your-cloud-security-visibility-with-updated-aws-cloudtrail-app)[ How to create and monitor an AWS Lambda function in Java 11 April 15, 2025 ](https://www.sumologic.com/blog/how-to-write-aws-lambda-function-in-java8) [AI Instructions](https://www.sumologic.com/ai-instructions.md)