---
id: collect-logs
title: Collect Logs from Azure Blob Storage (block blobs)
sidebar_label: Collect Logs using ARM
description: Configure a pipeline to ship logs from the Azure Blob Storage to an HTTP source on a hosted collector in Sumo Logic.
slug: /help/docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/collect-logs/
canonical: https://www.sumologic.com/help/docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/collect-logs/
---
import useBaseUrl from '@docusaurus/useBaseUrl';
import Iframe from 'react-iframe';
:::info
This section has instructions for configuring a pipeline for shipping logs available from Azure Blob Storage to an Event Hub, on to an Azure Function, and finally to an HTTP source on a hosted collector in Sumo Logic.
:::
## Requirements
* Only General-purpose v2 (GPv2) and Blob storage accounts are supported. This integration does not support General-purpose v1 (GPv1) accounts.
* Configure your storage account in the same location as your Azure Service.
* This solution supports only log files from Blob storage that have file extensions of .csv, .json, .blob, or .log.
* You also need to have Microsoft Authorization/role Assignments/write permissions, so they should be a "User Access Administrator" or "Owner".
:::training Micro Lesson
Watch this tutorial to learn how to collect logs from Azure Blob Storage.
:::
## Functional overview
1. You configure the Azure service to export logs to a container in a storage account created for that purpose.
1. The ARM template creates an Event Grid subscription with the storage container as publisher and the event hub (created by the Sumo Logic-provided ARM) as subscriber. Event Grid routes block blob creation events to event hub.
1. Event Hub streams the events to the BlobTaskProducer Azure function, which creates tasks (a JSON object that specifies start and end byte, container name, blob path) and pushes those tasks to the service bus task queue.
1. The BlobTaskConsumer Azure function, which is triggered when the service bus receives a new task, reads the block blob, from start byte to stop byte, and sends that data to Sumo Logic.
1. The set up also includes failure handling mechanism. For more information about the solution strategy, see [Azure Blob Storage(block blob)](/docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob).
## Step 1. Configure Azure storage account
In this step, you configure a storage account to which you will export monitoring data for your Azure service. The storage account must be a General-purpose v2 (GPv2) storage account.
If you have a storage account with a container you want to use for this purpose, make a note of its resource group, storage account name and container name, and proceed to [Step 2](#step-2-configure-an-http-source).
To configure an Azure storage account, do the following:
1. Create a new storage account General-purpose v2 (GPv2) storage account. For instructions, see [Create a storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=portal ) in Azure help.
1. Create a container(Optional) all services in azure create containers automatically. This step is needed only when you are exporting custom logs in some container.
* In the Azure portal, navigate to the storage account you just created (in the previous step).
* Select **Blobs** under **Blob Service**.
* Select **+ Container**.
* Enter the Name.
* Select **Private** for the Public Access Level.
* Click **OK**.
:::note
Make a note of the container name, as you will need to supply it later.
:::
## Step 2. Configure an HTTP source
In this step, you configure an HTTP source to receive logs from the Azure function.
1. Select a hosted collector where you want to configure the HTTP source. If desired, create a new hosted collector, as described on [Configure a Hosted Collector](/docs/send-data/hosted-collectors/configure-hosted-collector).
:::note
Make sure the hosted collector is tagged with tenant_name field for the out of the box Azure apps to work. You can get the tenant name using the instructions [here](https://learn.microsoft.com/en-us/azure/active-directory-b2c/tenant-management-read-tenant-name#get-your-tenant-name).
:::
1. Configure an HTTP source, as described on [HTTP Logs and Metrics Source](/docs/send-data/hosted-collectors/http-source/logs-metrics). Make a note of the URL for the source, you will need it in the next step.
## Step 3. Configure Azure resources using ARM template
In this step, you use a Sumo Logic-provided Azure Resource Manager (ARM) template to create an Event Hub, three Azure functions, Service Bus Queue, and a Storage Account.
1. Download the [blobreaderdeploy.json](https://raw.githubusercontent.com/SumoLogic/sumologic-azure-function/master/BlockBlobReader/src/blobreaderdeploy.json) ARM template.
:::note
The above template uses Consumption Plan which does not support VNet integration, you can use [blobreaderdeploywithPremiumPlan.json](https://raw.githubusercontent.com/SumoLogic/sumologic-azure-function/master/BlockBlobReader/src/blobreaderdeploywithPremiumPlan.json) which uses Elastic Premium plan.
:::
1. Click **Create a Resource**, search for **Template deployment** in the Azure Portal, and then click **Create.**
1. On the Custom deployment blade, click **Build your own template in the editor**.
1. Copy the contents of the template and paste it into the editor window.
})
1. Click **Save**.
1. On the Custom deployment blade, do the following:
1. Create a new Resource Group (recommended) or select an existing one.
1. Choose Location.
1. Set the values of the following parameters:
* **DeployingAgainForSameStorageAccount**. Choose 'yes' if you are deploying the ARM template again for the same storage account, else choose 'no'.
* **EventGridSystemTopicName**. If you are deploying the ARM template again for the same storage account, then provide the existing System Topic Name for the StorageAccount, else it will be created automatically.
* **SumoEndpointURL**. URL for the HTTP source you configured in [Step 2](#step-2-configure-an-http-source) above.
* **StorageAccountName**. Name of the storage account where you are storing logs from Azure Service that you configured in [Step 1](#step-1-configure-azure-storage-account) above.
* **StorageAccountResourceGroupName**. Name of the resource group of the storage account you configured in [Step 1](#step-1-configure-azure-storage-account) above.
* **StorageAccountRegion**. Name of the region of the storage account you configured in [Step 1](#step-1-configure-azure-storage-account) above.
* **Filter Prefix** (Optional). If you want to filter logs from a specific container, enter the following by replacing the variable with your container name: `/blobServices/default/containers//`
:::note
Resource group names should not consist of an underscore.
:::
1. Go to the **Review + create** tab, and then click **Create**.
})
1. Verify that the deployment was successful by looking at **Notifications** at the top right corner of the Azure Portal.
})
1. (Optional) In the same window, click **Go to resource group** to verify that all resources were successfully created, such as shown in the following example:
})
Example: Push NSG flow logs from a Network Security Group to Azure Blob Storage
This section describes how to push logs from a network security group into Azure Blob Storage by configuring nsg flow Logs. The instructions use a network security group as an example.
* Login to the Azure Portal.
* Click **Network security groups > Select a network security group**.
* Click on **NSG flow logs** when you see it under **Monitoring**, and click **Create**.
* Click on **Select resource** and choose a NSG that is present in the same region as the storage account configured in [Step 1](#step-1-configure-azure-storage-account).
* Under **Subscription > Storage Accounts**, select the storage account configured in [Step 1](#step-1-configure-azure-storage-account).
* Specify the **Retention (days)** and click **Review + create**.
})
* Review the configuration of the flow log and click **Create**.
})
:::tip
If logs from Azure Blob Storage do not start to flow into Sumo Logic, see the [Troubleshoot Azure Storage Log Collection](/docs/send-data/collect-from-other-data-sources/azure-blob-storage/troubleshoot-log-collection).
:::
## Ingesting from Multiple Storage Accounts (Optional)
If you want to ingest data into Sumo Logic from multiple storage accounts, perform following tasks for each storage account separately.
:::note
The following steps assume you have noted down the resource group name, storage account name, and container name where the blobs will be ingested from.
:::
* [Step 1: Authorize App Service read from storage account](#step-1-authorize-app-service-to-read-from-storage-account) - Enables the Azure functions to read from the storage account.
* [Step 2: Create an Event Grid Subscription](#step-2-create-an-event-grid-subscription) - Subscribes all blob creation events to the Event Hub created by ARM template in [Step 3](#step-3-configure-azure-resources-using-arm-template) above.
### Step 1: Authorize App Service to read from storage account
This section provides instructions on authorizing the App Service to list the Storage Account key. This enables the Azure function to read from the storage account.
To authorize the App Service to list the Storage Account key, do the following:
1. Go to **Storage Account** and click **Access Control(IAM)**.
})
1. Click the **Add** **+** at the top of the page.
})
1. Select **Add role assignment** from dropdown.
1. In the **Add role assignment** window, go to **Role** tab and choose **Storage Blob Data Reader**. Click **Next**.
})
1. In **Members** tab, select **Managed Identity**.
1. In the **Select Managed identities** window,
* **Subscription**: Choose Pay as you Go.
* **Managed Identity**: Choose Function App.
* **Select**: **Select SUMOBRDLQProcessor\** and **SUMOBRTaskConsumer\** app services which are created by the ARM template. Click **Select**.
1. Click **Review + assign**
1. Click **Save**.
### Step 2: Create an Event Grid Subscription
This section provides instructions for creating an event grid subscription, that subscribes all blob creation events to the Event Hub created by ARM template in [Step 3](#step-3-configure-azure-resources-using-arm-template) above.
To create an event grid subscription, do the following:
1. Go to the storage account which needs to be monitored additionally. Go under Events blade in left pane.
1. At the top of the **Event subscriptions** tab, click **+Event Subscription** to create new event subscription.
})
1. Specify the following values for **Event Subscription Details**:
* **Name:** Fill the event subscription name.
* **Event Schema:** Select **Event Grid Schema**.
1. Specify the following values for **Topic Details**:
* **System Topic Name**. Provide the topic name, if the system topic already exists then it will automatically select the existing topic.
1. Specify the following details for Event Types:
* Select **Blob Created** from the **Filter to Event Types** dropdown.
1. Specify the following details for Endpoint Types:
* **Endpoint Type**. Select **Event Hubs** from the dropdown.
* **Endpoint.** Click on **Configure an endpoint.**
The Select Event Hub dialog appears.
})
1. Specify the following Select Event Hub parameters, then click **Confirm Selection.**
* **Resource Group**. Select the resource group you created [Step 3](#step-3-configure-azure-resources-using-arm-template) in which all the resources created by ARM template are present.
* **Event Hub Namespace**. Select **SUMOBREventHubNamespace\<*unique string*\\>**.
* **Event Hub**. Select **blobreadereventhub** from the dropdown.
1. Specify the following Filters tab options(Optional):
* Check Enable subject filtering.
* To filter events by container name, enter the following in the **Subject Begins With** field, replacing `` with the name of the container from where you want to export logs. `/blobServices/default/containers//`
})
1. Click **Create**.
1. Verify the deployment was successful by checking **Notifications** in the top right corner of the Azure Portal.
## Enabling VNet Integration (Optional)
This assumes that your storage account access is not public and is enabled for selected networks i.e. your storage account is behind a virtual network. This requires you to used the modified template which uses standard/premium plan for BlobTaskConsumer and DLQTaskConsumer functions. In case you want the whole data pipeline sending logs to sumo logic, to be under a virtual network follow the instruction [here](/docs/send-data/collect-from-other-data-sources/azure-blob-storage/block-blob/full-vnet-integration.md).
1. Create a subnet in a virtual network using the instructions in the [doc](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-manage-subnet#add-a-subnet). If you have multiple accounts in the same region you can skip step 2 below and use the same subnet and add it to the storage account as mentioned in step 3.
1. Perform below steps for both BlobTaskConsumer and [DLQTaskConsumer](https://portal.azure.com/#blade/WebsitesExtension/FunctionMenuBlade/resourceId/%2Fsubscriptions%2Fc088dc46-d692-42ad-a4b6-9a542d28ad2a%2FresourceGroups%2Fleast%2Fproviders%2FMicrosoft.Web%2Fsites%2FSUMOBRDLQProcessorekbxzlepnhs4g%2Ffunctions%2FDLQTaskConsumer) function apps.
1. Go to **Function App > Settings > Networking**.
1. Under Outbound traffic, click on Vnet Integration.
})
1. Add the Vnet and subnet created in Step 1.
})
1. Go to your storage account from where you want to collect logs from. Go to Networking and add the same Vnet and subnet.
})
1. Verify by going to the subnet. You should see Subnet delegation and service endpoints as shown in the screenshot below.
})
## Upgrading Azure Functions
1. Go to the resource group where ARM template was deployed and go to each of the function apps.
})
1. Go to `Deployment -> Deployment Center` and click on `Sync`.
})
1. Go to `Logs` tab and check the `Status` column, it should show `Success`.
})
## Collection testing performance numbers
| File creation time in a single storage account | Number of files | Size of each file X Number of files | Sumo Logic Incoming Bytes Rate (MB/sec) | Sumo Logic Incoming Messages Rate (loglines/sec) | Sumo Logic total time taken for full ingestion | Sumo Logic Ingestion (GB) | Sumo Logic log count |
| :-- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| ~3 minutes | 100 | 100MB X 80 + 104MB X 20 | Avg 40MB/sec
MAX 190MB/sec | MAX 411811/sec | 3.77 minutes | 9.31 | 20,643,840 |