--- id: forward-data-from-sumologic title: Forward Data from Sumo Logic to S3 or GCS description: Configure data forwarding from Sumo Logic partitions or scheduled views to Amazon S3 or Google Cloud Storage (GCS) buckets in CSV or JSON format. slug: /help/docs/manage/data-forwarding/forward-data-from-sumologic/ canonical: https://www.sumologic.com/help/docs/manage/data-forwarding/forward-data-from-sumologic/ --- import useBaseUrl from '@docusaurus/useBaseUrl'; :::note Data forwarding to Google Cloud Storage (GCS) is an on-demand feature that requires an additional fee. To enable this capability, contact your Sumo Logic executive or [Support](https://support.sumologic.com/support/s). ::: This document outlines the instructions that needs to be followed to forward log data from a [partition](/docs/manage/partitions) or [scheduled view](/docs/manage/scheduled-views) to an S3 or Google Cloud Storage (GCS) bucket. Only new data is forwarded from a partition or scheduled view once it is set to forward data.  To forward data to a storage bucket: 1. [Configure forwarding destination](#configure-data-forwarding-destination). 1. [Forward data to destination](#forward-datato-forwarding-destination) from a partition or schedule view. After data forwarding is configured, you should start to see file objects posted within your configured bucket. If your scheduled view conducts aggregation, which is a best practice, your aggregate fields are automatically appended to the forwarded objects. :::note Data forwarding is not currently supported for data assigned to the Infrequent Tier.  ::: ## Prerequisites * An administrator role on the partition where you want to set up forwarding. * Follow the instructions on [Grant Access to an AWS Product](/docs/send-data/hosted-collectors/amazon-aws/grant-access-aws-product) to grant Sumo Logic permission to send data to the destination S3 bucket. * A partition or scheduled view to push to Amazon S3 or Google Cloud Storage (GCS). ## Forwarding interval  Messages are buffered during data ingest for either approximately five minutes or until 100MB of data is received, whichever is first. Then the buffered data is written to a new CSV file and forwarded after compression.  The limits mentioned here are upper limits. Actual file size may vary depending on the ingestion volume in scheduled views or partitions of an account.  :::note It takes approximately five minutes to propagate a new or changed data forwarding rule or bucket across the Sumo Logic service. So, it is possible after you create or modify a rule, the first five minutes of data forwarded might not be written to S3 or GCS. ::: ## File format of forwarded data After you start forwarding data, you should start to see file objects posted in your configured bucket. The log messages are accumulated and returned after being ingested by Sumo Logic. You can choose to forward only log data, log data and metadata, or log data with metadata and enriched fields, in either CSV or JSON format. The log messages are saved in CSV or JSON files in compressed gzip files and named according to the convention you specified when you configured Sumo Logic to start data forwarding. The file naming convention for legacy data forwarding is described below in [Legacy file naming format](#legacy-file-naming-format).  Messages are buffered during data ingest for either approximately five minutes or until 100MB of data is received, whichever is first. Then the buffered data is written to a new CSV or JSON file and forwarded.  These file objects will contain the messages received as well as the system metadata for the messages, including: * **messageId**: The unique ID for the specific message within Sumo Logic. * **sourceName**: Is returned blank. * **sourceHost**: Is returned blank. * **sourceCategory**: Is returned blank. * **messageTime**: The parsed message time from the log message, as epoch. * **receiptTime**: The time the service originally received the message, as epoch. * **sourceID**: The unique ID of the source configured to send the message to the service. * **collectorId**: The unique ID of the collector configured to send the message to the service. * **count**: The message number from the specific log source name. These should be sequential for a specific source file. * **format**: The timestamp format used to parse the message time from the log message. * **view**: The scheduled view or partition that the message is forwarded from. * **encoding**: The encoding of the original file contents. * **message**: The raw log message as read from the original source. * **field**: Aggregate fields are added based on your query. ### Ordering of fields in forwarded file * The order of the system fields is fixed, and the order is: `messageId, sourceName, sourceHost, sourceCategory, messageTime, receiptTime, sourceId, collectorId, count, format, view, encoding, message`. * Aggregate fields are represented in lowercase only. * Aggregate fields are ordered based on ascending ASCII value. * Aggregate fields are always present after the system or built-in fields. ### Example When forwarding data from Sumo Logic, the system will write structured logs that include the original message being forwarded, as well as additional metadata and quotation marks as seen in a structured JSON file. **Metadata fields** `messageId,sourceName,sourceHost,sourceCategory,messageTime,receiptTime,sourceId,collectorId,count,format,view,encoding,message,aggregatefield1,aggregatefield2` **Sample object** `"-9223371513354977010","","","","1472590091453","1472590094034","101688020","100607825","979","plain:atp:o:0:l:29:p:yyyy-MM-dd HH:mm:ss,SSSZZZZ","JchenTest2","UTF8","2016-08-30 13:48:11,453 -0700 WARN [hostId=nite-cqsplitter-1] [module=cqsplitter] [localUserName=cqsplitter] [logger=cqsplitter.engine.CQsMultiMatchersManager] [thread=DTP-cqsplitter.receiver.consumer.v2.threadpool-6] MultiMatcher queue for customer 0000000000000131 is at capacity, adding element will block.","25","0000000000000131"` ### Legacy file naming format The file naming convention for legacy data forwarding (prior to January 2017) is: `---.csv.gz` Where: * `start_epoch` is the epoch time representing the parsed message time of the first message contained within the file. * `end_epoch` is the epoch time representing the parsed message time of the last message contained within the file. * `objectid` is a unique ID for the file object, which is generated by Sumo Logic at creation time. ## Configure data forwarding destination Before you can [forward data](#forward-datato-forwarding-destination) from a partition or scheduled view, you must create a destination that indicates the S3 or GCS bucket where you want to send the forwarded data. 1. [**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Manage Data**, and then under **Logs** select **Data Forwarding**. You can also click the **Go To...** menu at the top of the screen and select **Data Forwarding**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Data Forwarding**. 1. Click **+ Add Destination** to add a new destination. 1. The **Create New Destination** popup appears. 1. **Destination Type**. You can either select **Amazon S3** or **Google Cloud Storage** as your destination type. - For **Amazon S3** as the destination type, follow the below steps:
Create S3 Destination popup 1. **Destination Name**. Enter a name to identify the destination. 1. **Bucket Name**. Enter the [exact name of the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/view-bucket-properties.html). :::note You can create only one destination with a particular bucket name.  If you try to create a new destination with the bucket name of an existing destination, the new destination replaces the old one. ::: 1. (Optional) **Description**. You can provide a meaningful description of the connection. 1. **Access Method**. Select **Role-based access** or **Key access** based on the AWS authentication you are providing. Role-based access is preferred. This was completed in the prerequisite step [Grant Access to an AWS Product](/docs/send-data/hosted-collectors/amazon-aws/grant-access-aws-product). * For **Role-based access** enter the Role ARN that was provided by AWS after creating the role. * For **Key access** enter the **Access Key ID** and **Secret Access Key**. See [Manage access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) for details. 1. **S3 Region**. Select the S3 region or keep the default value of Others. The S3 region must match the appropriate S3 bucket created in your Amazon account. 1. **Enable S3 server-side encryption**. Select the check box if you want the forwarded data to be encrypted. For more information, see [Using server-side encryption with Amazon S3 managed keys (SSE-S3)](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html) in AWS help. - For **Google Cloud Storage** as the destination type, follow the below steps:
Create S3 Destination popup 1. **Destination Name**. Enter a name to identify the destination. 1. **Bucket Name**. Enter the [exact name of the S3 or GCS bucket](https://cloud.google.com/storage/docs/buckets). :::note You can create only one destination with a particular bucket name.  If you try to create a new destination with the bucket name of an existing destination, the new destination replaces the old one. ::: 1. (Optional) **Description**. Provide a meaningful description of the connection. 1. For **HMAC Access Key** and **HMAC Secret Key** enter the values collected from the Google platform service account. See [Manage HMAC keys for service account](https://cloud.google.com/storage/docs/authentication/managing-hmackeys) for details. 1. **Active**. Select this check box to enable data forwarding for the entire bucket. To start forwarding data, you will also need to enable forwarding for the desired indexes, as described below. 1. Click **Save**.
If Sumo Logic is able to verify the credentials, the destination will be added to the list of destinations. If the destination is not added successfully, see [Error and alert conditions](#error-and-alert-conditions) for examples of errors that can occur. Once the destination is created, you can start data forwarding for specific partitions or scheduled views as described in [Forward data to forwarding destination](#forward-datato-forwarding-destination) below. ## Forward data to forwarding destination Once you [configure the data forwarding destination](#configure-data-forwarding-destination) that indicates the bucket to receive the data, you can forward data to the destination from partitions and scheduled views. 1. Depending on whether you want to forward data from a partition or a scheduled view: * **Partition**:
[**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Manage Data**, and then under **Logs** select **Partitions**. You can also click the **Go To...** menu at the top of the screen and select **Partitions**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Partitions**. * **Scheduled view**:
[**New UI**](/docs/get-started/sumo-logic-ui/). In the main Sumo Logic menu select **Manage Data**, and then under **Logs** select **Scheduled Views**. You can also click the **Go To...** menu at the top of the screen and select **Scheduled Views**.
[**Classic UI**](/docs/get-started/sumo-logic-ui-classic/). In the main Sumo Logic menu, select **Manage Data > Logs > Scheduled Views**. 1. Select the partition or scheduled view for which you want to enable data forwarding and click the **Edit** button. The edit dialog for the partition or scheduled view displays. Following is the edit dialog for a partition.
Enable Data Forwarding checkbox :::tip In addition to forwarding data from existing partitions and scheduled views, you can also enable data forwarding by selecting the **Enable Data Forwarding** check box when you first [create a partition](/docs/manage/partitions/flex/create-edit-partition-flex/) or [create a scheduled view](/docs/manage/scheduled-views/add-scheduled-view/). ::: 1. Click the **Enable Data Forwarding** checkbox. More options appear. 1. **Destination Type**. You can either select **Amazon S3** or **Google Cloud Storage** as your destination type. - For **Amazon S3** as the destination type, follow the below steps:
Forwarding destination options 1. **Forwarding Destination**. Choose one of the following: * **Existing Amazon S3 Destination**. If you select this option, select the destination in the **Amazon S3 Destination** field below. * **New Amazon S3 Destination**. Follow the instructions in [Configure data forwarding destination](#configure-data-forwarding-destination) above to create a new S3 destination. 1. **Amazon S3 Destination**. If you chose **Existing Amazon S3 Destination** for the forwarding destination, select the destination here. - For **Google Cloud Storage** as the destination type, follow the below steps:
Forwarding destination options 1. **Forwarding Destination**. Choose one of the following: * **Existing Google Cloud Storage Destination**. If you select this option, select the destination in the **Google Cloud Storage Destination** field below. * **New Google Cloud Storage Destination**. Follow the instructions in [Configure data forwarding destination](#configure-data-forwarding-destination) above to create a new S3 destination. 1. **Google Cloud Storage Destination**. If you chose **Existing Google Cloud Storage Destination** for the forwarding destination, select the destination here. 1. Click **Data Forwarding Configuration**. Options appear for forwarding the data.
Options to forward raw data 1. **Included Data**. Select the kind of data to forward: * **Raw**. Raw logs only. * **Raw + Metadata**. Raw logs and the metadata fields assigned to log entries. We recommend this option because the forwarded data has the optimal balance of raw data and metadata that Sumo Logic adds (for example, to indicate source, source category, and so on). * **All (Raw + Metadata + Enriched Fields)**. Raw logs, the metadata fields assigned to log entries, and enriched fields from field extraction rules. 1. **Forwarded data type**. Select the format for the forwarded data: * **Text**. Plain text. (Available only if you choose **Raw** above.) * **CSV**. Comma-separated values. (Available if you choose **Raw + Metadata** or **All** above.) * **JSON**. Java Script Object Notation. (Available if you choose **Raw + Metadata** or **All** above.) Select **JSON** if you want to ensure that forwarded data can be re-ingested easily. 1. **File Prefix**. Enter the path prefix to a directory in the S3 or GCS bucket. You can include any of the following variables: * `{index}` will be replaced by the name of the partition or scheduled view. * `{day}` will be replaced by the day of the year in the yyyy-MM-dd format. * `{hour}` will be replaced by the hour of the day (0-23). * `{minute}` will be replaced by the minute of the hour. * `{second}` will be replaced by the second of the minute. * `{uuid}` will be replaced by a randomly generated universal unique identifier.

:::note For example, to place data in a directory named `SumoDataForwarding` you could specify the **File Prefix** as: `SumoDataForwarding/{day}/{index}_{day}_{hour}_{minute}_{second}`
If you leave this field blank, the default format is used: `{index}_{day}_{hour}_{minute}_{second}` ::: 1. Click **Save** at the top of the panel to save your changes and start forwarding data.  For information about how the data is forwarded, see [Forwarding interval](#forwarding-interval) and [File format of forwarded data](#file-format-of-forwarded-data). ## Data forwarding example Let's say you want to take data from Sumo Logic and run additional analysis on it in tools separate from Sumo Logic. In this example, you can forward the data from Sumo Logic to an S3 or GCS bucket where it is available for download and analysis by your tools. Let's suppose you have an S3 or GCS bucket named `demo-bucket1` where you want to forward your Sumo Logic data. Do the following: 1. [Create a destination](/docs/manage/data-forwarding/forward-data-from-sumologic/#configure-data-forwarding-destination) that points to the `demo-bucket1` bucket. For example, name it **Test destination**. 1. Open the partition or scheduled view whose data you want to [forward data to the new destination](/docs/manage/data-forwarding/forward-data-from-sumologic/#configure-data-forwarding-destination). 1. In the partition or scheduled view, select **Enable Data Forwarding**, and fill out the fields that appear: 1. In **Destination Type** select **Amazon S3** or **Google Cloud Storage** depending on your requirement. 1. In **Forwarding Destination** select any **Existing Destination**. 1. In **Destination** select the name of the destination you created earlier, for example, **Test destination**. 1. Use the **Data Forwarding Configuration** section to specify whether to forward only log data, log data with metadata, or log data with metadata and enriched fields. 1. Click **Save** on the partition or scheduled view. The data will start forwarding to the selected destination bucket specified in the destination. ## Error and alert conditions An error or alert condition can occur with an S3 data forwarding destination for the following reasons: * If Sumo Logic is not able to verify the S3 credentials when the destination is saved, an error message indicates that the credentials were rejected by Amazon. If this occurs, verify **Access Key ID**, **Secret Access Key**, and the bucket configuration, re-select the **Active** check box, and save again.
Bad credentials message * Errors and alerts that are generated after the destination has been successfully saved and started are shown on the **Partitions** page. 
Errors and alerts on the partitions page * Hover over the icon to display the message.
Hover message In this example, Sumo Logic has disabled data forwarding due to errors in connecting to the S3 bucket. This occurs if the Amazon account or credentials change so that Sumo Logic is no longer able to authenticate to the bucket.