})
The Sumo Logic app for MongoDB Atlas allows you to monitor database operations, performance KPIs and provides visibility into the security posture of your clusters with the following dashboard types:
* **Operations** Dashboard. For monitoring database operations and cluster health.
* **Performance** Dashboard. For insights into slow queries, database and hardware metrics.
* **Security** Dashboard. For visibility into user logins, audit events, project and organizational activity, incoming threats, and IOCs.
[MongoDB Atlas](https://www.mongodb.com/cloud/atlas) is a global cloud database service designed specifically for cloud-based applications. MongoDB Atlas runs in AWS, Azure, or GCP environments, with guaranteed availability, scalability, and compliance with data security and privacy standards.
The MongoDB Atlas app supports MongoDB Version 3.4 and above.
## Log and metric types
MongoDB Atlas uses logs and metrics. MongoDB Atlas logs are in JSON format and metrics are in Carbon 2.0 format. The Sumo Logic app for MongoDB Atlas uses five types of logs and two types of metrics.
### Sample log messages
MongoDB Atlas logs are all in JSON format and metrics are in Carbon2.0 format. The MongoDB Atlas utilizes five types of logs and two types of metrics. This section provides examples of the log types utilized by the MongoDB Atlas app.
The Sumo Logic `mongodbatlas` collector enhances logs by adding a few metadata fields. For this reason, the raw logs from the MongoDB Atlas API may differ in format.
Some logs are not available for some cluster tier. Check the [MongoDB docs](https://docs.atlas.mongodb.com/reference/free-shared-limitations/). Some log types like for mongos instances are available for shared cluster only.
})
1. Specify the API key **Organization Member** permissions, under **Organization > Access Manager > API Keys**, as shown in the following example.
1. Go to **Project Settings** > **Access Manager** > **API Keys** and then click on **Invite To Project** to add the API key created above for this project as shown in the following example.})
1. Specify **Project Data Access Read Only** permission, under **Project Settings > Access Manager > API Keys**, as shown in the following example.
1. Go to your project, click **Settings**, and copy the **Project ID**, as shown in the following example.
1. Go to your organization by using context dropdown at the top, click **Settings**, and copy the **Organization ID**.
1. Enable Database Auditing for the Atlas project for which you want to monitor logs, as described in [this Atlas document](https://docs.atlas.mongodb.com/database-auditing/#procedure). Leave **Database Auditing** set to **ON**, as shown in the following example.
### Step 2: Add a Hosted Collector and HTTP Source
This section demonstrates how to add a hosted Sumo Logic collector and HTTP Logs and Metrics source, to collect logs for MongoDB Atlas. When you configure the HTTP Source, make sure to save the HTTP Source Address URL. You will need this later to configure the configuration file.
1. Do one of the following:
* If you already have a Sumo Logic Hosted Collector, identify the one you want to use; or
* Create a new Hosted Collector as described in [Configure a Hosted Collector](/docs/send-data/hosted-collectors/configure-hosted-collector).
1. Add two HTTP sources, one for logs and another for metrics.
1. Go to the source you created for ingesting logs, navigate to **Timestamp Format > Advanced Options**, and click **Specify a format**.
4. Enter the following information in the respective fields for the **log source**:
* Timestamp Locator: `\"created\":(.*)`
* Format: `yyyy-MM-dd'T'HH:mm:ss.SSS'Z'`
1. Click **Add**.
1. Enter the following information in the respective fields for the **metric source**:
* Timestamp Locator: `\"created\":(.*)`
* Format: `yyyy-MM-dd'T'HH:mm:ss'Z'`
1. Click **Add**.
### Step 3: Configure collection for MongoDB Atlas
In this section, we explore various mechanisms to collect database logs, events, metrics and alerts from MongoDB Atlas and send them to Sumo Logic, where they are shown in dashboards as part of the MongoDB Atlas app. You can configure Sumo Logic’s MongoDB Atlas collector in Amazon Web Services (AWS) using the AWS Lambda service, or by running a script on a Linux machine as a cron job. Choose the method that is best suited for you:
* [AWS Lambda-based collection](#deploy-the-sumo-logic-mongodb-atlas-sam-application) via a Serverless Application Model (SAM) application
* [Script-based collection](#configure-script-based-collection-for-mongodb-atlas) from a Linux machine
A single instance of the collector is responsible for collecting logs from a single project. If you have multiple projects, refer to the additional steps in these sections.
#### Deploy the Sumo Logic MongoDB Atlas SAM application
In this section, you deploy the SAM application, which creates the necessary resources in your AWS account. To deploy the Sumo Logic MongoDB Atlas SAM application, do the following:
1. Go to [https://serverlessrepo.aws.amazon.com/applications](https://serverlessrepo.aws.amazon.com/applications).
1. Search for **sumologic-mongodb-atlas**, select the **Show apps that create custom IAM roles or resource policies** check box, and click the app link when it appears.
1. When the Sumo Logic app page appears, click **Deploy**.
1. In the **AWS Lambda** > **Functions** > **Application Settings** panel, specify the following parameters in the corresponding text fields:
* **HTTPLogsEndpoint**: Copy and paste the URL for the HTTP Logs source from this [Step 2](#step-2-add-a-hosted-collector-and-http-source).
* **HTTPMetricsEndpoint**: Copy and paste the URL for the HTTP Metrics source from [Step 2](#step-2-add-a-hosted-collector-and-http-source).
* **OrganizationID**: Copy and paste the Organization ID from [Step 1](#step-1-acquire-authentication-info-from-mongodb-atlas-portal).
* **ProjectID**: Copy and paste the Project ID from [Step 1](#step-1-acquire-authentication-info-from-mongodb-atlas-portal).
* **Private API Key**: Copy and paste the Private Key from [Step 1](#step-1-acquire-authentication-info-from-mongodb-atlas-portal).
* **Public API Key**: Copy and paste the Public Key from [Step 1](#step-1-acquire-authentication-info-from-mongodb-atlas-portal).
1. Click **Deploy**.
1. Search for Lambda in the AWS console, select Functions tab and open the function just created.
1. Go to the **Configuration** > **Permissions** tab of the function, and click on the Execution role name link to open up the IAM window containing all the permission policies.
1. Click on **Add permissions** > **Create inline policy**. Choose JSON and copy this policy statement:
```json
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Action":[
"ec2:DescribeNetworkInterfaces",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:AttachNetworkInterface"
],
"Resource":"*"
}
]
}
```
1. Click on **Review policy**, and provide an appropriate name. Then click on Create policy. Some users might already have these permissions enabled.
1. We then [follow these steps](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/generate-a-static-outbound-ip-address-using-a-lambda-function-amazon-vpc-and-a-serverless-architecture.html) to create elastic IP/IPs for the lambda function and add a VPC to our function. We note down the elastic IPs.
1. We go to the mongo console, click on **Organization Access** > **Access Manager** > **API Keys**, and click on ‘...’ of the API Key as mentioned in step 2. Then click on Edit Permissions.
1. Click **Next** > **Add Access List Entry**. Enter the elastic IPs noted above and, then click Save to save the elastic IPs, and click on Done to apply the settings.
The lambda function should be working now in sending logs to Sumo. You can check the CloudWatch logs in **Monitor** > **Logs** to see the logs of the function.
##### Configure collection for multiple projects
If you are already collecting Atlas data for one project, perform the following steps to configure for additional projects:
1. [Deploy the MongoDB Atlas SAM application](#deploy-the-sumo-logic-mongodb-atlas-sam-application) with the configuration for a new project.
1. From the Lambda console, go to the **mongodbatlas.yaml** file and comment out `EVENTS_ORG`, as shown in the following example. This prevents the collection of `Organisation Events` in the second SAM app deployment, because these events are global and are already captured by first collector.
1. After editing the file, choose **Deploy**. The next Lambda invocation will use the new configuration file.
##### Filtering log types and metrics
By default the solution collects all log types and metrics for all the clusters. If you want to filter based on cluster alias and log types, do the following:
1. After the deployment is complete, go to the Lambda console, and open the **mongodbatlas.yaml** file and uncomment the `Clusters` parameter under the `Collection` section, as shown in the following example. Add your cluster names for which you want to collect logs and metrics. The cluster name should be the same as what you have specified during [cluster creation](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/#specify-a-name-for-the-cluster-in-the-name-box).})
1. By default the solution collects logs types and metrics which are used in the app. If you want to collect specific log types and metric types uncomment to collect the respective log type or metric name as shown below.})
1. After saving the changes in your file, the next invocation (as per cron job schedule) will use the new configuration file.
### Step 4: Configure Webhooks for Alerts Collection
You configure Webhooks for real-time alerts. This section explains how to configure alert collection using Webhooks.
1. Go to the **MongoDBAtlas** console and select **Project Integrations.** Click **Configure** under **Webhook Settings**.
1. Copy and paste the Logs endpoint from [Step 2](#step-2-add-a-hosted-collector-and-http-source) to set up Webhook.
1. When configuring an alert, specify the **Webhook** as shown in the following example, and then click **Save**.
### Advanced Configuration
This section is common for both [AWS Lambda-based collection](#deploy-the-sumo-logic-mongodb-atlas-sam-application) and [script-based collection](#configure-script-based-collection-for-mongodb-atlas).
This table lists MongoDB Atlas variables that you can optionally define in the configuration file (click to expand).
| Variable | Usage |
| LOG_TYPES in MongoDBAtlas Section |
DATABASE
AUDIT EVENTS_PROJECT EVENTS_ORG ALERTS Remove or comment any one of the lines if you do not want to collect metric of that type. |
| PROCESS_METRICS in MongoDBAtlas Section |
CACHE_DIRTY_BYTES CACHE_USED_BYTES CONNECTIONS CURSORS_TOTAL_OPEN CURSORS_TOTAL_TIMED_OUT DB_STORAGE_TOTAL DB_DATA_SIZE_TOTAL DATABASE_AVERAGE_OBJECT_SIZE EXTRA_INFO_PAGE_FAULTS GLOBAL_LOCK_CURRENT_QUEUE_TOTAL MEMORY_RESIDENT MEMORY_VIRTUAL MEMORY_MAPPED NETWORK_BYTES_IN NETWORK_BYTES_OUT NETWORK_NUM_REQUESTS OPCOUNTER_CMD OPCOUNTER_QUERY OPCOUNTER_UPDATE OPCOUNTER_DELETE OPCOUNTER_GETMORE OPCOUNTER_INSERT OP_EXECUTION_TIME_READS OP_EXECUTION_TIME_WRITES OP_EXECUTION_TIME_COMMANDS QUERY_EXECUTOR_SCANNED QUERY_EXECUTOR_SCANNED_OBJECTS QUERY_TARGETING_SCANNED_PER_RETURNED QUERY_TARGETING_SCANNED_OBJECTS_PER_RETURNED SYSTEM_NORMALIZED_CPU_USER SYSTEM_NORMALIZED_CPU_KERNEL SYSTEM_NORMALIZED_CPU_IOWAIT PROCESS_CPU_USER PROCESS_CPU_KERNEL SYSTEM_NORMALIZED_CPU_STEAL Remove or comment any one of the lines if you do not want to collect metric of that type. |
| DISK_METRICS in MongoDBAtlas Section |
DISK_PARTITION_IOPS_READ DISK_PARTITION_IOPS_WRITE DISK_PARTITION_UTILIZATION DISK_PARTITION_LATENCY_READ DISK_PARTITION_LATENCY_WRITE DISK_PARTITION_SPACE_FREE DISK_PARTITION_SPACE_USED Remove or comment any one of the lines if you do not want to collect metric of that type. |
| PAGINATION_LIMIT in Collection Section |
Number of events to fetch in a single API call. |
| LOG_FORMAT in Logging Section |
Log format used by the python logging module to write logs in a file. |
| ENABLE_LOGFILE in Logging Section |
Set to TRUE to write all logs and errors to a log file. |
| ENABLE_CONSOLE_LOG in Logging Section |
Enables printing logs in a console. |
| LOG_FILEPATH in Logging Section |
Path of the log file used when ENABLE_LOGFILE is set to TRUE. |
| NUM_WORKERS in Collection Section |
Number of threads to spawn for API calls. |
| MAX_RETRY in Collection Section |
Number of retries to attempt in case of request failure. |
| BACKOFF_FACTOR in Collection Section |
A backoff factor to apply between attempts after the second try. If the backoff_factor is 0.1, then sleep() will sleep for [0.0s, 0.2s, 0.4s, ...] between retries. |
| BACKFILL_DAYS in Collection Section |
Number of days before the event collection will start. If the value is 1, then events are fetched from yesterday to today. |
| TIMEOUT in Collection Section |
Request time out used by the requests library. |
| MIN_REQUEST_WINDOW_LENGTH in Collection Section |
Minimum window length for the request window. The default is 60 seconds. |
| MAX_REQUEST_WINDOW_LENGTH in Collection Section |
Maximum window length for the request window. The default is 900 seconds. |
| MAX_PAYLOAD_BYTESIZE in Collection Section |
Maximum size of the chunk to be sent to sumo logic. Default is 4MB. |
| LOGS_SUMO_ENDPOINT in MongoDBAtlas section |
HTTP source endpoint url created in Sumo Logic for ingesting Logs. |
| METRICS_SUMO_ENDPOINT In MongoDBAtlas section |
HTTP source endpoint url created in Sumo Logic for ingesting Metrics. |
You can also avoid this error by running the script with config file path as first argument. ```bash Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/sumomongodbatlascollector/main.py", line 190, in main ns = MongoDBAtlasCollector() File "/usr/local/lib/python2.7/dist-packages/sumomongodbatlascollector/main.py", line 29, in __init__ self.config = Config().get_config(self.CONFIG_FILENAME, self.root_dir, cfgpath) File "/usr/local/lib/python2.7/dist-packages/sumomongodbatlascollector/common/config.py", line 22, in get_config self.validate_config(self.config) File "/usr/local/lib/python2.7/dist-packages/sumomongodbatlascollector/common/config.py", line 34, in validate_config raise Exception("Invalid config") Exception: Invalid config ``` 1. Look for common error codes: * **TENANT_CLUSTER_LOGS_FOR_HOST_NOT_SUPPORTED**. This means that logs are not supported for that tier. * **IP_ADDRESS_NOT_ON_ACCESS_LIST**. This means that you may also have to configure [ip access list](https://www.mongodb.com/docs/atlas/configure-api-access-org/#edit-the-api-access-list) for the host from which you want Atlas to accept API requests. * **METRIC_TYPE_UNSUPPORTED**. This means that some metric type has been deprecated, refer the logs and remove that particular metric type from mongodbatlas.yaml file. ## Installing the MongoDB Atlas app import AppInstall from '../../reuse/apps/app-install-v2.md';
})
### Security dashboards
The MongoDB Atlas Security dashboards aim to provide a comprehensive view of Atlas security and audit aspects via the Atlas audit logs, alerts, and events logs.
#### Events
The **MongoDB Atlas Events** dashboard provides information on Atlas project and organization-level changes. Panels show information on the type, number, location and recent events.
Use this dashboard to:
* Monitor most recent and past Atlas change events to ensure that the number and type of events are in line with expectations.
* Identify any violations in your security policies (such as users accessing Atlas without MFA).
})
#### Alerts
The **MongoDB Atlas Alerts** dashboard provides an at-a-glance view view of alerts triggered in your Atlas environment. Panels show information around the total number and number of open alerts, recent alerts, alert types and status.
Use this dashboard to:
* Identify and address all open and recent alerts.
* Monitor unusually high number of alerts by analyzing trend graphs.
})
#### Audit
The **MongoDB Atlas Audit** dashboard provides information around security events in your Atlas environment such as failed authentication, authorization and audit events, audit event trends, and originating geographic locations. Panels also display details on audit events by action type and user, and recent audit events by created and deleted resources.
Use this dashboard to:
* Identify unusually high number of eventsfailed eventby analyzing trend graphs.
* Identify most affected hosts and users associated with failed audit events.
* Monitor most recent audit events and database operations to ensure they are in line with expectations.
* Track database read and write operations, spikes in failed events, as well as the users who performed the events.
})
### Performance dashboards
Performance dashboards provide visual representations of key MongoDB Atlas metrics. You can use this data to correlate system health and performance and use Atlas logs to identify and fix the root cause of any problems. These dashboards are also provide the ability to fine tune your queries and database operations.
#### Cluster Metrics
The **MongoDB Atlas - Cluster Metrics** dashboard provides a high-level view of cluster performance, disk and cache usage. Panel graphs show details for read and write operations and their execution times, network traffic, connections, data sizes, and disk read and write IOPs.
Use this dashboard to:
* Identify anomalous changes in database metrics.
* Monitor resource usage and determine how to optimize your Atlas databases and clusters.
})
#### Metrics
The **MongoDB Atlas Metrics** dashboard provides an at-a-glance view of database performance with graphs showing memory availability, data size, cache in bytes, and returned document ratios.
Use this dashboard to:
* Determine node health based on page faults, cache dirty bytes, replication headroom, queued operations, and disk write latency.
* Monitor resource usage (cache and disk) and active connections. You can set up alerts for notification on these metrics.
* Drill-down into the underlying queries and correlate performance metrics with underlying logs to identify the root cause of performance degradations.
})
#### Slow Queries
The **MongoDB Atlas Slow Queries** dashboard provides details on the number of slow queries by type,trends, and slow server status.
Use this dashboard to:
* Identify and fix slow queries.
* Identify changes in the number and types of slow queries by analyzing trends.
* Identify databases, connections collections experiencing slow queries.
* Determine queries and operations that are using Scanned Objects/Returned objects and Keys Scanned Keys/Returned objects ratios to identify the potential fields for indexing.
})
### Operations dashboards
The Operations dashboards monitor database operations, such as indexing, sharding, and replication. These dashboards allow you to view detailed error logs for troubleshooting and track login activities in your database including failed attempts.
#### Errors and Warnings
The **MongoDB Atlas Errors and Warnings** dashboard provides information on errors, warnings by component, severity and type.. Panels also show information on daily error and warning summaries, socket exceptions, timeout events, and MongoDB exit events.
Use this dashboard to:
* Identify and fix errors and warnings.
* Troubleshoot problems in your cluster by analyzing errors and warnings.
* Identify hosts with most errors and correct issues accordingly.
})
#### Logins and Connections
The **MongoDB Atlas Logins and Connections** dashboard provides a high-level view of the geographic locations of client connection requests, logins from malicious remote IPS, and geographic locations of failed logins. Panels also display information on overall failed login attempts, threat intel by remote IP, and failed logins by user.
Use this dashboard to:
* Identify requests coming in malicious remote IPs and their geographic locations and use this information to fix your firewall or WAF devices.
* Validate locations of incoming client connections are in line with expected locations.
})
#### Replication
The **MongoDB Atlas Replication** dashboard provides information on changes in replication configurations. Panels display details on replication errors, warnings, events, as well as information around primary and secondary nodes.
Use this dashboard to:
* Identify and fix replication and availability errors.
* Monitor changes in replication lag performance.
})
#### Sharding
The **MongoDB Atlas Sharding** dashboard provides information on Atlas cluster sharding. Panels show details on warnings, errors, chunk split failures, balancer failures, chunks moving by collection, and chunks moving between shards.
Use this dashboard to:
* Identify and fix sharding and balancing related errors and warnings.
* Track chunk move operations to ensure they are in line with expectations.
})
## Upgrade/Downgrade the MongoDB Atlas app (Optional)
import AppUpdate from '../../reuse/apps/app-update.md';