How to evaluate SIEM solutions

You may be wondering, how do I know if I need a new security information and event management (SIEM) solution? These are the most common reasons why it may be time to seek a new SIEM solution:

• Recent security incidents and pen tests are a wake-up call. They expose glaring vulnerabilities and gaps you can no longer ignore and highlight critical capabilities you might need in your solution going forward.
• Regulatory requirements and changes aren’t just bureaucratic hurdles—they’re crucial mandates that require robust security measures to avoid crippling penalties.
• Growth initiatives, whether expanding into new markets or adopting cutting-edge technologies, demand a security solution that can scale and adapt.
• Budget cycles offer a prime opportunity to invest wisely in security rather than continuing with outdated or inadequate solutions.
• Industry mergers and acquisitions (M&A) may force you to confront whether your SIEM can withstand the shockwaves of market consolidation and still provide robust protection.

What generation of SIEM do you have?

Before you can evaluate SIEM solutions, you need to understand which generation of SIEM you are currently using:

• First generation: Initially focused on basic log management and simple correlation capabilities, relying on manual log analysis for threat identification.
• Second generation: Improved correlation engines, dashboards, and basic incident response workflows were introduced, enhancing visibility and response times.
• Third generation: Featured real-time monitoring, integration with external threat intelligence feeds, and User Behavior Analytics (UBA) for insider threat detection.
• Fourth generation: Integrated advanced analytics, machine learning, and SOAR capabilities, with better support for cloud environments and hybrid infrastructures.

The latest generation of SIEM

On the horizon, fifth-generation SIEM represents a significant advancement over previous generations. Here are its key features:

• AI-driven insights provide recommendations and predictive threat detection and response
• A unified platform for holistic security management
• DevSecOps integration to embed security across development lifecycles
• Automated remediation to contain and mitigate threats swiftly.
• Large language models (LLMs) that allow a broader set of DevSecOps teams to interface with the solution using natural language and serve up recommended actions, correlate with real-time threat intelligence, facilitate multi-stage detections, and provide custom insights tailored to their organization.

Five steps to evaluate your SIEM

Evaluating your SIEM involves addressing threat detection, investigation, and response (TDIR) challenges. There are five essential questions to ask when evaluating your current SIEM solution.

1. Are you collecting the right logs?

Logs are the most fundamental artifact of digital computing, providing powerful insights about customers’ cloud and corporate infrastructure and applications. Effective log collection is the cornerstone of a robust SIEM solution. When investigating an incident, all critical logs must be online, available, and part of the analysis. 

Typically, the challenge with log collection is aggregating various disparate data sources accurately and efficiently. Scalability and speed are hallmarks of cloud-native SaaS solutions, which you’ll want to prioritize when evaluating SIEM log collection capabilities.

Solutions that provide licensing and consumption to store and analyze all your enterprise-wide logs effectively can cut hours and even days out of investigations that would otherwise require data staging, processing, and preparation before you can drive analytics and insights. 

A robust SIEM solution must seamlessly integrate with various data sources, support real-time data ingestion, and handle multiple data formats, including vendor-agnostic, open-source collection technologies such as OpenTelemetry, to provide a comprehensive and unified security view. 

Comprehensive data collection is essential to ensure that no critical data are missed and your security team has the information to detect and respond to threats promptly and effectively.

Evaluation criteria for log collection:

• Comprehensive source integration: Ensure the SIEM can collect data from all relevant sources, including on-premises, cloud, and hybrid environments. This includes logs, network flows, endpoint data, and data from various SaaS applications.
• Real-time data ingestion: Verify that the SIEM can ingest data in real time to ensure timely detection and respond quickly to emerging threats.
• Support for diverse data types: The SIEM should support a wide range of data types, including logs, events, metrics, and other relevant data formats, to provide a holistic view of your organization’s security posture.
• Log storage and data retention: Check that security log data are stored securely with AES 256 encryption at rest and TLS encryption in transit and retained for up to seven years or under regulatory bodies in your industry. 
• Log economics: Make sure your SIEM includes log economics and flexible pricing of security data to keep critical data flowing to the SIEM while mitigating cost overruns; optimally, your SIEM will include pricing per scan versus ingest-based pricing models.

Must-Have Capability according to Gartner®: Collection of infrastructure details and security-relevant data from a wide range of assets on-premises and/or in cloud infrastructure. Standard Capability according to Gartner®: Allow for collecting event data from disparate event sources, using multiple mechanisms (log stream, API, file processing) for threat detection, use cases, reporting, and incident investigation. Gartner, “Security Information and Event Management Magic Quadrant,” Andrew Davies, Mitchell Schneider, Rustam Malik, Eric Ahlm, 8 May 2024.

2. How is data transformed in your SIEM?

After data collection, the SIEM transforms the data into a usable format for analysis and to inform actionable next steps. Data transformation involves normalization, enrichment, and correlation processes that convert raw data into meaningful insights. In information security, analysts often face the daunting task of making sense of vast amounts of data from diverse sources. With proper transformation, this data can be cohesive and easier to analyze. 

Normalization is paramount for understanding network activities. It transforms disparate data from various sources into a unified format or schema that simplifies analysis. Despite many solutions touting normalization capabilities, their implementations often fail in effectiveness and ease of use. Normalization is the backbone of detection engineering, threat hunting, and security operations, converting raw messages into standardized records for seamless querying and analysis. 

Not all schemas are created equal. The strength of normalization lies in the use of parsers and mappers to handle both structured and unstructured data. Parsers decode and extract crucial information from raw data, converting it into a readable, structured format. Mappers then align this data with a predefined schema, ensuring uniformity across diverse data sources. This process is especially critical when dealing with unstructured data, which can vary wildly in format and content yet contain some of the most critical, custom application insights.

Enrichment enhances normalized data with contextual information, such as threat intelligence feeds and asset data, helping analysts understand its significance and making detecting and responding to threats easier. Enriching valuable context to records makes them more informative and actionable. For example, command-line data enriched with threat intelligence can identify known malicious commands, improving detection accuracy and investigations.

Evaluation criteria for data transformation:

• Effectiveness of normalization: Ensure the SIEM consistently and accurately applies a common schema and mapping field names.
• Parser and mapper quality: Check the accuracy of parsers, efficiency of mappers, and ease of parser updates.
• Performance and scalability: Evaluate processing speed, ability to handle large data volumes, unstructured data, and efficient resource use.
• Integration capabilities: Assess compatibility with various data sources, API and plugin support, and interoperability with existing systems.
• Contextual data integration: The SIEM must robustly integrate threat intelligence feeds and asset data.
• Accuracy and relevance: Check the precision and relevance of the contextual information added during the SIEM data transformation process.
• Ease of use: Check if the user interface is intuitive, the configuration is simple and flexible, the SIEM provides automation and investigation triggers, and offers robust support.
• Impact on detection and response: Ensure the SIEM offers options to improve detection accuracy and investigation efficiency.
• Performance and scalability: The SIEM must efficiently process data volumes and scale elastically with data volume growth.

Must-Have Capability according to Gartner®: Provide SIEM vendor content and facility for client-created content in areas including analytics, data normalization, collection, and enrichment. Standard Capability according to Gartner®: Normalization, enrichment, and risk-score data from third-party systems. Gartner, “Security Information and Event Management Magic Quadrant,” Andrew Davies, Mitchell Schneider, Rustam Malik, Eric Ahlm, 8 May 2024.

3. Does your SIEM offer advanced analytics?

Data analytics powers the detection of sophisticated cyber threats within an SIEM solution. Advanced analytics capabilities, including AI-driven threat detection and insight management, use machine learning to analyze large data volumes and identify patterns that traditional methods might miss. SIEM solutions incorporating machine learning models enhance threat detection by adapting to new threat patterns and improving over time.

User and entity behavior analytics (UEBA) is crucial in modern SIEM solutions, offering deeper insights into user and entity activities. By creating detailed profiles of normal behavior, UEBA can more accurately identify deviations indicating security threats, thus detecting sophisticated threats that traditional monitoring might miss. In particular, entity-centric detection correlation and pattern recognition are key components of effective threat detection. By linking data from various sources, SIEM solutions can detect complex attack patterns that individual data points might not reveal.

SaaS-based SIEM solutions offer significant benefits in maintaining, updating, and creating detection rules, including a customizable rules engine that aligns the SIEM solution with an organization’s unique security posture. Another key aspect of the SaaS delivery model is the ability to instantly incorporate new and updated detection rules curated by expert threat research teams to keep security content current and secure against emerging attacks.

Incorporating these AI-driven features ensures your SIEM solution provides security teams with the necessary tools to protect their organizations. The SaaS model further enhances this capability by keeping the SIEM solution updated with the latest advancements in threat detection and analytics, ensuring continuous protection against emerging threats.

Evaluation criteria for advanced analytics: 

• Anomaly detection: Verify that the SIEM highlights deviations from normal user and system behavior, which is crucial for identifying potential security incidents.
• Behavioral analysis: Assess how well the SIEM monitors user and entity activities to detect unusual behavior, aiding in identifying unknown and insider threats.
• Holistic view: Ensure the UEBA capabilities provide a comprehensive view of activities across users and entities, identifying correlations that single-point monitoring might miss for insider threat detection.
• Enhanced contextual analysis and awareness: Review how the SIEM incorporates various data sources for more accurate threat detection and uses contextual information for risk assessment, alert prioritization, and reducing false positives.
• Entity profiling and behavior baselines: Determine if the SIEM builds comprehensive profiles and establishes baselines for each entity, improving anomaly detection.
• Continuous updates and reduced maintenance: Validate the SIEM is a true SaaS solution to ensure detection rules and models are updated with the latest threat intelligence, reducing maintenance burden.
• AI-infused capabilities: Check that the SIEM has AI interwoven into its capabilities, such as AI-powered log clustering and noise reduction, automatically adjusting for seasonality and reducing alert fatigue.
• Scalability and flexibility: Assess how well the solution scales resources based on demand for optimal performance and cost-efficiency.
• Transparency and ease of customization: Ensure the SIEM allows for easy rule construction and operation visibility with a user-friendly interface for customization.
• Predefined rules and templates: Apprise the provided library of customizable predefined rules and templates for tailored detection strategies.
• Rule testing and simulation: Review the ability to test and simulate rules before deployment to ensure performance and accuracy.

Must-Have Capability according to Gartner®: End-users can self-develop, modify, and maintain threat detection use cases utilizing correlation, analytic, and signature-based methods. Standard Capability according to Gartner®: Advanced analytic capabilities using user and entity behavior analytics (UEBA) and data sciences (i.e., supervised and unsupervised machine learning, deep learning/recurrent neural networks).Threat intelligence platform (TIP) capabilities to manage intelligence and supply contextual information about threats. Gartner, “Security Information and Event Management Magic Quadrant,” Andrew Davies, Mitchell Schneider, Rustam Malik, Eric Ahlm, 8 May 2024.

4. Does your SIEM offer effective investigation?

Identifying potential threats is only the beginning. Effective threat investigation is crucial for understanding security incidents’ full scope and impact. This process involves detailed analysis, identifying the root cause, and efficient alert management to address genuine threats while promptly minimizing false positives. 

Security investigation features in a SIEM solution should support detailed analysis of security incidents, including drilling down into events, viewing detailed timelines, and correlating related events across different data sources. Root cause analysis is essential for identifying the underlying cause of a security incident, understanding its origin and spread, and preventing future occurrences. Effective alert triage and false positive reduction ensure security teams focus on genuine threats.

Evaluation criteria for investigation: 

• Identifying initial compromise: Ensure the SIEM helps trace incidents back to the initial point of compromise, identifying the first affected system or user and how the threat entered the environment.
• Tracking threat propagation: Assess the provided tools to track the lateral movement of the threat within the network, identifying affected systems and understanding the attacker’s methods.
• Determining impact: Verify the SIEM helps quantify the full impact of the incident, including data exfiltration, system downtime, and potential regulatory implications, to inform effective response and remediation.
• Alert prioritization: Confirm the SIEM prioritizes alerts based on severity and potential impact, helping analysts focus on critical threats first.
• Contextual information: Determine if alerts include relevant details about affected assets, involved users, and associated threat intelligence to help analysts quickly assess relevance and urgency.
• Automated triage: Check the automated triage capabilities using a combination of machine learning and predefined rules to reduce analysts’ time spent on initial alert assessments by automatically classifying and prioritizing alerts.
• False positive reduction: Validate that the SIEM minimizes false positives by refining detection rules and incorporating analyst feedback to ensure that alerts are meaningful and actionable, reduce alert fatigue, and improve efficiency.
• Rapid root cause identification: Ensure the SIEM features flexible case management workflows to sift through vast amounts of data and provide complete visibility across on-premises and cloud environments, such as custom searches via an in-built query engine and machine learning to reduce and deduplicate repeat logs, to speed accurate root cause identification.

Must-Have Capability according to Gartner®: Provision of case management and support of incident response activities. Standard Capability according to Gartner®: Orchestration and automation of tasks and workflows to enhance investigations and limit the impact of incidents. Gartner, “Security Information and Event Management Magic Quadrant,” Andrew Davies, Mitchell Schneider, Rustam Malik, Eric Ahlm, 8 May 2024.

5. How does your SIEM facilitate response?

Collaboration is essential for effectively managing and responding to cybersecurity incidents, especially when incidents and breaches can occur at an application level. Kubernetes containers, GitHub repositories, or insecure cloud infrastructure components can all be exploited during an attack, but security teams do not typically monitor or maintain these. Furthermore, with resource-constrained teams across all three DevSecOps teams, leveraging powerful automation is critical for speed and efficiency.

A SIEM solution that facilitates teamwork and communication can significantly enhance a security team’s efficiency and effectiveness through features like customizable dashboards, automated reporting, compliance tracking, and streamlined response processes. Additionally, a SIEM solution’s response capabilities are vital for promptly and effectively managing incidents, including predefined and customizable incident response workflows, automation and orchestration of repetitive tasks, and tools for post-incident reviews to improve the security posture continuously.

Security Orchestration and Automated Response (SOAR), which used to be considered a separate solution, is increasingly becoming table stakes for SIEM functionality.

A SIEM dashboard provides real-time visualizations of security data to efficiently monitor the organization’s security posture. Reports are crucial in communicating findings, progress, and outcomes to stakeholders within and outside the organization. Compliance features in a SIEM ensure adherence to regulatory requirements and industry standards, which is crucial for avoiding penalties and maintaining trust. Robust response capabilities are essential for effectively managing and mitigating security incidents.

Evaluation criteria for response:

• Customizable dashboards: Review users’ ability to create dashboards to display key metrics, trends, and alerts relevant to their role, with real-time updates for prompt responses to threats.
• Single source of truth: Verify your SIEM gathers all critical data in one place, enabling DevSecOps practices and boosting cross-team collaboration. Also, pursue pricing and consumption models that provide unlimited users, enabling all relevant teams to leverage the solution.
• Role-based access controls and views: Ensure permissions can be assigned by role to enable access for authorized users, with information tailored for each team member, enhancing data relevance and usability.
• Automated and custom reporting: Check that automated report generation ensures consistency and saves time, while custom reports cater to specific needs, such as compliance audits or executive summaries featuring visual summaries and detailed data.
• Regulatory compliance: Assess how well the SIEM facilitates data collection, retention, and reporting for regulations such as GDPR, HIPAA, and PCI-DSS, with comprehensive audit trails for transparency.
• Compliance dashboards and reports: Evaluate specialized tools for monitoring and demonstrating regulatory adherence.
• Incident response workflows: Ensure predefined and customizable workflows guide analysts through security incidents.
• Automation and orchestration: Check how well the SIEM integrates with log analytics platforms and external SOAR tools to automate response actions, improve efficiency, and ensure orchestration supports human-in-the-middle workflow designs.
• Collaboration tools: Validate the SIEM, facilitating effective communication and coordination during incident response with shared workspaces and real-time chat.
• Post-incident reviews: Ensure the solution supports response effectiveness analysis, identifying improvement areas, and updating response plans for continuous enhancement.

Must-Have Capability according to Gartner®: Report generations to support business, compliance, and audit needs as needed. Standard Capability according to Gartner®: Storing essential security event data long-term and making it available for searching.Fully featured security orchestration automation response (SOAR) functionality. Gartner, “Security Information and Event Management Magic Quadrant,” Andrew Davies, Mitchell Schneider, Rustam Malik, Eric Ahlm, 8 May 2024.

Disclaimer:

Gartner, Security Information and Event Management Magic Quadrant, Andrew Davies, Mitchell Schneider, Rustam Malik, Eric Ahlm, 8 May 2024.

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally, and MAGIC QUADRANT is a registered trademark of Gartner, Inc. and/or its affiliates and are used herein with permission. All rights reserved