Enhanced Linux visibility with Sumo Logic

In the continually evolving digital landscape, the importance of effective and efficient logging cannot be
overstated. When we journey into the realm of Linux, this rings particularly true. Today, we’ll delve into why Linux
logging is vital, the challenges customers commonly encounter with it, and how Sumo Logic has emerged as a market
leader in providing unparalleled SIEM solutions.

The need for Linux logging: A historical perspective

Since the early days of computing, logging has played a crucial role in system management and security. In the realm
of Linux, a stalwart in the backbone of the digital world, this is especially important. System logs provide not only
a comprehensive view of what is happening within your Linux environment but also serve as a first line of defense
against potential security threats. They are a record of users’ activities, system errors, and other key events.
Without these logs, we’re navigating through the digital world with a blindfold on, vulnerable to hidden threats.

Linux auditing challenges: The labyrinth unveiled

Despite the necessity of Linux logging, the path to effective auditing is often strewn with obstacles. One of the
major challenges is the sheer volume of data produced. Linux systems generate a staggering amount of log data, which
can be overwhelming to manage and analyze. Further, Linux log files are spread across different locations, each with
its own format, making consolidation and standardization a daunting task.

Also, as Linux is an open-source system, it is subject to continual updates and patches from its developer community.
While this ensures Linux’s robustness and adaptability, it presents an ever-changing landscape that poses additional
challenges for effective auditing.

Illuminating the dark with Sumo Logic

At Sumo Logic, we understand the complexities and challenges inherent in Linux auditing. To navigate through this
labyrinth, we’ve developed a sophisticated, market-leading SIEM solution that not only addresses these challenges but
also provides actionable insights to enhance your system’s security posture.

Linux defensive arts

Let’s take a look at some techniques used by threat actors to attack Linux systems. In order to perform these tests,
we’ll be utilizing the awesome Atomic Red Team Framework.

T1556.003 Modify Authentication Process: Pluggable Authentication Modules

Modification to PAM modules on Linux hosts can lead to credential theft avenues as well as allowing threat actors to
persist on hosts. We can test this activity with the following

Invoke-AtomicTest T1556.003

Using Laurel telemetry, we
can then go ahead and search for this activity:

  _sourceName="/var/log/laurel/audit.log"
  | where %"path[0].name" = "/etc/pam.d/"
  | values(%"proctitle.argv") as arguments by %"syscall.ppid.exe"

And the results:

T1014 Rootkit

Rootkits are very dangerous for defenders as they could potentially blind existing security controls on Linux hosts
and provide threat actors with deep persistence mechanisms.

We can go ahead and run the atomic test for this with the following command: Invoke-AtomicTest T1014

Sumo Logic’s Cloud SIEM is able to normalize and parse the Laurel telemetry and has existing coverage for this
technique:

T1548.003 Abuse elevation control mechanism: Sudo and sudo caching

This test suite attempts to modify the sudo configuration on Linux hosts, something that is definitely worth
monitoring for and alerting on, we can run a test that modifies the sudo cache timeout on a Linux host with the
following command: Invoke-AtomicTest T1548.003 -TestNumbers 2

We can then go ahead and look for this activity:

  _sourceName="/var/log/laurel/audit.log" 
  | %"execve.argv" as args
  | %"path[0].name" as path
  | where args matches /(/etc/sudoers)/
  | where args matches /(timestamp_timeout)/
  | values(%"execve.argv") as arguments by path  

And looking at the results:

T1070.002 – Indicator removal on host: Clear Linux or Mac system logs

Indicator removal on a host and generally clearing/tampering with security telemetry is definitely something that
security teams should be on the lookout for and be constantly testing. If not legitimate activity, this can be a very
high-fidelity signal that something suspicious is occurring in the environment. 

Once again, we can test this using the
following: Invoke-AtomicTest T1070.002 -TestNumbers 13

We can then go ahead and look for this activity: 

 _sourceName="/var/log/laurel/audit.log" 
  | %"execve.argv" as args
  | %"path[0].name" as path
  | if(args matches /(rm /var/log/journal)/,1,0) as journal_removal
  | if(args matches /(journalctl --vacuum-time=0)/,1,0) as journal_clear
  | where journal_removal = "1" or journal_clear = "1"
  | values(%"execve.argv") as arguments by path

This query has two if statements within it, one for the outright removal of the journal log file and the second for
setting the vacuum time via journactl to 0 which, as the Atomic test states, clears the journal file while keeping the
journal field in place. 

And the results:

T1496 – Resource hijacking

For this test, we’ll use the Mythic C2 framework on our test machine in order to establish a command and control connection
between our host and C2 server. We will then download and execute Xmrig on the host:

At this point, we could go ahead and look at Laurel telemetry for process information containing either our C2
payload or the Xmrig process. However, the keyword here is resources. In other words, we need a way to monitor and
alert on the resource utilization of our host in order to fully wrangle this technique. 

Thankfully Sumo Logic’s OpenTelemetry collector for Linux provides defenders with the ability to collect
metrics data from hosts, in addition to log data. 

We can use the Sumo Logic Metrics Explorer to build a metrics query similar to the following:

And looking at the results:

We can see that an outlier has been detected for our Xmrig process – from here we can start an investigation and work
backwards utilizing our host-based telemetry. 

The future of Linux logging

In the realm of cybersecurity, Security Information and Event Management (SIEM) systems are pivotal in detecting and
managing attacks, particularly on Linux-based systems, which are globally renowned for their robustness and ubiquity
in enterprise environments. It is crucial to highlight the evolving sophistication of SIEM solutions in pinpointing
abnormal activities and potential breaches within Linux systems. 

The continuous advancements in SIEM technologies are focusing on the enhancement of real-time analysis of security
alerts generated by hardware and applications, which is pivotal for Linux environments. This integration of advanced
analytical tools and an increased emphasis on real-time responses is pivotal in mitigating the multifaceted and
ever-evolving threats, thereby illustrating a prospective future where Linux logging is more intuitive, adaptive, and,
most importantly, secure. 

The future will likely see the intertwining of AI and machine learning with SIEM solutions to facilitate the
prediction and prevention of attacks on Linux systems, making Linux logging more proactive and intelligent than ever
before. Sumo is embracing this intertwining of AI and machine learning with SIEM by providing what we call, Global Confidence scores. 

At Sumo Logic, we’re committed to guiding you through the labyrinth of Linux logging and auditing. We offer not just
a product, but a partnership, providing the technical expertise and support you need to secure your Linux environment
effectively.

We invite you to discover how Sumo Logic can transform your Linux logging and auditing experience. Get in touch with our technical
team to learn more, or if you’re interested in trying our strategies within your own systems. Let us help you bring
light to the dark, enhancing visibility and security in your Linux environment.

Learn more about threat hunting on Linux endpoints.