--- title: "Tail your logs with tailing sidecar operator" page_name: "Tail your logs with Tailing Sidecar Operator" type: "blog" slug: "tailing-sidecar-operator" published_at: "2021-04-08" modified_at: "2025-05-09" url: "https://www.sumologic.com/blog/tailing-sidecar-operator" canonical: "https://www.sumologic.com/blog/tailing-sidecar-operator" markdown_url: "https://www.sumologic.com/blog/tailing-sidecar-operator.md" lang: "en" excerpt: "When migrating to Kubernetes and re-architecting your applications into containers, logging is a critical piece to consider; announcing the release of the Tailing Sidecar Operator." taxonomy_blog_category: - "DevOps & IT Operations" --- [ All blogs ](https://www.sumologic.com/blog "blog")[DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) # Tail your logs with Tailing Sidecar Operator [Katarzyna Kujawa](#blog-author-block-189) April 8, 2021 4 min read [DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) ##### Table of contents When migrating to Kubernetes and re-architecting your applications into containers, logging is a critical piece to consider. The [twelve-factor app methodology](https://12factor.net/logs) has a section dedicated to logging and outlines the importance of not worrying about routing and storage of your logs. As a best practice, applications running in containers should rely 100% on standard output (STDOUT). Unfortunately, getting logs from applications that do not write to STDOUT is non-trivial and has many things to consider. Logs which are not sent to STDOUT are not accessible through the conventional means we use regularly. You cannot use the kubectl logs command as they are not available in the /var/log/containers directory so they cannot be gathered in the same logging pipeline as other logs coming from containers. Not only have we at Sumo Logic hit this challenge in our migration to Kubernetes, but also have some of our customers. We all wanted to do something about it. That’s why I am excited to announce the release of the [Tailing Sidecar Operator](https://artifacthub.io/packages/helm/tailing-sidecar-operator/tailing-sidecar-operator) to make these challenges much easier. ## Real example when Tailing Sidecar Operator is needed Sometimes different kinds of logs are available in separate files. Let’s see an example with [Slow Query Logs ](https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html)from MySQL database. MySQL database was deployed in Kubernetes cluster with Slow Query Logs enabled. The commands below provide an example: ``` helm repo add bitnami https://charts.bitnami.com/bitnami helm repo update helm install my-release bitnami/mysql -f values.yaml ``` Slow Query Logs are by default available in /bitnami/mysql/data/my-release-mysql-0-slow.log and these logs are accessible only from the container. For a Kubernetes administrator it is not difficult to get them using kubectl exec command in the form like this: ``` $ kubectl exec -it my-release-mysql-0 -- tail -f /bitnami/mysql/data/my-release-mysql-0-slow.log # Time: 2021-03-25T14:13:46.150212Z # User@Host: root[root] @ [10.1.37.76] Id: 1779 # Query_time: 15.000808 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1 SET timestamp=1616699611; SELECT SLEEP(15); # Time: 2021-03-25T14:14:20.697720Z # User@Host: root[root] @ [10.1.37.76] Id: 1779 # Query_time: 15.000605 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1 SET timestamp=1616699645; SELECT SLEEP(15); ``` However it is not the first thing in the mind of a Kubernetes administrator when something goes wrong in the cluster. Mostly kubectl logs command is the first idea, so it would be good to have logs available through this command. The other issue is that Slow Query Logs are not available in /var/log/containers directory on the Kubernetes node the pod is running on. There is only one log file with logs from STDOUT for theMySQL Pod in /var/log/containers so the standard logging pipeline, like [this](https://github.com/SumoLogic/sumologic-kubernetes-collection/blob/66d89c07ea85c68be4ed3e089b56db8413b2375d/deploy/helm/sumologic/values.yaml#L873), does not include Slow Query Logs logs. ## How does the Tailing Sidecar Operator help in getting logs? The Tailing Sidecar Operator adds additional sidecar containers to Pod and redirects logs from files to standard outputs of added containers. It makes logs from different files accessible through kubectl logs command and available in /var/log/container directory. The concept for single tailing sidecar container is illustrated on schema below: [Tailing sidecar container](https://github.com/SumoLogic/tailing-sidecar/tree/main/sidecar) uses [Fluent Bit](https://fluentbit.io/), with [tail plugin](https://docs.fluentbit.io/manual/pipeline/inputs/tail) and [output plugin](https://github.com/SumoLogic/tailing-sidecar/tree/main/sidecar/out_gstdout) that sends records to the standard output, benefiting from its performance and features. ## Tailing Sidecar Operator configuration Configuration for tailing sidecar operator is provided through tailing-sidecar annotation added to Pod metadata: ``` metadata: annotations: tailing-sidecar: ``` Tailing sidecar container needs to have provided volume with the log file and a path to the log file in the container, optionally a name of the sidecar container can be configured. Configuration in annotation should be provided using following syntax: ``` metadata: annotations: tailing-sidecar: ::;:;: ``` Configuration for each tailing sidecar container is separated by “;” but elements in configuration of single tailing sidecar container are separated by “:”. ## Tailing Sidecar Operator installation Installation of Tailing Sidecar Operator is available through Helm Chart so before installation add Tailing Sidecar Operator Helm repository: ``` helm repo add tailing-sidecar https://sumologic.github.io/tailing-sidecar helm repo update ``` And then install operator: ``` helm upgrade --install tailing-sidecar tailing-sidecar/tailing-sidecar-operator -n tailing-sidecar-system --create-namespace ``` ## See the power of Tailing Sidecar Operator Coming back to the example with Slow Query Logs from MySQL database, there is only one thing which needs to be done to see the power of Tailing Sidecar Operator – the annotation must be added to Pod: ``` metadata: annotations: tailing-sidecar: slow-query-logs:data:/bitnami/mysql/data/mysql-release-0-slow.log ``` For MySQL deployment used in example this section needs to be added to values.yaml: ``` primary: podAnnotations: tailing-sidecar: slow-query-logs:data:/bitnami/mysql/data/mysql-release-0-slow.log ``` Upgrade MySQL deployment, e.g.: ``` ROOT_PASSWORD=$(kubectl get secret --namespace default my-release-mysql -o jsonpath="{.data.mysql-root-password}" | base64 --decode) helm upgrade my-release bitnami/mysql --set auth.rootPassword=$ROOT_PASSWORD -f values.yaml ``` And see Slow Query Logs using kubectl logs: ``` $ kubectl logs my-release-mysql-0 slow-query-logs # Time: 2021-03-25T16:38:16.709509Z # User@Host: root[root] @ [10.1.37.82] Id: 94 # Query_time: 15.000483 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1 SET timestamp=1616704681; SELECT SLEEP(15); # Time: 2021-03-25T16:47:38.064183Z # User@Host: root[root] @ [10.1.37.82] Id: 19 # Query_time: 15.000392 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1 use my_database; SET timestamp=1616705243; SELECT SLEEP(15); ``` Thanks to the Tailing Sidecar Operator there is a log file with Slow Query Logs in /var/log/containers so these logs will be gathered by a standard logging pipeline. There is one important fact to mention – containers cannot be added to existing Pod. Tailing Sidecar Operator is able to add containers to Pod leveraging [mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook). In short, the Tailing Sidecar operator receives information about requests for creating a new Pod before it is created. Because of that, the operator can modify Pod specification. It is important to remember that adding or deleting elements in tailing-sidecar annotation corresponds to adding or deleting containers from Pod specification. ## Benefits for Sumo Logic Customers For Sumo Logic customers, the Tailing Sidecar Operator has been incorporated into the [2.1.0 release of our Kubernetes Collection Helm chart](https://github.com/SumoLogic/sumologic-kubernetes-collection) as an optional component. Now all Sumo Logic customers can easily leverage the Tailing Sidecar Operator to get easier logs collection from containers that do not write to STDOUT. ## More about Tailing Sidecar Operator We decided this project has great benefit to the Kubernetes community and can help anyone faced with the challenges of legacy applications that might not be able to write to STDOUT. We wanted to release this project as an open source project from which all could benefit. To get more information about Tailing Sidecar Operator please see Github repository: **Happy Tailing!** ### Article Tags - [DevOps & IT Operations](https://www.sumologic.com/blog/devops-it-operations) Katarzyna Kujawa Senior Open Source Engineer Katarzyna Kujawa is a Senior Open Source Engineer at Sumo Logic working on things which she likes – getting metrics and logs and delivering them in an easy way. She has put her heart into several open source projects related to metrics and logs collection and for sure there will be more. [](https://www.sumologic.com/feed "RSS Feed")[](https://twitter.com/intent/tweet?text=Tail%20your%20logs%20with%20Tailing%20Sidecar%20Operator&url=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Ftailing-sidecar-operator "X")[](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Ftailing-sidecar-operator "Facebook")[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.sumologic.com%2Fblog%2Ftailing-sidecar-operator "Linkedin") [Previous blog Extend AWS observability beyond CloudWatch](https://www.sumologic.com/blog/extend-aws-observability-beyond-cloudwatch)[Next blog Accelerate incident resolution by benchmarks-enriched on-call contexts](https://www.sumologic.com/blog/accelerate-incident-resolution) People who read this also enjoyed [ Sumo Logic AWS Region European Sovereign Cloud is now generally available June 2, 2026 ](https://www.sumologic.com/blog/sumo-logic-aws-region-european-sovereign-cloud-generally-available)[ How to secure cloud workloads without building a full-scale SOC April 30, 2026 ](https://www.sumologic.com/blog/secure-cloud-workloads-with-limited-resources)[ Join operator and Query Agent for smarter log analysis April 22, 2026 ](https://www.sumologic.com/blog/using-the-join-operator)[ 92% of security leaders say their SIEM is effective. 51% say it’s exceptional. What’s living in that gap? April 16, 2026 ](https://www.sumologic.com/blog/from-effective-to-exceptional-siem) [AI Instructions](https://www.sumologic.com/ai-instructions.md)