Viewing logs with Kubernetes native tools is completely centered around the kubectl command line utility. The single-most useful piece of documentation around kubectl is the cheat sheet that is part of the official documentation, as it tracks all the options and parameters that are available through the command.
First up is the most basic commands that will get used to view the logs from a known container. You can view individual log streams (stdout or stderr, for example), but most people just view all the logs for more context.
$ kubectl logs apache-httpd-pod
10.2.1.1 - - [15/Aug/2017:21:30:32 +0000] "GET / HTTP/1.1" 200 576 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36" "127.0.0.1"
If you wish to follow the live stream of log entries (i.e., tail -f in Linux/UNIX) then add the -f flag to the above command before the pod name, and it will provide the same functionality. It would look like:
$ kubectl logs -f apache-httpd-pod
In the event that you only want to view logs that have been generated within a certain time period, then you can use the --since flag to provide that functionality. This command will pull back all log entries created in the last hour.
$ kubectl logs --since=1h apache-httpd-pod
If the pod has multiple containers, and the logs you need are from just one of the containers, then the logs command allows for further refinement by appending -c container_name to the end of the command.
$ kubectl logs apache-httpd-pod -c httpd-server
Kubernetes has the ability to group pods into namespaces for segmentation and easier applications of things like role-based access control. Because of the sheer number of namespaces and pods that can often be in a Kubernetes cluster, it is a common occurance to need to reference a pod or resource that is defined in another namespace. To access other namespaces without changing your default, you can add -n namespace_name to the beginning of a kubectl command to context switch.
$ kubectl -n f5-namespace logs nginx-pod
There are multiple other options available within logs that can be useful, including displaying logs from pods that are part of a specific deployment.
$ kubectl logs deployment/random-deployment
Beyond the logs, which have some traces in them, if you want to get more metrics to see the holistic view of the cluster and get closer to the idea of the Three Pillars of Observability, you can use additional commands like kubectl get pods to list running pods and kubectl top to see how many resources are being used by individual pods or nodes in the cluster.
Example of Creating and Collecting Application Logs in Kubernetes
To show what logging looks like in Kubernetes, we first need to create a pod that will generate logs. For this purpose we will create a simple pod using busybox that will run a continuous loop, and output the current date and time every second.
$ cat <<EOF | kubectl apply -f -
- name: count
args: [/bin/sh, -c,
'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
Will create one pod
The output from these logs will look like
$ kubectl logs counter
0: Mon Jan 1 00:00:00 UTC 2001
1: Mon Jan 1 00:00:01 UTC 2001
2: Mon Jan 1 00:00:02 UTC 2001
Next step is to configure node-level logging so we can see and ship all the logs on each node to a central server. To do this with Fluentd requires it to have a namespace, a secret with several variables set for things like log target and api keys, and finally the actual deployment of Fluentd using something like a DaemonSet so it runs on every node. Explicit details on the installation are maintained by SumoLogic on GitHub.
In the event that logs are produced outside of stdout and stderr, the pod will need to mount a local volume on the node so the logs are available outside of the running containers and then the logging-agent – in this case Fluentd – can be configured to pick up those log files. This is done by adding a new source section to the fluentd.conf file, and restarting the service. Specific details on how this would work are located in Fluentd’s documentation. This below example would pick up an Apache access_log.