Kubernetes is more popular than ever. As a result, securing Kubernetes clusters is now more important than ever. This is not necessarily an easy task, because a container environment consists of multiple layers, all of which need to be secured.
This article explains best practices for securing a Kubernetes cluster at all layers, including the host operating system, the network and the application itself. While some of these lessons apply to any type of containerized environment, Kubernetes has its own ways of addressing security concerns; as a result, Kubernetes security best practices vary somewhat from the security strategy you would implement in a different type of container environment.
Kubernetes Host Operating System Security
The first step is to start with a minimized host operating system, which has only the services required to run containers. (There is no problem with using a full operating system—That option just has more services which will need to be monitored, configured, and patched.) Examples of minimized systems would be Red Hat Project Atomic, CoreOS Container Linux, Rancher OS, and Ubuntu Core.
Next, you’ll need to have SELinux enabled, which allows better isolation between processes. Most Kubernetes distributions enable SELinux by default. Another layer of security available seccomp, which can restrict the actual system calls that an individual process can make by assigning profiles.
As a side note, if you are using a container service on a public cloud provider, host security is not as important as the other layers of security, as containers are usually a single tenant system. This means that each container is actually running in its own VM so it will always have the latest OS patches and have no process isolation concerns.
Kubernetes Network Security
Microservice-based architectures continue to increase in popularity for the building and deploying applications and services in a container environment. They involve multiple containers interacting within pods and across hosts to provide the full suite of required business functionality.
Most public cloud providers use a single-tenant model for their container runtimes and leverage the access control lists and security groups they have built for their existing compute environments. Using this type of deployment where access can easily be controlled at the point the compute node accesses the network, it is easy to group similar containers under one group, or ACL, and limit access or provide extra security features like Web Application Firewalls and API Gateways.
In a multi-tenant environment where multiple containers can and will run on the same host, a higher level of network segmentation is needed. Multi-tenant environments are more common in private and hybrid cloud deployments where there is more control of the hardware layer and higher density is a goal.
Unlike in a single-tenant model, traffic can not simply be controlled as it enters the individual compute nodes, as there can be traffic within the individual nodes between container processes that also need to be limited. Kubernetes and the industry have both proven and new networking plugins that can handle this type of traffic. These are always based on SDN (Software-Defined Networking), and range from Open vSwitch (OVS) to Project Calico and the up-and-coming Istio.
Securing Kubernetes Container Images and Registries
For the purposes of this article’s focus on Kubernetes, we are going to assume that best practices are being followed on the coding side and only address the building, storing, and running of the container images. For more information on secure coding best practices, both the CMU Software Engineering Institute and OWASP have lists to get you started.
First, let’s address container registries. The best place to retrieve container images to either use as a base for in-house development, or to run as-is, is from known and trusted public registries. While these images hosted may not be perfect, they have enough eyes looking at them that security issues are often found and resolved in a timely manner, especially on larger projects. The largest and most prevalent of the public Docker registries is Docker Hub.
Most organizations rely on private registries as part of their container strategy. Private registries allow RBAC and other enterprise-friendly features (such as on-premises), which allow use in even the most highly secure environments. Most organizations will even store the container images from the public registry in their private registry to ensure they have a copy of what is running in their environment. You never know when their might be an outage on a public registry or a project might be unavailable.
Using a private registry provides the opportunity to have a repository that can be scanned for known vulnerabilities. Ideally images will be scanned before they are stored in this registry and as they are retrieved (new issues may have happened since the last build) to be deployed by Kubernetes. Black Duck Software and SonarSource are among the companies that provide solutions to scan applications and container images.
Kubernetes Logging and Analytics
Kubernetes includes cluster-based logging which can be shipped to a centralized logging facility. These logs become increasingly valuable when combined with the application logs that are consolidated to the same centralized logging facility. Having a single location for log storage and analysis allows better trending and rapid detection of security and other application incidents.
There is no single solution or playbook to follow that will provide the security profile for every organization that is using Kubernetes. Kubernetes and the community supporting it have supplied multiple options and best practices for securing the cluster and the applications it will run. With these building blocks, every organization can implement the security practices that best fit their needs.