New Kubernetes Best Practices & Security Checks

September 30, 2020

Despite this new evolutionary step in the virtualization industry providing significant benefits, naturally Kubernetes out-of-the-box is insecure and requires careful tuning to secure the cluster resources and workloads. Therefore, it was naturally our first thought to add automated checks for Kubernetes best practices and CIS security compliance standards in Runecast Analyzer.

No system admin was born with the Kubernetes experience, so we’re happy that we can help to address these challenges in an automated way and assist in hardening the Kubernetes cluster so they can be used in production.

What is Kubernetes (aka K8s)?

The simple definition, as defined by the Kubernetes.io website:

“Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.

The name Kubernetes originates from Greek, meaning helmsman or pilot. Google open-sourced the Kubernetes project in 2014. Kubernetes combines over 15 years of Google's experience running production workloads at scale with best-of-breed ideas and practices from the community.”

Google has a long history of running containers. Have you ever been in your GMail and tried to open a message, but you need to refresh in order to do so? That’s a new container spinning up to handle your workload. Google built a cluster management system called Borg way back in 2003/2004, which they used to eke the maximum utilization out of the servers that they were running. This engine eventually ran the Google Cloud Platform, and then as Google moved on to launch their IaaS offering Google Compute Engine, they noticed that customers were running lots of VMs, with very low utilization rates.

What if they could take what they learned with Borg, and were able to offer customers a similar service? Docker already existed at this point, but it had no coherent management system. From this idea, Project 7 was born. When you find out that this was a reference to Seven of Nine, you realise that there are some hardcore Star Trek fans working at Google. They took all of their efforts and their learnings from the development of Borg and it’s successor Omega, gave it a simple to use UI, and they had a prototype that they were ready to share. This prototype was open-sourced as Kubernetes. If you check the Kubernetes logo you’ll see that there are seven sides to the wheel, a reference to its origins in Project 7. (Source: https://cloud.google.com/blog/products/gcp/from-google-to-the-world-the-kubernetes-origin-story)

Why is Kubernetes so popular?

While there have been numerous projects that aimed to orchestrate containers (Docker Swarm and Apache Mesos were both early frontrunners), the industry has long since standardised on Kubernetes. It's the leading open-source orchestrator for executing and scaling container-based workloads.

It’s perfect for scalable cloud-native application environments, and managing container-based workloads wherever they may run.

Again, Kubernetes explains this rather well:

Container deployment became popular for it’s agility and clear resources isolation and utilization. Containers are similar to Virtual Machines (VMs), but they have relaxed isolation properties in order to share the OS among the applications. Thus, containers are considered to be lightweight. Similar to VMs, a container has its own filesystem, CPU, memory, process space, for example. And as they are decoupled from the underlying infrastructure, they are portable across clouds.

Some additional benefits that containers provide:

Agile application creation and deployment
Dev and Ops separation of concerns
Environmental consistency across development, testing, and production: Runs the same on a laptop as it does in the cloud
Cloud and OS distribution portability
Resource isolation: predictable application performance
Resource utilization: high efficiency and density
Allows you to use Kubernetes RBAC with the least privilege concept
And much more...

Kubernetes operational & security Best Practices

It's rather simple for hackers to identify and make use of Kubernetes clusters by watching specific ports and encountering an insecure Kubernetes API service.

Any vulnerabilities in the Kubelet API (used by Kubernetes), or the kube-apiserver could even allow hackers to execute code in your containers, which can result in a compromised cluster. (For example, see https://github.com/kayrus/kubelet-exploit, or https://nvd.nist.gov/vuln/detail/CVE-2019-11253)

According to online sources, these are some of the most common Kubernetes recommendations that IT admins should follow:

Use minimal base images
Don’t add unnecessary components
Use up-to-date images only
Use an image scanner to identify known vulnerabilities
Integrate security into your CI/CD pipeline
Label non-fixable vulnerabilities
Implement defense-in-depth
Use namespaces to isolate sensitive workloads
Use Kubernetes network policies to control traffic between pods and clusters
Prevent overly permissive access to secrets
Disallow:

Root user
Privileged containers
Adding new capabilities
Changes to kernel parameters
Use of bind mounts (hostPath volumes)
Access to the docker socket bind mount
Use of host network and ports

12. Require:

Read-only root filesystem
Pod resource requests and limits
LivenessProbe and readinessProbe

And the list keeps growing, becoming ever more complex – and thus challenging (if not impossible) for IT admins to keep up with proactively.

Even with experts at hand, and after successfully hardening a Kubernetes environment for production use, the bigger challenge is in maintaining a secure environment over time. This is due to the following reasons:

The amount of security best practices to be followed is considerable already and it is constantly growing. Therefore a manual evaluation becomes quickly uneconomical, even for smaller environments.
The applicable security best-practices change quickly due to an ever-changing landscape of vulnerabilities for the complete system, because of:

changes to the configuration of the Kubernetes cluster
changes in Kubernetes itself (fixing and introducing bugs and security holes)
discovery of new vulnerabilities of different aspects of the environment

Kubernetes security model: “the 4C's of cloud native security”

Kubernetes’ 4C’s model of Cloud Native security states that ‘’each layer of the Cloud Native security model builds upon the next outermost layer. The Code layer benefits from strong base (Cloud, Cluster, Container) security layers. You cannot safeguard against poor security standards in the base layers by addressing security at the Code level.’’

*Source: https://kubernetes.io/docs/concepts/security/overview/*

Runecast Analyzer assists with the security and best practices checks on the Cloud and Cluster level (both components and application), to keep your environment safe and audit-ready all the time.

How to conquer Kubernetes challenges

The different aspects of security on the node-, cluster- and workload level in a containerized environment rule out conventional security metrics.

Other security pain points include:

Teams distributed across multiple sites and regions
Diverse infrastructures, with no one-size-fits-all approach
Containers are just one aspect of infrastructure operations
Enterprise-specific security requirements (in addition to industry compliance standards)

Compliance to various security standards is still crucial for most companies, especially when transitioning to containerized environments.

Auditors will be even more diligent and careful when dealing with containerized environments and therefore using trusted tools becomes even more of a must-have when dealing with audit preparations.
Also consider that as an (extremely) actively developed, open source software project that things are in a constant state of flux. What was a best practice today may well be a terrible idea next month, so being able to see the state of your environment over time, and when you drift away from those best practices is vital.

Transitioning a Kubernetes environment from a development to a production state is hardly possible without the help of well-experienced Kubernetes experts. And let’s face it, nobody has “12+ years’ experience in Kubernetes” at this point.

*Source: https://www.reddit.com/r/mildlyinfuriating/comments/hm4u15/requirement_for_a_job/*

‍

Runecast Analyzer offers automated Kubernetes configuration analysis at the node-level, cluster-level, and workload level by covering common cluster operational and security best practices for Kubernetes, as well as the CIS benchmark for Kubernetes (security standard).

‍

Team Runecast

Therefore your team can benefit from automated checks against best practices and security standards compliance, without reactive and time-consuming manual efforts (which aren’t necessarily even effective).

Runecast is bound to continuously increase the number of compliance standards for Kubernetes as containerizing parts of the infrastructure will become one of the must-haves to drive cost savings and responsiveness in coming years.

In the Runecast Analyzer 4.5 release, we’ve also released new Custom Profiles – find more about this feature here.

Let Runecast Analyzer demystify your Kubernetes experience by helping you with operational transparency. You can explore new features in our Online demo, or test it in your environment within the 14-day free trial. For now, licensing for Kubernetes insights works the same as standard Runecast Analyzer Licensing & Pricing (subject to change at a future date).