Auditing Kubernetes with Kubeaudit - Conducting an Assessment

Introduction

kubeaudit, an open source tool created by the folks at Shopify, can be used to perform a security audit of Kubernetes clusters to find common low hanging fruits that are often exploited by attackers.

In this post we will see how to run the tool with its options and interpret the results.

Conducting a cluster assessment with Kubeaudit

kubeaudit can be run in 3 modes based on the location of the cluster and your access.

1. Manifest mode
2. Local mode
3. Cluster mode

Manifest mode

Kubernetes manifest files are descriptive yaml files that describe a resource or a group of resources within the cluster. Manifest files can be used to create the resources described within them using kubectl apply -f manifest.yaml.

kubeaudit can be used to audit a manifest file before the manifest resources are deployed to the cluster. This is a good way to get an idea of the security of the resource before they are created.

The following is an example manifest file of a pod that pulls the nginx image and is configured to be privileged

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      privileged: true

To try it out, save the yaml to a file and run the following command.

kubeaudit all -f pod.yaml

The output contains weaknesses and configurations that can be hardened. The [error] messages are dangerous findings, while [warning] are good to have improvements.

If no errors with a given minimum severity are found, the following is returned:

All checks completed. 0 high-risk vulnerabilities found

A very cool feature of the Manifest mode is the Autofix command. You can use this to generate a new manifest that is devoid of the security misconfigurations that kubeaudit flagged in the audit. Run the following command to generate a new manifest yaml with the data updated.

kubeaudit autofix -f pod.yml  -o fixed-pod.yaml

The output pod manifest is now the following

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
  annotations:
    container.apparmor.security.beta.kubernetes.io/nginx: runtime/default
spec:
  containers:
    - name: nginx
      image: nginx
      securityContext:
        privileged: false
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        readOnlyRootFilesystem: true
        runAsNonRoot: true
      resources: {}
  automountServiceAccountToken: false
  securityContext:
    seccompProfile:
      type: RuntimeDefault

Cluster mode

In cluster mode, kubeaudit detects whether its running within a container in a cluster and will audit Kubernetes resources in that cluster.

You can use the kubeaudit docker image to run it directly within a cluster as a container using kubectl.

kubectl version     # confirm you are connecting to the correct cluster
kubectl run kubeaudit --image shopify/kubeaudit  --rm -it     # delete container after scanning

Local mode

In this mode, you can run kubeaudit on a remote cluster by pointing it to a kubeconfig or letting it use the current configured cluster context either via KUBECONFIG env variable or ~/.kube/config file.

kubeaudit all --kubeconfig "/path/to/config" -c <CLUSTER-CONTEXT>   # context only if kubeconfig contains more than one context.

Audit results

kubeaudit produces results with three levels of severity:

• Error: A security issue or invalid kubernetes configuration
• Warning: A best practice recommendation
• Info: Informational, no action required. This includes results that are overridden

The minimum severity level can be set using the --minSeverity/-m flag.

By default kubeaudit will output results in a human-readable way. If the output is intended to be further processed, it can be set to output JSON using the --format json flag.