Note: This talk, presented at DeveloperWeek Europe 2021, is identical to the one presented at the KubeSec Enterprise Online conference. The talk at DeveloperWeek Europe, however had some interesting questions come in through the chat whose answers have been expanded and added here.
The idea for the talk came via a discussion we had internally at Kloudle when brainstorming on the different approaches that an attacker would use in an attempt to gain access to a Kubernetes cluster and what they would do when they have "Attacker in a Pod" level access.
Some of the questions we posed to each other were
What does an organization think it exposes to the Internet and what it actually exposes
What tactics and techniques do attackers use to gain access to a Kubernetes cluster
What can a cluster administrator do to prevent attackers from reaching their cluster or slow them down if they do gain access
We imagined an attacker employing different tactics and techniques when attacking a Kubernetes cluster and created this talk with real world examples to support our story.
Here are all the slides from the DeveloperWeek Europe 2021 talk presented virtually on 27th April 2021.
The Threat Matrix describes Tactics, Techniques and Procedures (TTPs) for technologies like Windows, Linux, AWS, GCP, Azure, Android, iOS etc. and can be used by both offensive security folks to plan test cases and defenders to identify attack paths that the bad guys will take.
The talk was heavily based on the Kubernetes MITRE ATT&CK like Threat Matrix that Microsoft has created.
For each of the Tactic that the Threat Matrix covers, we took examples from real world assessments, breaches, proof of concept setups and real world configurations and mapped them to the framework.
The presentation contains additional information and real world data that was not covered in the blog posts.
Initial Access: Exposed Dashboard
The Exposed Dashboard is a technique covered under Initial Access that covers the exposure of the Kubernetes dashboard (not installed by default) to the attacker. If accessible to the attacker, this allows for potential cluster management and configuration updates.
The screenshot shown in the slide is of a Kubernetes Dashboard belonging to Tesla which was discovered by a security company. The discovery led to another startling discovery where the Kubernetes cluster was found to be running crypto-mining workloads as the attackers had already made their way to the Impact Tactic of the Threat Matrix.
Initial Access: Application Vulnerability
This slide is another example of a technique that attackers often use, especially if the cluster exposes workloads to the attacker via services. An application running within the cluster and exposed to the attacker, if vulnerable, could lead to the attacker gaining file read/write capabilities, ability to make server-side network requests or the ability to execute arbitrary commands.
There are multiple other techniques covered in the Threat Matrix, some of which are explained below
Execution: New container
The next Tactic covers techniques that describe what attackers do to execute code. The example that we covered was the launch of a new container. In this case, the container was a privileged container that allows access to the underlying nodes network and process namespaces.
The Pod yaml shown in the slide is given here and can be used to launch a new privileged container within Kubernetes
Other techniques within Execution tactic include exec'ing into an existing container, an application vulnerability that results in a shell or code execution at the OS level or rarely gaining access to a SSH server running inside a container.
Persistence: Kubernetes Cronjobs
The next Tactic covers techniques that allow the attacker to "persist" within the target cluster. Although not exhaustive, the techniques covered here do allow an attacker to plant their feet within the cluster and use the new setup access to continue working within the cluster.
The example that was covered in the slides was the usage of a Kubernetes Cronjob that will at periodic intervals attempt to send a reverse shell to the attacker IP.
The following yaml was used to create the Kubernetes cronjob that would give you a reverse shell every minute. So if a connection drops or the shell is killed, a new shell will be connected at the end of a minute.
Replace the <attacker-ip> with the IP address where netcat is listening on port 9091. You can change the port number as well if you would like.
Attackers can use various other techniques to remain hidden within the cluster, some other techniques are highlighted in the next slide.
Once attackers have gained code execution capabilities, attacker can choose to persist by performing out of band attacks as well like adding an SSH key to the underlying host, create shadow admin users and roles, update app code to provide shell access to containers, reuse secrets and tokens from Secrets and ConfigMaps or add a backdoor binary to running containers after gaining exec.
In my Opinion, attackers like to persist in most networks for the thrill of seeing how long it takes for owners to boot them out, to use the network as a means of a further attack/abuse or as a symbolic trophy in their "pwned" collection.
Privilege Escalation: Cluster-admin binding
The next Tactic is that of Privilege Escalation. Attackers are constantly on the lookout for mis-configurations, overly permissive policies or credentials that can help them impersonate another user, or even better a higher privilege user.
The example that we covered was that of Cluster-admin binding. In Kubernetes, a cluster-admin has the ability to administer the entire cluster and is considered to be the highest privilege user. The cluster-admin clusterrolebinding is used to map a user to the cluster-admin clusterrole privilege.
We enumerate all users to whom a clusterrolebinding is bound and look at the one's that have the cluster-admin clusterrole using the following command
kubectl get clusterrolebindings -o=custom-columns=NAME:.metadata.name,BINDING:.roleRef.name --sort-by=.roleRef.name
An attacker can then choose to discover the token for the listed clusterroles or attempt to create a new serviceaccount that will have the clusterrolebinding bound to it.
The following set of commands is used to create a serviceaccount called cluster-sa in the kube-system namespace and then bind it to a cluster-admin clusterrolebinding.
An administrative API request can be made to the API server using the obtained JWT token.
To be able to ensure they remain stealthy, inflict maximum damage, access protected resources or to even evade defenses, escalation to a user with higher privileges is a sure way of completing attacker objectives.
Some other commonly applied techniques are the reading and reuse of credential information, using the mapped service token mounted at /run/secrets/kubernetes.io/serviceaccount/token, identifying privileged containers to gain access to underlying nodes and the leaking of secrets through metadata endpoint and cloud platform service accounts.
Defence Evasion: Pod name similarity
This tactic comes into play when attackers are attempting to hide their presence and malicious activity within the cluster.
The example that we covered was that of a non-destructive technique in which attackers choose to use the naming convention of legitimate workloads to create malicious pods that have similar sounding names.
In the yaml that was shown on the slide had a name that was very similar to the naming convention of the coredns deployment within the kube-system namespace.
The yaml can be updated to reflect an existing deployment to hide in plain sight.
The techniques covered under this tactic can be destructive as the attacker may simply choose to wipe all log information about their activities within the cluster.
For example, an attacker can perform destructive actions to clear all pod logs by deleting the contents of /var/log/pods or clear Kubernetes events using kubectl delete events -all.
Credential Access: Container service account
Credential Access Tactic allows attackers to identify additional credentials, keys and tokens that can be misused to move laterally within the cluster or to attempt to breakout to the underlying infrastructure, be it the underlying node or the cloud platform itself.
The example that we covered from the various techniques under this tactic was the usage of the container service account mounted in each container (unless explicitly directed not to do so).
In the screenshot within the slide, an attempt is being made to access the Kubernetes API server from within a pod using the cURL command - curl https://kubernetes/api/v1 --insecure, without success.
However, when the token extracted from the mount point within the pod is used as an Authorization bearer token, the API responds to requests as the authentication requirement has now been met.
Most attackers will try and identify alternate regions, resources and networks that they can gain access to from their vantage point once they are within the cluster.
Attackers try to read and reuse secrets from the Kubernetes secrets store, service principal mounted for the cloud platform, the mount point token (whose example we saw earlier) and app credentials, tokens and keys from configMaps and pod descriptions.
Discovery: Instance Metadata API
The Discovery tactic covers techniques used by attackers to gather additional information from their vantage point once privilege escalation and credential access is completed.
In my opinion, the Discovery tactic is covered too late within the Threat Matrix as most attackers would perform discovery based scans, attempts to identify access etc. pretty early on in the attack chain.
The example technique that was covered in the talk was the access to the Instance Metadata API. Although, the screenshot shows the access to an Azure instance metadata endpoint, the technique is applicable to all cloud platforms.
Different instance metadata endpoints reveal different pieces of information depending on the cloud platform on which they are running.
To find additional resources that attackers can reach and exploit, various combination of techniques, some of them already covered are executed.
These include access to the Kubernetes API server, Kubelet REST API on ports 10250 and 10255, network service discovery using tools like Nmap, additional resource information obtained via access to the Kubernetes Dashboard and access to the cloud platform via credentials leaked within the cluster or via the Instance metadata endpoint.
Lateral Movement: Dashboard Access
The Lateral Movement tactic covers techniques that attackers use to move laterally to a different resource or escape the cluster into the cloud platform itself.
The example we covered was the access to the Kubernetes dashboard that allows an attacker to move to any container within the cluster (based on access) and from there moving on to the underlying node or the cloud platform via privileged pods, service principals and instance metadata credentials.
All the techniques covered under the Lateral Movement tactic have been covered under various other tactics so far, except the Helm Tiller endpoint technique.
Older versions of Helm (up to v2) had a server side component called Tiller which exposed an API endpoint that could be accessed without authentication on port 44134. This could be used to make a request to install resources using the helm client as shown below
The last tactic in the Kubernetes Threat Matrix is the Impact tactic. The techniques covered under here simply describe the actions that an attacker takes that describe the attacker's objectives.
In the example covered in the talk, we looked at a screenshot of the Kubernetes Dashboard which showed the hijacking of a cluster belonging to Tesla to run cryptomining workloads.
The techniques under the Impact tactic are used to describe the final objective of an attacker in the attack chain.
Some of the documented techniques include data destruction by rolling back deployments, deleting volumes and claims, altering the state of the cluster to create a Denial of Service condition (remove or alteration of configuration data for example).
Tips and Tricks to secure your cluster
The last leg of the talk covers tips and hardening tricks that can be employed to reduce the cluster's attack surface and improve the attack repelling capabilities of the components.
Quick things to note
Keep Kubernetes updated, including core components, API server, etcd etc. Ensure that any new updates, especially security updates are applied. Patch test before you update the core components.
Restrict SSH access to the nodes to only trusted IP addresses. Setup a jumpbox from which the nodes are accessible. Hide SSH for all other sources using the cloud platform's Security Group or in case of standalone servers, using local iptable rules.
Use Kubernetes NetworkPolicies judiciously, allow only what is required with a default deny otherwise. Create an elaborate access map of the services within your cluster and use NetworkPolicies to hide other services.
Audit roles and bindings, serviceaccounts, IAM access, look for shadow admins, use RBAC judiciously. Periodically, review and remove users, roles and bindings within the cluster.
Scan images in your pipeline using tools like Trivy to identify vulnerabilities that may become potential hazards. Image scanning has become a mainstream concept now. Run vulnerability scans for images in the pipeline or as a separate activity.
Remove deployments that are no longer required. Unused deployments can linger and may provide unwanted access to resources. In the case of privileged pods, a deployment may result in privilege escalation.
Run pods with a security context and a defined user, say no to root. Using a root user for everyday workloads must be avoided, else an attacker can abuse the access already available to move to a different part of the cluster.
Use namespaces to create logical segregation of resources. Logical segregation allows for easy monitoring, maintaining and troubleshooting of resources.
Do not store secrets in environment variables, in pod description or configMaps. Secrets in these locations can easily be read by an attacker in the event of a cluster compromise. Additionally, the official documentation of Kubernetes asks users to not store secrets within these resources but instead using the Kubernetes secret store or a Vault program.
Ensure minimum visibility from the outside. The less the attacker can see, the more difficult to attack it is for the attackers to discover services to attack.
Take application security into consideration, practice defense in depth. Application vulnerabilities can lead the attacker directly into a pod from where the attacker can launch other attacks or steal data.
Be wary of suspicious looking outbound traffic. Monitor egress to identify any tell-a-tale signs of a breach within your cluster.
Use a hardening guideline like the CIS Kubernetes benchmark and tools like kube-bench to identify potential misconfigurations. This will also help your cluster stay compliant.
Monitor and create an actionable roadmap into what your cluster is doing, who has access, what kind of access, what privileges are different components running, where are the logs going, what privileges does the IAM user have etc. Review this roadmap periodically for changes due to business decisions or the introduction of new configuration or software.
The following questions were asked by the audience during the talk. Some of the questions have been expanded to add more context.
Q1: Is Kubernetes vulnerable to any issues which have been assigned CVEs?
Short answer, yes.
Long answer, Kubernetes has a lot of moving parts and it is not by itself a single process. Vulnerabilities have been discovered in the past, affecting the parts within Kubernetes, the container runtime, the networking components etc.
Q2: How to find open Kubernetes unauthenticated end points?
Kubernetes has an API endpoint that is exposed on port 6443 or 443 based on variations of Kubernetes across different platforms. You can even change this port by supplying --secure-port <value> to the kube-apiserver. By additionally supplying --anonymous-auth=true you can enable anonymous requests to API server. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated. But for these requests to succeed, none of the numerous authentication methods that Kubernetes supports must be enabled.
In any case, making an API request to an endpoint that would otherwise return a 403 Forbidden HTTP status code should be a good way to check if the API server is accepting unauthenticated requests. You can do this by using the following cURL example:
Q3: How to know whether target is running Kubernetes or not and how to distinguish target is using AKS or other managed Kubernetes?
This is an interesting one. Purely from an attacker's perspective from the outside how do you know the target endpoint is Kubernetes. Like any other HTTP based service, Kubernetes also has some tell-a-tale signs, information that is leaked in response headers for example that could point to the target being a Kubernetes API endpoint.
For example, every HTTP response from an API server with the priority and fairness feature enabled has two extra headers: X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID.
Additionally, for an application running in Kubernetes there is no easy way to determine underlying host unless you have the ability to execute a server side request or a shell command on the host running the app.
To identify managed Kubernetes, you can check the A record or CNAME of the hostname of the HTTP endpoint. There are also IP ranges published by all major cloud platforms. Checking for the PTR record for an IP address after obtaining its A record may also point to the hostname and give away information whether is a managed kubernetes cluster or not.
Q4: How should someone perform an internal audit of Kubernetes resources for an organization? Are there any tools which do most of these tasks automatically?
The MITRE ATT&CK based threat matrix that was shown during the presentation is very context driven. Based on what is discovered, the next step in progression is identified, researched and executed.
There are tools available that can be used to identify common misconfigurations though like kube-bench, kube-hunter and kubeaudit etc.
Q5: It is not possible to attack Kubernetes directly, right ? What is covered in the slides is all post escalation, isn't it?
Like any system with an exposed interface, it is possible to attack Kubernetes directly. Also the term "post exploitation" needs to be anchored to where "exploitation" has actually occurred. For example, if you have got a reverse shell to a machine and that was your objective then everything before the shell was "pre-exploitation" and everything after is "post exploitation". However, if shell was only a means to gain access to a database server running on an internal network, then the description changes.
Similarly, the Threat Matrix describes Tactics and Techniques that can be used to attack Kubernetes, however, the pre and post exploitation is defined by context of what you set out to do with the target cluster.
Understanding how attackers move across different tactics using the Threat Matrix as a reference gives us a clear picture of what to expect when they do come calling. In the talk we covered examples of multiple techniques that attackers use in the real world to go from Initial Access, all the way to the Impact tactic for a Kubernetes cluster.
Kubernetes has a lot of moving parts. Every layer, plugin, app and configuration needs to be secured if you want the overall system to remain functionally secure.
ABOUT THE AUTHOR
Riyaz is a security evangelist, offensive security expert and researcher with over a decade of experience in the cyber security industry. His passion to break into some of the most well defended networks and systems in his career spanning 15 years has earned him a lot respect within the security industry. He has led Security Assessment and Penetration Testing teams at Pricewaterhouse Coopers (PwC) and Appsecco, and the Product Security Team at Citrix before co-founding Kloudle. Riyaz now specializes in cloud native, container and cloud security in general, helping build an easy to use security management platform to help companies enhance their visibility in the cloud, identify security misconfigurations and automate remediation for security gaps enabling compliance and operational security in multi-cloud environments. He is also an avid speaker and trainer and presents his research and findings at security conferences and community meetups around the world including BlackHat USA, BH Europe, BH Asia, nullcon and OWASP AppsecUSA.Specialties: Cloud (AWS, GCP, Azure, IBM, Others) Security, Cloud-Native Security, Kubernetes, Container Security, Web Application Security, Network and System Penetration Testing, Wireless Network Security, Malware Analysis and Reverse Engineering, Threat Modelling, Windows Forensics, Security Code Review, Vulnerability Research, Exploit Development and Reverse Engineering. Certifications: CKA, CKAD, OSCP
Enjoyed this read?
Subscribe to our newsletter and stay ahead with more great insights and resources on cloud security!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.