Part 5 - Mapping the MITRE ATT&CK framework to your Kubernetes cluster: Defense Evasion

Posted by
Riyaz Walikar
on
January 21, 2021
· 5 min read

(This is Part 5 of a 9 part blog series that explains the Kubernetes MITRE ATT&CK like Threat Matrix created by Microsoft from an attacker perspective and attempts to provide how real world attackers use the techniques covered in the framework to gain access, execute, persist and explore Kubernetes cluster environments.)

Use this index to navigate to and read the rest of the posts in this series

(This blog post discusses the fifth tactic described in the MITRE ATT&CK framework for Kubernetes - Defense Evasion)

In the last post, we saw the techniques in the Privilege Escalation tactic of the MITRE ATT&CK framework for Kubernetes. Let's look at the next tactic, Defense Evasion and the techniques that attackers use within this tactic. For reference, here's the framework that Microsoft created as a visual cue to the overall tactics and techniques that attackers use when attacking a Kubernetes cluster.

Kubernetes ATT&CK matrix

Defense Evasion

There are always going to be defenses that attackers will have to overcome, bypass, defeat or remove in order to fulfil their objectives. From a system's engineering perspective, defenses are created inherently when the system is designed or are added as part of the overall hardening and security controls that need to be present for the system to become functional.

In the world of Kubernetes, many features and controls are already present that prevent an attacker from moving to higher privileges or underlying architecture etc. efficiently. The Defense Evasion tactic of the MITRE ATT&CK framework talks about some of the techniques that an attacker could employ to avoid being detected and hide their activity within the cluster.

Clear container logs

Although a destructive practice, this remains a very easy win for an attacker who wants to remain concealed within a given system. Depending on the kind of application service the container provides, the attacker may choose to remove application specific logs, OS level logs or fudge data if logs are being streamed to a remote collector.

By design, log files can only be tampered by privileged processes, hence the Defense Evasion tactic is listed after the Privilege Escalation tactic described in an older blog post.

Kubernetes log information can come from various places. Some applications may print logs to stdout and stderr which can be obtained using kubectl logs <container-name>. In cases where applications do not log to console, a sidecar container may be used that streams the logs from a shared mount path and send it to a log aggregator and querying setup like ELK.

Additionally, some clusters may also use external tools like Fluentd or Stackdriver (GKE, managed) to perform log aggregation.

Node and container runtime logs are found in the /var/log/containers/ directory (which is in turn symlinked to the /var/log/pods directory with directories for each namespace in the cluster). These are on every node within the cluster.

Attackers can reach various log locations within a Kubernetes cluster owing to their privileged access and persistence tactics. An example of an attacker using a privileged container and a hostPath mount to access the nodes log files is shown below

Accessing logs using privileged container

An attacker may disrupt logging or tamper with existing logs if access to the underlying nodes is obtained using a privileged container, providing root capabilities on the underlying node.

Delete Kubernetes Events

Kubernetes events allow administrators to monitor resource state changes or errors, such as container creation, image pulls, resource quota exceeding, pending pods etc. Kubernetes events can be very useful for identifying changes that occur in the cluster.

To view events for your cluster across all namespaces you can run kubectl get events -A (add -o yaml for a more detailed yaml format output).

Kubernetes events in a cluster

Pod specific events can also be seen using kubectl describe pods pod-name under the Events section. The output here is the same as kubectl get events but for specific pods.

Attackers may want to delete these events to in an attempt to hide themselves within the cluster and to avoid detection using commands like kubectl delete events --all (do not run this in production).

Pod / container name similarity

This technique has some overlap with the Persistence tactic that we covered couple of posts ago. Pods that are created by controllers such as Deployment or DaemonSet have a random suffix in their names. Attackers can use this fact and name their backdoor pods as though they were created by the existing controllers and are part of a deliberate cluster deployment. For example, an attacker could create a malicious pod named coredns-{random suffix} (for example, coredns-7944c66d8d-wfq47) which would look related to the CoreDNS Deployment.

A malicious pod yaml using standard pod name to avoid detection

Additionally, attackers can deploy their containers in the kube-system namespace where the administrative containers reside to ensure even greater evasion.

A malicious pod running in the kube-system namespace

Connect from Proxy server

This technique applies to network security in general and is not restricted to Kubernetes. More often than not, attackers use proxy servers, compromised machines on the Internet or anonymous networks like TOR to hide their true origin IP address. In case of Kubernetes, this anonymous and incorrect source of communication can be used to communicate with the control plane (kubectl API access) or to communicate and compromise application's running within the cluster.

Conclusion

Defense Evasion requires escalated privileges and a deeper understanding of the cluster as the attacker will need to identify where the logs are, if they are being streamed elsewhere and how to continue being hidden within the cluster using privileged access to other namespaces.

In the next post we will see how attackers attempt to gain access to secrets and credentials within the cluster using the framework as our guide.

References

More Articles

An Attacker's Approach to Pentesting IBM Cloud - fwd:cloudsec 2021
Posted by
Riyaz Walikar
on
September 14, 2021

Slides of the talk presented at fwd:cloudsec 2021, titled - "An Attacker's Approach to Pentesting IBM Cloud". The talk contains examples of attack vectors, interesting things in IBM Cloud and future work to improve documentation.

Read More
Kloudle is a BlackHat USA 2021 Sponsor
Posted by
Akash Mahajan
on
August 4, 2021

From being trainers at BlackHat to sponsoring at the conference. A bit about our journey in brief.

Read More
You do not want to miss these talks at Black Hat USA!
Posted by
Riyaz Walikar
on
August 3, 2021

A list of talks in the Cloud Security and Platform domain that we have added to our must watch list of talks as the sheer number of sessions at the Black Hat briefings can be overwhelming!

Read More

Ready to give Kloudle a try?

We help you monitor and prevent any data breaches.

Let's Talk