(This is Part 5 of a 9 part blog series that explains the Kubernetes MITRE ATT&CK like Threat Matrix created by Microsoft from an attacker perspective and attempts to provide how real world attackers use the techniques covered in the framework to gain access, execute, persist and explore Kubernetes cluster environments.)
Use this index to navigate to and read the rest of the posts in this series
(This blog post discusses the fifth tactic described in the MITRE ATT&CK framework for Kubernetes - Defense Evasion)
In the last post, we saw the techniques in the Privilege Escalation tactic of the MITRE ATT&CK framework for Kubernetes. Let’s look at the next tactic, Defense Evasion and the techniques that attackers use within this tactic. For reference, here’s the framework that Microsoft created as a visual cue to the overall tactics and techniques that attackers use when attacking a Kubernetes cluster.
There are always going to be defenses that attackers will have to overcome, bypass, defeat or remove in order to fulfil their objectives. From a system’s engineering perspective, defenses are created inherently when the system is designed or are added as part of the overall hardening and security controls that need to be present for the system to become functional.
In the world of Kubernetes, many features and controls are already present that prevent an attacker from moving to higher privileges or underlying architecture etc. efficiently. The Defense Evasion tactic of the MITRE ATT&CK framework talks about some of the techniques that an attacker could employ to avoid being detected and hide their activity within the cluster.
Although a destructive practice, this remains a very easy win for an attacker who wants to remain concealed within a given system. Depending on the kind of application service the container provides, the attacker may choose to remove application specific logs, OS level logs or fudge data if logs are being streamed to a remote collector.
By design, log files can only be tampered by privileged processes, hence the Defense Evasion tactic is listed after the Privilege Escalation tactic described in an older blog post.
Kubernetes log information can come from various places. Some applications may print logs to stdout and stderr which can be obtained using kubectl logs <container-name>. In cases where applications do not log to console, a sidecar container may be used that streams the logs from a shared mount path and send it to a log aggregator and querying setup like ELK.
Additionally, some clusters may also use external tools like Fluentd or Stackdriver (GKE, managed) to perform log aggregation.
Node and container runtime logs are found in the /var/log/containers/ directory (which is in turn symlinked to the /var/log/pods directory with directories for each namespace in the cluster). These are on every node within the cluster.
Attackers can reach various log locations within a Kubernetes cluster owing to their privileged access and persistence tactics. An example of an attacker using a privileged container and a hostPath mount to access the nodes log files is shown below
An attacker may disrupt logging or tamper with existing logs if access to the underlying nodes is obtained using a privileged container, providing root capabilities on the underlying node.
Kubernetes events allow administrators to monitor resource state changes or errors, such as container creation, image pulls, resource quota exceeding, pending pods etc. Kubernetes events can be very useful for identifying changes that occur in the cluster.
To view events for your cluster across all namespaces you can run kubectl get events -A (add -o yaml for a more detailed yaml format output).
Pod specific events can also be seen using kubectl describe pods pod-name under the Events section. The output here is the same as kubectl get events but for specific pods.
Attackers may want to delete these events to in an attempt to hide themselves within the cluster and to avoid detection using commands like kubectl delete events —all (do not run this in production).
This technique has some overlap with the Persistence tactic that we covered couple of posts ago. Pods that are created by controllers such as Deployment or DaemonSet have a random suffix in their names. Attackers can use this fact and name their backdoor pods as though they were created by the existing controllers and are part of a deliberate cluster deployment. For example, an attacker could create a malicious pod named coredns-{random suffix} (for example, coredns-7944c66d8d-wfq47) which would look related to the CoreDNS Deployment.
Additionally, attackers can deploy their containers in the kube-system namespace where the administrative containers reside to ensure even greater evasion.
This technique applies to network security in general and is not restricted to Kubernetes. More often than not, attackers use proxy servers, compromised machines on the Internet or anonymous networks like TOR to hide their true origin IP address. In case of Kubernetes, this anonymous and incorrect source of communication can be used to communicate with the control plane (kubectl API access) or to communicate and compromise application’s running within the cluster.
Defense Evasion requires escalated privileges and a deeper understanding of the cluster as the attacker will need to identify where the logs are, if they are being streamed elsewhere and how to continue being hidden within the cluster using privileged access to other namespaces.
In the next post we will see how attackers attempt to gain access to secrets and credentials within the cluster using the framework as our guide.
This article is brought to you by Kloudle Academy, a free e-resource compilation, created and curated by Kloudle. Kloudle is a cloud security management platform that uses the power of automation and simplifies human requirements in cloud security. If you wish to give your feedback on this article, you can write to us here.
