Let’s face it – containers are at the forefront of virtualization technology, a somewhat new paradigm for DevOps and agile organizations across our industry. We spoke with several organizations who have shed apprehension around the technology, stepped off the edge and plunged directly into containers, as well as companies just dipping their toes. Through this, we’ve noticed a particular trend across the industry regarding containers; they have the potential to be one of the most secure deployment mechanisms for applications and business data. Although containers have many characteristics that imply inherent security, certain measures must be taken to ensure they meet the level of protection required for enterprises.
Overview of container security, gaps, and vulnerabilities
As companies begin to understand container technology and the benefits that come with containerization, a sense of comfort tends to set in. In today’s ever-changing security landscape, the concern moves away from the containers themselves, and becomes security based; more specifically, if we store our data in containers, how can we assure it’s secure?
In my initial introduction to containers last month, I explained what most companies first come to understand about the technology – that container images are immutable. This means they can’t be changed. From a security perspective, it would seem they’re inherently secure. The perception is, if container images can’t be changed, it would be very difficult for threat actors to tamper with them.
The more you uncover about container technology, the more you come to understand attackers are finding innovative ways around the seemingly secure characteristics of containers. Threat actors can take advantage of security gaps in the container deployment methods as well as vulnerabilities in the images themselves. Hence, companies need to adopt a process for securing containers to address risks at each phase of the container life cycle.
Container Security
As companies seek new ways to secure containers for their different environments, there are several factors to keep in mind. The following provides a process that organizations can adopt to secure containers no matter the environment they’re deployed in:
Digitally sign images before committing them to registry
In a typical container-driven ecosystem, developers write code, package that code as an image and commit it to a registry. DevOps engineers and other operations team members then pull those images from the registry, run the code in a test environment and then, if tests are successful, push to production.
The number of container images and repositories is growing at a rapid pace and most are easily accessible through the internet. As a developer, it can be very tempting to pull useful images even if you have absolutely no idea where they originated or who created them, making it hard to identify whether they’re malicious or not. This free-for-all atmosphere provides the perfect opening for attackers, who can dupe users into pulling malicious images.
You can avoid pulling malicious items from the repository by prohibiting the use of unverified images in your organization. Additionally, incorporating digital signatures or some PKI (public key infrastructure)-based chain of trust as part of your image validation process will add an extra layer of security.
One way to set up a validation process is by having developers digitally sign their images before committing them to the registry. In Docker, for example, you can use what is known as Docker Content Trust. When the image gets pulled from the registry, it verifies the signature ensuring the image comes from the proper organization and matches the created code.
Scan images for security gaps/vulnerabilities
‘Shadow Container’ is part of a multi-stage APT (advanced persistent threat) attack demoed by Aqua Security at Black Hat USA 2017. The shadow container adds persistence and concealment abilities to malicious containers in the victim’s host system or hypervisor. This ability is crucial for an attacker, because changes on a running container are automatically deleted each time the host or Docker daemon is restarted. Since the VM boots from a base image, any malicious code that was installed on it will also be removed. The shadow container attack basically preserves the container’s state and, in turn, the malicious script in it.
Additionally, in a study conducted by Banyan, a substantial number of images – 30% of official images and 40% of general images – in Docker Hub (arguably the most widely used source of publicly available container images) have been found to have vulnerabilities like ShellShock, Heartbleed, Poodle, etc.
Although these vulnerabilities were found in Docker Hub container images, it’s highly likely this kind of problem could also manifest itself in other registries; even private registries used by enterprises. This has huge implications since, by design, each of these images can be easily launched into a large number of container environments. If an image has vulnerabilities, all containers launched from it would automatically inherit those same vulnerabilities.
To mitigate this risk, it’s recommended that images be scanned for known vulnerabilities in both the operating system and application-level packages. This behavior ensures real-time analysis in order to gain detailed information about the image contents.
Audit images
Image auditing is important for several reasons. Auditing helps developers keep track of the images running in production as well as the multiple versions of code used to determine whether images are up to date. Consider this basic security hygiene, which serves to weed out expired or out-of-date images and replace them with updated or ‘patched’ versions.
Furthermore, audits enable organizations to catch containers whose versions are still vulnerable to specific types of threats, e.g, Heartbleed. Once vulnerable containers are identified, the required patch can be applied to the gold image and cascade those changes to containers created from it. These changes can be fairly tedious and time consuming if done manually, but there are tools that can automate the process.
Apply Zero Trust and Isolation
Another secure characteristic of containers is that they’re inherently isolated from both the host system and other containers, however, it’s still possible for a container (and anyone using it) to bypass isolation mechanisms and gain access to files on the host or even perform privilege escalation in what is known as a “container breakout.” One proof of concept exploit takes advantage of certain Linux kernel capabilities.
As fundamental concepts of information security, zero trust and isolation can also be applied to containers, especially to prevent container breakout. One of the key pillars of zero trust is the principle of least privilege, which preaches the practice of limiting user access rights to bare minimum permissions to get work done.
By applying this principle to containers, you continue to add layers of security complexity for any unwanted guests. For example:
- Set the filesystem to read only to hinder the writing of malicious scripts or modification of file contents.
- Limit access to memory. This effectively prevents attackers from consuming all memory resources while also preventing other containers from accessing resources.
- Don’t allow your containers to run as root to avoid a myriad of potential issues.
Apply runtime threat detection
During runtime, you will want to monitor and detect for signs of malicious activity such as resource abuse, anomalous behavior, user deletions, and so on. In addition to monitoring and detection, you will also want to take appropriate action when necessary. So, after looking at logs and gaining a bit of additional context, you will need to initiate a response. This may include additional monitoring and logging, applying more isolation rules, disabling user(s), or even deleting a container.
Remember, automation is key. If you can reduce the human element, you can reduce the probability of error while improving the quality, consistency, timeliness and efficiency of each process. Tools like Cloudify, Aqua, and Twistlock, which leverage automation, can help simplify these processes.
Containers have several qualities that make them inherently secure, like having fewer running processes, isolation, immutability, and so on. But as we’ve outlined here, there are security gaps and vulnerabilities in the container lifecycle as well as in container images themselves that can be exploited by attackers. It’s your responsibility to know where these gaps and vulnerabilities lie and take appropriate action to secure them.