What is container security?

Containers have become hugely popular. In 2019, Gartner reported that 30% of organizations worldwide were running containerized applications. They predicted this figure would jump to 75% by 2022. So far, their prediction seems to be spot-on.

Unfortunately, container security is much more difficult to achieve than security for more traditional compute platforms, such as virtual machines or bare metal hosts. For evidence, referring to the research that we recently published in which 62% of CISOs told us that container runtime environments have negatively impacted their ability to detect and manage software vulnerabilities.

To understand why, let’s first look at what container security is, then what makes containers challenging to secure, and finally, steps you can take to do container security right.

Container security is the practice of applying security tools, processes, and policies to protect container-based workloads. Container security has two main functions:

• Secure the container image. Application developers commonly leverage open-source software when building containerized applications. In fact, the market research firm Forrester says that the average container image is comprised of 70% open-source software.[1] And unfortunately, open-source software is often fraught with security vulnerabilities. To properly secure applications, developers need to discover and eliminate these vulnerabilities.
• Secure container runtime configuration. To function effectively, containers need to be able to communicate with each other and with network services. However, to be secure, containers must be properly isolated from each other and from the host system itself. If containers are run with privileged flags, or if they receive details about host processes, they can easily become points of compromise for corporate networks. Properly managing Linux namespaces, cgroups, and access controls are all critically important for container security.

Between the two objectives listed above, securing container runtime configuration is the easier one to accomplish. Many good security tools provide that function, and benchmarks from the Center for Internet Security (CIS) are clear and prescriptive.

Ensuring that your container images are free of vulnerabilities is much harder—especially when the image is largely based on open-source components. Four types of tools are commonly used to detect software vulnerabilities:

• Source-code tests that are used in development environments
• Image scanners that are used after the container image has been built
• Network scanners that see systems from the “outside” perspective
• Run-time detectors that see the container from the “inside” perspective

Let’s look at each type.

Source code tests

Products that scan source code before the container is built are known as Software Composition Analysis (SCA) tools and Static Application Security Test (SAST) tools. The main difference between the two is the SCA tools are only able to detect vulnerable open-source code, whereas the SAST tools can also detect vulnerabilities in custom code.

Some SCA and SAST vendors have automated their products to align with the fast pace of modern DevOps teams, but many are still slow and cumbersome. In fact, in our recent CISO research, 28% of CISOs told us that application teams sometimes bypass these types of tests to speed up delivery.

Another problem is that these types of security tools tend to produce a large number of false positives. The CISOs who participated in our survey told us that, on average, only 42% of application vulnerability alerts produced by these types of tools required action; the rest were false positives. False positives waste your time. They are highly frustrating to both security practitioners and developers.

Despite the limitations of SCA and SAST tools, they serve very important roles in container security. For example, SCA tools will tell you about the license restrictions associated with open-source software, and SAST tools will tell you about security deficiencies associated with custom code. These are both essential functions.

Image scanners

Once a container image has been built, it is typically stored in a registry until it is needed. By examining the various packages that are contained within the image, and by conducting a build dependency analysis to identify any new dependencies that were brought into the application during the build process, you can identify known vulnerable packages in the container.

This is how most Cloud Workload Protection Platforms (CWPPs) operate as well as popular image scanners such as Anchore and Clair.

The good news is that this type of test tends to be highly automated, so it doesn’t slow developers down. However, it is prone to the same false-positive problem that plagues most SCA and SAST tools (see above).

Network scanners

Most enterprise security teams are familiar with traditional network-based scanners because they have been around for a long time—long before containers ever existed. Well-known vendors include Qualys, Rapid7, and Tenable.

These products see systems from the “outside” perspective—which is to say, the attacker’s perspective. They can tell you everything you need to know about vulnerabilities in host operating systems, application frameworks, and network devices like switches and routers. They can tell you what ports are exposed and whether encryption is being used. But they are not able to tell you much about the source-code vulnerabilities inside a container. Containers are opaque to traditional vulnerability scanners.

As a result, in our recent CISO survey, 74% of CISOs told us that traditional vulnerability scanners are no longer fit for purpose in today’s cloud-native world.

Runtime vulnerability detectors

The new kid on the block is a runtime vulnerability detector that uses an agent to examine the behavior of the processes inside the running container. The agent watches as each process loads a file or makes a call to a certain function that is performed by an open-source library. In this way, the security agent can avoid the problem of false positives triggered by vulnerable libraries that are present but not used by the application, or not used in a certain manner that exposes a vulnerability.

An example is a math library which can be used for many different functions—addition, subtraction, multiplication, logs, etc. Suppose the multiplication function contains a vulnerability, but the other functions do not. The mere presence of the library inside the container will cause an SCA tool or an image scanner to issue an alert. But a runtime agent that is watching the behavior of the container can go further: it can tell you whether the multiplication function is being used. And that type of precision can help developers and security practitioners from wasting a lot of time fixing something that isn’t broken.

Pretty neat, isn’t it?

Security for containers isn’t a singular solution — it’s a process that starts with building the container, expands to evaluating its contents and configuration, and incorporates run-time assessment and risk analysis. Here is a checklist of best practices for how to approach container security.

• Stay slim. To reduce an application’s attack surface, developers need to remove any components that aren’t needed.
• Use only trusted base images. As part of your CI/CD process, developers should only be allowed to use images that have been previously scanned and found to be trustworthy.
• Harden the host operating system. Use scripts to configure hosts properly based on the CIS benchmarks. Consider using lightweight Linux distributions that are specifically designed for hosting containers, such as Red Hat Enterprise Linux Atomic Host or CoreOS.
• Remove privileges. Running privileged containers is a security risk because it creates an opportunity for a malicious user to take control of the host system, and this endangers your entire infrastructure.
• Manage secrets. Secrets, such as database credentials, API keys, SSL keys or encryption keys, need to be managed to protect them from discovery. Consider using one of the many good secrets management systems that are commercially available.
• Run source code tests. Although legacy SCA and SAST tools can be slow and cumbersome to use, many have been evolving in recent years to support DevOps initiatives and automation, and they are still an important part of container security. Use one or more of these mainstay types of application security tests to keep track of open-source software, unacceptable license restrictions, and vulnerabilities in custom code.
• Observe the application in runtime. To fully eliminate the problem of false-positive detections described above, use a runtime vulnerability detection product such as Application Security from Dynatrace. In addition to detecting vulnerabilities, Dynatrace’s application monitoring platform can also help you monitor and manage application performance, reliability, and other metrics.

By combining a runtime vulnerability detection system with an observability platform, not only can you eliminate false positives, but you can also better understand the business risk that each vulnerability represents.

An example of such a system is Dynatrace Application Security. Since Dynatrace operates at runtime, it can observe whether a potentially vulnerable library is actually being loaded by the application and whether it is being used in a manner that exposes the application to attack. This eliminates the false positive problem that still plagues most SAST, SCA, and CWPP tools.

The Dynatrace observability platform has a full understanding of the runtime context of your application, which means it knows whether your application is connected to the Internet and whether it has access to sensitive company data. The Dynatrace AI engine, Davis, automatically combines this information with threat context information (for example, whether there is a public exploit available for each vulnerability) and generates a security score that reflects the true business risk of each application vulnerability.

Knowing the relative business risk of each vulnerability—as represented by the Davis Security Score—allows you to correctly prioritize remediation actions. The Davis Security Advisor goes one step further by recommending which *libraries* represent the most business risk, thus which ones should be prioritized for updates. A single library often represents a large number of vulnerabilities across multiple applications.

Once you have correctly identified priorities and removed false positives, container security becomes a workflow and execution problem. To make this process efficient, Dynatrace:

• Automatically identifies which of your developers is responsible to remediate each vulnerability, based on tags that you configure. This helps you automatically assign tickets in systems such as Jira.
• Automatically identifies the required upgrades and/or potential workarounds to remediate each vulnerability, which saves developers time.
• Identifies in real-time when the vulnerability has been removed from the application and automatically updates priority lists.

[1] Forrester “The State of Application Security, 2021″, Sandy Carielli, March 23, 2021.