Bringing Content Trust into the World of docker-compose

Call it trusted-compose

A central question in application security is: how do we ensure that our Docker containers actually run the code that we intend to run? At SSE, we noticed that this question does not always have a simple answer. What’s for sure, however, is that integrity is by no means guaranteed in a vanilla setup. What’s more: as soon as more than one container is involved, the standard approach for ensuring image authenticity, known under the name of Docker Content Trust (DCT), breaks down.

Why is that? It’s a combination of two factors: first, Docker images are typically tagged with short labels (such as latest) and then uploaded to registries that operate outside the developer’s control (such as the Docker Hub); second, it is not technically enforced that image contents for a given tag are not tampered with while they are stored at the registry, waiting for deployment.¹ In other words, if you push my-app:latest (or any other image/tag combination), you can’t be sure that you actually end up with the same image when pulling it. As a result, it’s hard to be certain what code is actually running on your infrastructure.

So, how can this problem be addressed? Can we achieve zero-trust Docker deployments, as far as registry storage is concerned? Luckily, it turns out that the standalone docker binary has a mechanism to enable verification of image signatures. Container orchestration solutions, however, are lacking native support. (Update: There is now Connaisseur, a solution for Kubernetes by Philipp, one of my colleagues at SSE. Read his Medium article on Connaisseur!) We are thus introducing trusted-compose (git), a tool for adding end-to-end image integrity to projects orchestrated with docker-compose.

tl;dr

trusted-compose is a wrapper that transparently adds image signature verification to docker-compose. Provided you make some small configuration adjustments (see below), it will deliver the following:

• trusted-compose pull: Pull images for all services, verify signatures, push valid images to a local registry (fire it up with docker run).
• trusted-compose push: Sign images for all services, push to the remote registry.
• trusted-compose build: Pull parent images for all services, verify signatures, push valid images to the local registry, and call docker-compose build. All subsequent pull operations will be directed at the local registry.
• trusted-compose ... (anything else): Call docker-compose with the same arguments, but with local registry enforcement turned on.

A Standalone Concept: Docker Content Trust (DCT)

Docker itself approached this problem by adding Docker Content Trust (DCT) to their standard tool chain. The feature is enabled by setting the environment variable DOCKER_CONTENT_TRUST=1 and takes care of image signing during docker push as well as signature verification during pull operations (docker pull as well as the implicit pulls done by docker build). The underlying technology is based on the Notary implementation of The Update Framework (TUF).

Unfortunately, DCT only works out of the box with Docker’s standalone binary (docker). If you’re using docker-compose or another orchestration tool, the functionality is not readily available. I’d like to show how to make it work with docker-compose.

A Real-World Solution: trusted-compose

The core inspiration of trusted-compose (git) is the realization that docker-compose’s pull and push operations are equivalent to running several docker pull or docker push commands in a row, one for each service defined in docker-compose.yml.

The challenge is thus to create a wrapper that accepts the same command line arguments as docker-compose, but intercepts the pull and push commands in order to replace them with the corresponding sequential calls of the standalone docker command. We also need to intercept the build command in order to safely pull the parent images specified in each service’s Dockerfile before handing control back to docker-compose.

All other docker-compose commands, such as up, down, run and the like, receive no special treatment: we simply invoke docker-compose itself and pass on the arguments.

The Devil is in the Detail

Caution, however, is advised: we need to carefully consider a series of edge cases to avoid ending up with a flawed and insecure implementation. In particular, when running the build command, it does not suffice to pre-pull parent images with DCT enabled before handing control to docker-compose build. We also have to prevent docker-compose from pulling any additional images on its own account!

Here’s how that may happen: after trusted-compose build has completed a trusted pull for each parent image, docker-compose build is called to run the regular build process, using the existing images. At this point, a race condition may occur: if a parent image has been pruned from the local system between the two steps, docker-compose build will pull the image again — this time without verifying the signature!

This type of race condition is known as a TOC/TOU conflict. trusted-compose avoids it as follows:

1. Fire up a registry on the local system, such as on localhost:5000.
2. Whenever we pull an image and successfully verify its signature, re-tag the image and push it to the local registry.
3. Make sure that all pull operations initiated by docker-compose are directed to the local registry.

This construction ensures that any images pulled by docker-compose have been validated. If an image is not available locally (e.g. because its signature could not be verified), the pull operation will fail. The registry functions as an ephemeral cache for validated images, and as such can be disposed of and repopulated at any time.

Configuration Adjustments

Step 3, the redirection of image pulls to the local registry, requires some fiddling with your image specifiers. The idea is to prefix each image specifier with a registry placeholder which is then dynamically replaced with the local registry whenever we invoke docker-compose from within trusted-compose. In order to perform the sequential trust-enabled pulls from the remote registry (with DCT turned on), the placeholder will just be replaced by the empty string. The replacement happens by automatically injecting the option --build-arg DOCKER_REGISTRY=localhost:5000 when appropriate.

For this to take effect, Dockerfiles need to have their FROM lines adapted accordingly. For example,

FROM debian:stable

needs to be modified to read:

ARG DOCKER_REGISTRY
FROM ${DOCKER_REGISTRY}debian:stable

Similarly, all image values configured in docker-compose.yml need to be prepended with ${DOCKER_REGISTRY}. This is necessary to allow trusted-compose to inject its logic.

Et voilà: With these adjustments in place, trusted-compose will bring Docker Content Trust into the world of docker-compose. 🎉

Note, however, that some limitations exist (e.g. limited support of command line options for pull and push). As these limitations may be lifted in the future, it’s best to check in the documentation for up-to-date information.

Conclusion

While the idea of containerizing applications has caused a paradigm shift in application deployment and maintenance, new issues of software integrity enforcement have not been treated with the necessary vigor. Although Docker Content Trust provides a suitable framework for solving the problem for single Docker images, solutions for container orchestration setups including docker-compose have been lacking.

During our work at SSE, we time and again encountered this problem on various hardening projects. We thus set out to improve the situation by releasing trusted-compose, a transparent wrapper to close the integrity gap.

¹ While there generally is no reason to assume that registries act maliciously, there is also no compelling reason why they should deserve your utmost trust — after all, they are in the prime spot for injecting malicious code into your production infrastructure. It can’t be denied that in certain cases, that would surely be “interesting” for particular parties. Besides, it’s always a good idea to uphold the obvious principle: trust is good, control is better.