Digests in Docker
June 10, 2020
Most Docker images are run as containers by invoking something along the lines of
docker run <image>:<tag>. However, it’s also possible–and often necessary for security purposes–to run by also specifying
@sha256:<digest>. But what does this actually do, and what security vectors does this actually provide?
What is a Docker digest?
Starting Docker containers requires an image, which is presented in the format of
[namespace/]<image-name> followed by a tag
:<tag-name>. If no tag is specified,
latest is used. This tag is a pointer to an image–a set of files and metadata that Docker can use to run a container. Some images fall under the root namespace, such as
ubuntu, which we’ll see in the demos below.
Docker digest is provided as a hash of a Docker image supported by the Docker v2 registry format. This hash is generated as
sha256 and is deterministic based on the image build. This means that so long as the Dockerfile and all build components (base image selected in
FROM, any files downloaded or copied into the image, etc) are unchanged between builds, the built image will always resolve to the same digest. This is important as a change in digest indicates that something changed in the image.
More information about images can be found in Docker documentation
Let’s see it in action
Digests for pulled images can be listed by running
docker images --digests (lines wrapped for better viewability):
$ docker images --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE ubuntu 16.04 sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209 005d2078bdfa 6 weeks ago 125MB ubuntu 20.04 sha256:8bce67040cd0ae39e0beb55bcb976a824d9966d2ac8d2e4bf6119b45505cee64 1d622ef86b13 6 weeks ago 73.9MB
Here we can see that we’ve pulled two tags for the Ubuntu image - 2016 and 2020 LTS based containers. Let’s take a look at them and see what release they’re running.
uname -a, which is commonly used to check a host’s running version, won’t work inside containers as it reports based on the running kernel, which is shared from the host OS.
$ docker run --rm ubuntu:20.04 /bin/bash -c "cat /etc/lsb-release" DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04 LTS"
$ docker run --rm ubuntu:16.04 /bin/bash -c "cat /etc/lsb-release" DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
All good. We’ve run the tags we wanted and verified that the container contents are what we’d expect.
Now, let’s re-run 20.04 with a checksum (copied from the output of
docker images --digests above):
$ docker run --rm ubuntu:20.04@sha256:8bce67040cd0ae39e0beb55bcb976a824d9966d2ac8d2e4bf6119b45505cee64 /bin/bash -c "cat /etc/lsb-release" DISTRIB_ID=Ubuntu DISTRIB_RELEASE=20.04 DISTRIB_CODENAME=focal DISTRIB_DESCRIPTION="Ubuntu 20.04 LTS"
Also what we’d expect to see. But now, let’s instead copy the digest of the 16.04 image:
$ docker run --rm ubuntu:20.04@sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209 /bin/bash -c "cat /etc/lsb-release" DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
Now, we’re showing a tag of
20.04, but our image is
16.04. Let’s investigate:
I’ll kick off an idle process in new container:
docker run --rm -ti ubuntu:20.04@sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209 /bin/bash using the tag and digest from the last scenario above, then check the
docker ps output (again wrapped for viewability):
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 44739fa037d2 ubuntu:20.04 "/bin/bash" 2 seconds ago Up 2 seconds xenodochial_maxwell
We can now see that
docker ps output shows the tag that was specified by the
docker run command.
We can even use tags that don’t exist:
$ docker run --rm ubuntu:tag-does-not-exist@sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209 /bin/bash -c "cat /etc/lsb-release" DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"
docker pull works for that non-existent tag:
$ docker pull ubuntu:tag-does-not-exist@sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209 sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209: Pulling from library/ubuntu e92ed755c008: Pull complete b9fd7cb1ff8f: Pull complete ee690f2d57a1: Pull complete 53e3366ec435: Pull complete Digest: sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209 Status: Downloaded newer image for ubuntu@sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209 docker.io/library/ubuntu:tag-does-not-exist@sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209
The digest is scoped to the image namespace and name, however, so the following does not work when pulling a non-existent image even though that digest exists on the Docker host:
$ docker pull not-an-image:tag-does-not-exist@sha256:db6697a61d5679b7ca69dbde3dad6be0d17064d5b6b0e9f7be8d456ebb337209 Error response from daemon: pull access denied for not-an-image, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
So what’s actually going on here
The above may seem counter-intuitive. Often sha256 checksums are used to verify a payload, such as when using package managers to install or upgrade applications.
This implementation behaves more like
git. In git, each commit has a hash - a checksum of the commit contents - that can be used to uniquely identify a change set within a repository. Unlike git, however, which allows brute-forcing an update to a commit that would break the checksum (i.e. new changes on an existing commit), the Docker digest is used both as an identifier (ability to look up) and a verification (ensure that the image pointing to what’s intended).
Docker tags behave much like git tags as well. A tag can point to one and only one digest, but while digests are immutable, tags can be updated to move the pointeer to a new digest. That means that running
docker run ubuntu:20.04 may yield a different result between runs. This can present risk in terms of security and operational impact as it allows for changes with no obvious visibility.
The solution to preventing change is pinning a digest to the running container - done by appending
@sha256:<digest>. This solves the issue of allowing a container to change unexpectly.
That said, given the examples above, we can see that when a digest is specified at container startup, the tag is no longer respected in any way. The Docker registry simply tracks pointers to which digest a tag references at present, so in order to maintain backwards compatibility for previously started images, it does not check the tag against the registry at startup.
Consider this scenario:
You’re running a production container, and you want to pin a particular distribution version (let’s stick with Ubuntu here), so you pin the image tag to
ubuntu:20.04@sha256:8bce67040cd0ae39e0beb55bcb976a824d9966d2ac8d2e4bf6119b45505cee64. A week later, that image may receive security patches, and the tag will be updated to a new digest as its contents have changed. If the Docker runtime were to check the registry at startup to ensure that the tag specified maps properly to the digest specified, things get complicated in a hurry. To keep things functional, when this scenario is encountered, Docker prioritizes the digest specified and runs that regardless of whether the tag is or even ever was existent or pointing to the referenced digest.
The behavior also defaults to show the tag being run (i.e.
docker ps output) as the tag specified at container creation time and does not automatically map a running digest back to a local image tag that maps to the image. This also would be quite complicated as multiple tags can point to the same digest and tags can point to different digests over time.
tl;dr - When running Docker containers, it is a good idea to specify the digest. Specifying a digest takes precedent over the tag, and unless verified at some other stage, the tag may not and my never have correlated with a running container’s base image.