0

Given a repository containing a Dockerfile that defines the build environment used by the CI pipeline as well as by the developer (e.g. as a Visual Studio Code devcontainer), the CI pipeline shall fulfill the following requirements:

  1. The CI pipeline shall build the repository's contents in the context of a docker image built from the very Dockerfile present in the working branch (or the result of its merge to the target branch where applicable in the context of a pull request).
  • Rationale: Builds shall be deterministic so that old repository versions shall be built with their original docker image, not with the newest one.
  • Rationale: Changes to the Dockerfile shall be considered automatically without the need for manual user intervention (i.e. local docker build followed by a manual push).
  1. The CI pipeline shall reuse an already built docker image if the applicable Dockerfile has already been "baked" into such an image pushed to the registry.
  • Rationale: Building a docker image on each commit is resource-consuming (time, energy).

I couldn't find an existing best practice to implement that out of the box (my CI environment being Azure DevOps Pipelines if it matters) so I came with the following concept:

  • Calculate the Dockerfile's hash.
  • Load docker_image_name:$hash from the docker registry.
  • If the load fails, build docker_image_name:$hash from the Dockerfile and push it to the docker registry.
  • Use docker_image_name:$hash (from the registry / from the local cache) to run the CI pipeline (using Azure's container jobs in my case).

Questions:

  1. Does this procedure make sense as a solution to my use case?
  2. I can't imagine being the first to realize this use case. Is there an existing mechanism (as part of the docker utilities, as part of the Azure DevOps Pipelines framework or from something completely different) that fulfills my needs?
Improve this question
3
  • I wasn't exactly sure if you was asking how to use a vscode dev container in a azure pipeline or if you was asking how to replicate that dev container functionality from scratch. Commented Aug 2, 2022 at 5:28
  • @Levi: My mentioning vscode's devcontainers might have been misleading since it was just part of the context, but not part of the actual question. The question was specifically about how to use the very docker image specified by the currently valid Dockerfile without building the image on each and every commit. Commented Aug 2, 2022 at 6:53
  • I think the question boils down to "How do I avoid rebuilding my container image in CICD" which probably has been asked and answered before. I went ahead and gave a new answer though. Commented Aug 2, 2022 at 13:13

2 Answers 2

Reset to default
0

Does this procedure make sense as a solution to my use case?

vscode devcontainers are one of the greatest inventions in the DevOps space and I can't believe more people are not talking about them. I think having the local dev environment and the pipeline in sync makes a lot of sense. Though I think the procedure you outlined above could be simplified.

I can't imagine being the first to realize this use case. Is there an existing mechanism (as part of the docker utilities, as part of the Azure DevOps Pipelines framework or from something completely different) that fulfills my needs?

I'm currently doing this in some of my side projects. It is also supported on azure.

Improve this answer
1
  • As mentioned above my questions are about using the Dockerfile hash to tag the docker image to get a commit-specific build environment without building the docker image on each and every commit. Commented Aug 2, 2022 at 6:58
0

Docker has this capability built-in and you can use it in a variety of ways.

One way would be to keep a "cache" tag for every build. This is good if your repo normally doesn't have a lot of parallel work being done.

docker pull ${MY_IMAGE}:cache || true
docker build --cache-from ${MY_IMAGE}:cache -t ${MY_IMAGE}:${VERSION} -t ${MY_IMAGE}:cache .
docker push ${MY_IMAGE}:${VERSION}
docker push ${MY_IMAGE}:cache

If your repo does have a lot of parallel work being done, don't create a cache tag, instead use the latest version.

docker pull ${MY_IMAGE}:latest || true
docker build --cache-from ${MY_IMAGE}:latest -t ${MY_IMAGE}:${VERSION} .
docker push ${MY_IMAGE}:${VERSION}

This is basically the following the process you outlined in your question but instead, we are relying on docker to do the work for us. The first step is to pull the image layers down locally. This is faster (usually) than building a new image and if your CI/CD is setup correctly should already be present for the use of caching.

Then the --cache-from flag tells Docker to use that image as cache. Which will prevent Docker from building any layers that haven't changed.

Improve this answer
4
  • You would then put the Dockerfile hash into $VERSION, correct? I fail at understanding how old builds would be able to use the old image (i.e. the very same image that used to be the current one at that time) and not the current latest one. Commented Aug 4, 2022 at 9:25
  • I guess it's a bad example because in my pipeline I'm building and deploying an application using semantic versioning. But it doesn't matter what you label the new image as, you can even remove the version label. The magic is in --cache-from. Cache from makes sure that docker doesn't build image layers that didnt chage. Commented Aug 4, 2022 at 18:14
  • docs.docker.com/engine/reference/commandline/build/… seems to require --build-arg BUILDKIT_INLINE_CACHE=1 to be set when building the initial image. Commented Aug 5, 2022 at 9:42
  • Why dont you just try it @RolandSarrazin ? This command works for me and I am not using build args and passing BUILDKIT_INLINE_CACHE in my image. I think that is required for external cache sources and not for local caches. Commented Aug 5, 2022 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.