How to cache downloaded go dependencies in a containerized custom script execution?

Question

One of my build steps is a custom script:

bash test/test.sh

The script itself sits in the source code and launches the testing:

cd test && go test github.com/geniot/blaze-test/src -json

I use run step within container golang:latest, I've also tried the Run in Docker feature.

But the problem is that this configuration launches the download of all go dependencies for each run. Which takes around 10-15 seconds (my guess). The whole step takes 23 seconds.

22:48:10 Step 4/6: Test Unstable (test.blaze.ad) (Command Line)
22:48:10   Running step within container golang:latest
22:48:10   Created container 60a19864abc1...
22:48:11   Starting: /bin/sh -c . /opt/buildagent/temp/agentTmp/docker-wrapper-env-export-13473230305602170975.sh && docker exec -w /opt/buildagent/work/a993f8719008f409 60a19864abc17f5fb5927176fb150f5f7f85559b599e14d8b657bc2e15109d78 /bin/sh -c /opt/buildagent/temp/agentTmp/docker-shell-script-1481671475381782727.sh
22:48:11   in directory: /opt/buildagent/work/a993f8719008f409
22:48:11   go: downloading github.com/sirupsen/logrus v1.9.3
22:48:11   go: downloading github.com/golang-jwt/jwt/v5 v5.3.0
22:48:11   go: downloading github.com/num30/config v0.1.3
...

Is there a way to avoid this downloading? I've noticed that a simple docker build of a Go project uses cached results if the source code has not changed:

22:48:09   #21 [backend  5/10] RUN go mod download
22:48:09   #21 CACHED
22:48:09   #23 [backend 10/10] RUN go build -o /blaze github.com/geniot/blaze/src
22:48:09   #23 CACHED

The newer answers to Cache "go get" in docker build describe the relatively new Dockerfile RUN --mount=type=cache option, which will provide a persistent cache for individual files such as downloaded packages. Does adding a similar option to the RUN test/test.sh line help your setup? — David Maze
– David Maze, Commented Oct 20 at 11:04
The problem with docker builds that run tests inside while building an image is that the test results stay inside and are not reachable for TeamCity to analyze them. At least I haven't found so far how to get to them. Simply using -json doesn't expose the test results. — Vitaly Sazanovich
– Vitaly Sazanovich, Commented Oct 20 at 20:22
Your unit tests should produce the same result whether they're running in a container or not, and the unit-test result doesn't affect the final image content. Can you use a non-container Go toolchain to run these tests, before you build an image? — David Maze
– David Maze, Commented Oct 20 at 23:06
It's too complicated. First build an image with tests. Then run a container (with mounted volume for test results), run the tests inside the container. Then what? Somehow TC should know when to stop if there's a failure. Container can return an exit code. The test results also available after container removal. Possible but too complicated right now. — Vitaly Sazanovich
– Vitaly Sazanovich, Commented Oct 21 at 11:38
Right, that's my point: do all of that testing setup with an ordinary non-container toolchain. Don't build a container, just go test . and see if it works. If it passes, then docker build your image. — David Maze
– David Maze, Commented Oct 21 at 12:21

Vitaly Sazanovich · Accepted Answer · 2025-10-21 11:44:47Z

So I decided to run my integration tests while building a Docker image. This gives me caching when running with TeamCity. The build now takes 13 seconds (was 33).

Dockerfile:

FROM golang:latest AS test
ARG RESET_TOKEN="sometoken"
ENV RESET_TOKEN=${RESET_TOKEN}
WORKDIR /test
COPY go.mod .
COPY go.sum .
RUN go mod download
COPY . .
RUN BASE_API_URL=https://test.blaze.ad/api \
    BASE_UI_URL=https://test.blaze.ad \
    RESET_URL=https://reset.blaze.ad/reset \
    DATABASE_URL=postgres://blaze_test:blaze_test@blaze-test-pg:5432/blaze_test \
    RESET_TOKEN=${RESET_TOKEN} \
    go test github.com/geniot/blaze-test/src

There is a freshly deployed application at test.blaze.ad. The database is reset using another reset application which I wrote (I decided not to redeploy PG containers, instead I reset the db with raw sql like "delete from").

TeamCity doesn't see the test results of such image building. I scroll through the build log to find the failures.

There is an option to build an image with tests without actually running the tests. Just compiling them into an executable.

Then I can run tests in a temporary container and mount a host folder for test results. TeamCity should somehow detect those results and present them nicely.

Collectives™ on Stack Overflow

How to cache downloaded go dependencies in a containerized custom script execution?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related