Disable cache for specific RUN commands

Question

I have a few RUN commands in my Dockerfile that I would like to run with -no-cache each time I build a Docker image.

I understand the docker build --no-cache will disable caching for the entire Dockerfile.

Is it possible to disable cache for a specific RUN command?

This can be useful during debugging.

For example, if a Dockerfile fails on a particular line, it may be useful to run all previous lines up to a certain point, while maintaining the cache of some of those previous lines.

The reason being some Docker commands can take a long time (perhaps several minutes?) to run.

Once you disable the cache for a single command, if the result doesn't match past cached run, you'd need to rebuild all remaining steps. Is that your goal, or do you hope to only rebuild a single layer and somehow inject that into where prior cached data was stored? — BMitch
– BMitch, Commented Jun 13, 2016 at 20:35
I was hoping to rebuild specific layers, for example a "git pull" command. Right now the "git pull" command will be cached, even though the repo is updated. — Vingtoft
– Vingtoft, Commented Jun 14, 2016 at 7:31
It's easy enough to force a pull by passing an unused argument. But the result of that cached entry being rebuilt is that all following layers will need a rebuild. See my answer over here for an example. — BMitch
– BMitch, Commented Jun 14, 2016 at 11:56
If looking to invalidate the cache when a git remote has changed take a look at: How to prevent Dockerfile caching git clone. All credit to @anq for the linked answer. — hpgmiskin
– hpgmiskin, Commented Sep 9, 2020 at 8:06

Pang · Accepted Answer · 2020-06-04 00:39:28Z

195

There's always an option to insert some meaningless and cheap-to-run command before the region you want to disable cache for.

As proposed in this issue comment, one can add a build argument block (name can be arbitrary):

ARG CACHEBUST=1

before such region, and modify its value each run by adding --build-arg CACHEBUST=$(date +%s) as a docker build argument (value can also be arbitrary, here it is current datetime, to ensure its uniqueness across runs).

This will, of course, disable cache for all following blocks too, as hash sum of the intermediate image will be different, which makes truly selective cache disabling a non-trivial problem, taking into account how docker currently works.

edited Jun 4, 2020 at 0:39

Pang

10.2k146 gold badges87 silver badges126 bronze badges

answered Apr 11, 2018 at 10:26

Vladislav

2,0691 gold badge12 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Pylinux Over a year ago

Doesn't seem to work anymore, just got ---> Using cache under my ``ARG CACHEBUST=1` line... (and yes I did do --build-arg CACHEBUST=$(date +%s) in my docker command)

Oliver Over a year ago

Does not work for me either, maybe it is platform dependent. I would have expected any ARG change to invalidate the cache.

Sidharth V Over a year ago

You have to add RUN echo "$CACHEBUST" as just using ARG will not invalidate the cache

Hi-Angel Over a year ago

Confirm working as of Podman 3.2.2 and Docker 20.10.6. Side note: please do not assign the default value as in ARG CACHEBUST=1, instead just write ARG CACHEBUST. The difference is that in later case, may you forget to pass the --build-arg, you'll get a warning WARN[0000] missing "CACHEBUST" build argument. […] At least Podman behaves this way.

B. Baron Over a year ago

It is also possible to define a label instead of the echo, i.e. LABEL cachebust=$CACHEBUST. Better choices could be of course buildDate or buildNumber or something else providing valuable meta data.

|

Slava Fomin II · Accepted Answer · 2019-11-18 21:28:54Z

92

Use

ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache

before the RUN line you want to always run. This works because ADD will always fetch the file/URL and the above URL generates random data on each request, Docker then compares the result to see if it can use the cache.

I have also tested this and works nicely since it does not require any additional Docker command line arguments and also works from a Docker-compose.yaml file :)

edited Nov 18, 2019 at 21:28

Slava Fomin II

29k36 gold badges136 silver badges208 bronze badges

answered Nov 11, 2019 at 12:19

steve

3,4941 gold badge21 silver badges15 bronze badges

15 Comments

Andre Leon Rangel Over a year ago

whats going to happen if random.org decides to change that endpoint? how would you control that behaviour?

Kathi Over a year ago

This failed some times... when site is down!!! I think it's not perfect solution for this. ADD failed: failed to GET random.org/cgi-bin/randbyte?nbytes=10&format=h with status 503 Service Unavailable: <!DOCTYPE HTML>

Brad Root Over a year ago

random.org has added DDOS protection which breaks this solution now

Filipe Pina Over a year ago

not sure why noone highlighted yet, but this does add a layer with a useless skipcache file. using ARG CACHEBUST + RUN echo $CACHEBUST does not

jaksco Over a year ago

why don't people use a local source of randomness like /dev/urandom ???

|

Guillaume Boudreau · Accepted Answer · 2021-01-17 14:56:43Z

43

If your goal is to include the latest code from Github (or similar), one can use the Github API (or equivalent) to fetch information about the latest commit using an ADD command.
docker build will always fetch an URL from an ADD command, and if the response is different from the one received last time docker build ran, it will not use the subsequent cached layers.

eg.

ADD "https://api.github.com/repos/username/repo_name/commits?per_page=1" latest_commit
RUN curl -sLO "https://github.com/username/repo_name/archive/main.zip" && unzip main.zip

answered Jan 17, 2021 at 14:56

Guillaume Boudreau

2,93732 silver badges28 bronze badges

3 Comments

n13 Over a year ago

This is great. When the code changed, my git clone and tests run; when the code has not changed, cached values are used. Perfection.

David Parks Over a year ago

This should be the accepted solution, very clean and simple, just include the ADD step before whatever you currently do to install the package. Please upvote this answer to raise attention to it.

SiburNY Over a year ago

One note to add is that it works only for public repositories.

Jens · Accepted Answer · 2022-01-21 05:06:57Z

26

Building on @Vladislav’s solution above I used in my Dockerfile

ARG CACHEBUST=0

to invalidate the build cache from hereon.

However, instead of passing a date or some other random value, I call

docker build --build-arg CACHEBUST=`git rev-parse ${GITHUB_REF}` ...

where GITHUB_REF is a branch name (e.g. main) whose latest commit hash is used. That means that docker’s build cache is being invalidated only if the branch from which I build the image has had commits since the last run of docker build.

answered Jan 21, 2022 at 5:06

Jens

9,2789 gold badges65 silver badges84 bronze badges

Comments

Vingtoft · Accepted Answer · 2016-02-02 13:25:41Z

11

As of February 2016 it is not possible.

The feature has been requested at GitHub

answered Feb 2, 2016 at 13:25

Vingtoft

14.8k26 gold badges92 silver badges146 bronze badges

Comments

user2915097 · Accepted Answer · 2016-02-01 16:26:50Z

10

Not directly but you can divide your Dockerfile in several parts, build an image, then FROM thisimage at the beginning of the next Dockerfile, and build the image with or without caching

answered Feb 1, 2016 at 16:26

user2915097

32.4k6 gold badges61 silver badges62 bronze badges

1 Comment

user_mda Over a year ago

Will this enable updating the commited layers in the base docker image?

Tha Sami · Accepted Answer · 2019-11-11 11:16:55Z

10

the feature added a week ago.

ARG FOO=bar

FROM something
RUN echo "this won't be affected if the value of FOO changes"
ARG FOO
RUN echo "this step will be executed again if the value of FOO changes"

FROM something-else
RUN echo "this won't be affected because this stage doesn't use the FOO build-arg"

https://github.com/moby/moby/issues/1996#issuecomment-550020843

answered Nov 11, 2019 at 11:16

Tha Sami

1171 silver badge2 bronze badges

Comments

Mike Sadler · Accepted Answer · 2020-09-29 12:47:50Z

4

I believe that this is a slight improvement on @steve's answer, above:

RUN git clone https://sdk.ghwl;erjnv;wekrv;[email protected]/your_name/your_repository.git

WORKDIR your_repository

# Calls for a random number to break the cahing of the git clone
# (https://stackoverflow.com/questions/35134713/disable-cache-for-specific-run-commands/58801213#58801213)
ADD "https://www.random.org/cgi-bin/randbyte?nbytes=10&format=h" skipcache
RUN git pull

This uses the Docker cache of the git clone, but then runs an uncached update of the repository.

It appears to work, and it is faster - but many thanks to @steve for providing the underlying principles.

answered Sep 29, 2020 at 12:47

Mike Sadler

1,8482 gold badges24 silver badges45 bronze badges

1 Comment

sehe Over a year ago

Nice. Keep in mind this breaks when there are rewrites

Strator · Accepted Answer · 2024-07-22 11:11:37Z

3

Divide your dockerfile in stage and put the RUN command under a specific stage. When you run the docker build command, add "--no-cache-filter " change with

answered Jul 22, 2024 at 11:11

Strator

311 bronze badge

Comments

kta · Accepted Answer · 2024-11-19 01:12:54Z

1

Use --no-cache-filter to disable docker cache while running docker build for each target. For example,

docker build --no-cache-filter install --no-cache-filter rebuild .

one of my dockerfile has the following lines:

RUN yarn install --no-cache --network-timeout 1000000 && echo "installed package"
RUN npm rebuild node-sass && echo "rebuild node-sass"

also look into their offical docs for more ways to invalidate docker cache.

edited Nov 19, 2024 at 1:12

answered Nov 19, 2024 at 0:57

kta

20.3k7 gold badges67 silver badges48 bronze badges

Comments

Ryan Wheale · Accepted Answer · 2024-06-18 04:17:50Z

I really hope this helps someone who reads this far down. I was having the opposite problem - RUN directives were not being cached when I really needed them to. The problem was that I had inadvertently implemented the solution explained above, but couldn't see it.

For our monorepo, we were using a build arg (SCOPE) to specify which project we wanted to build. We like to keep all of our ARG directives near the top. However, was invalidating everything below it... and it was very difficult to see. Let me show by code:

FROM ...
ARG SCOPE
# the next line was not getting cached
RUN echo "hello" 
RUN echo "${SCOPE}"

But making this subtle change worked:

FROM ...
# now it gets cached
RUN echo "hello"
ARG SCOPE
RUN echo "${SCOPE}"

Mark · Accepted Answer · 2019-04-19 02:26:37Z

-7

Another quick hack is to write some random bytes before your command

RUN head -c 5 /dev/random > random_bytes && <run your command>

writes out 5 random bytes which will force a cache miss

answered Apr 19, 2019 at 2:26

Mark

1,47012 silver badges14 bronze badges

1 Comment

Icy Defiance Over a year ago

The result of writing those random bytes gets cached as well, so if no files have changed before that command, it won't run the command again. This doesn't solve anything.

Collectives™ on Stack Overflow

Disable cache for specific RUN commands

12 Answers 12

8 Comments

15 Comments

3 Comments

Comments

Comments

1 Comment

Comments

1 Comment

Comments

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

12 Answers 12

8 Comments

15 Comments

3 Comments

Comments

Comments

1 Comment

Comments

1 Comment

Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related