8

Here I have two workflows under a job. The only target we want to achieve is that, we want to reuse the container images by using cache or some other means. Similar way we do for node_modules

jobs:
    build:
        name: build
        runs-on: [self-hosted, x64, linux, research]
        container:
          image: <sample docker image>
        env:
          NPM_AUTH_TOKEN: <sample token>
        steps:
          - uses: actions/checkout@v2
          - name: Install
            run: |
              npm install
          - name: Build
            run: |
          npm build
    Test:
        name: Test Lint
        runs-on: [self-hosted, x64, linux, research]
        container:
          image: <sample docker image>
        env:
          NPM_AUTH_TOKEN: <sample token>
        steps:
          - uses: actions/checkout@v2
          - name: Install Dependencies
            run: npm ci
          - name: Lint Check
            run: npm run lint
Improve this question
5
  • From what i understand with GitHub-Actions every run that's hosted is unique. So every "VM" is blank and couldn't have a cache of container images. Commented Mar 5, 2022 at 16:48
  • On the other hand your self-hosted runners could in theory have that cache of container images, helping speed up your build. Commented Mar 5, 2022 at 16:48
  • @JJAsghar Thanks for your reply. I understand every time its hosted is unique. But under a single job, here we have two different workflows with same images. So is there a way to address that? Commented Mar 6, 2022 at 11:34
  • Yeah, I’m pretty sure you can only do that with a self-hosted runner on a specific machine with a local cached container. It’s very roundabout and unless you have an airgapped environment it’s hard to understand the trade-off of the engineering than just leveraging typical containerization. Commented Mar 6, 2022 at 23:05
  • It's not a single job, it's two jobs in the same workflow. GHA Jobs don't share much data - they can require that one passes before another, that's pretty much it. Otherwise they can only share state by pushing it somewhere in one job and pulling it back in another. In this case your use of Docker is just to ask GHA to run your steps inside a container. You have no control over what happens to that container - its state is lost the moment the job ends, and because you are inside it you cannot snapshot it as an image and push it somewhere else. Commented Mar 8, 2022 at 9:04

2 Answers 2

Reset to default
8

I would suggest using the Docker's Build Push action for this purpose. Through the build-push-action, you can cache your container images by using the inline cache, registry cache or the experimental cache backend API:

Inline cache

name: Build and push
uses: docker/build-push-action@v2
with:
    context: .
    push: true
    tags: user/app:latest
    cache-from: type=registry,ref=user/app:latest
    cache-to: type=inline

Refer to the Buildkit docs.

Registry Cache

name: Build and push
uses: docker/build-push-action@v2
with:
    context: .
    push: true
    tags: user/app:latest
    cache-from: type=registry,ref=user/app:buildcache
    cache-to: type=registry,ref=user/app:buildcache,mode=max

Refer to Buildkit docs.

Cache backend API

name: Build and push
uses: docker/build-push-action@v2
with:
    context: .
    push: true
    tags: user/app:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

Refer to Buildkit docs.

I personally prefer using the Cache backend API as its easy to setup and provides a great boost in reducing the overall CI pipeline run duration.

By looking at the comments, it seems you want to share Docker cache between workflows. In this case you can share Docker containers between jobs in a workflow using this example:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      -
        name: Checkout
        uses: actions/checkout@v2
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Build and push
        uses: docker/build-push-action@v2
        with:
          context: .
          file: ./Dockerfile
          tags: myimage:latest
          outputs: type=docker,dest=/tmp/myimage.tar
      -
        name: Upload artifact
        uses: actions/upload-artifact@v2
        with:
          name: myimage
          path: /tmp/myimage.tar

  use:
    runs-on: ubuntu-latest
    needs: build
    steps:
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      -
        name: Download artifact
        uses: actions/download-artifact@v2
        with:
          name: myimage
          path: /tmp
      -
        name: Load Docker image
        run: |
          docker load --input /tmp/myimage.tar
          docker image ls -a
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

What if the Dockerfile is a multi-stage build, and I want to cache the first FROM image?
4

In general, data is not shared between jobs in GitHub Actions (GHA). jobs actually will run in parallel on distinct ephemeral VMs unless you explicitly create a dependency with needs

GHA does provide a cache mechanism. For package manager type caching, they simplified it, see here.

For docker images, you either can use docker buildx cache and cache to a remote registry (including ghcr), or use the GHA cache action, which probably is easier. The syntax for actions/cache is pretty straightforward and clear on the page. For buildx, documentation always has been a bit of an issue (largely, I think, because he people building it are so smart that they do not realize how much we do not understand what is in their hearts), so you would need to configure the cache action, and then buildx to cache it.

Alternatively, you could do docker save imagename > imagename.tar and use that in the cache. There is a decent example of that here. No idea who wrote it, but it does the job.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.