Angular CLI cache has different results in CI and Local

I have enabled cache in .angular.json:

"cli": {
  "cache": {
    "enabled": true,
    "environment": "all"
  }
}

And cache the .angular folder in GitHub actions:

steps:
  - name: Cache Angular build
    uses: actions/cache@v3
    id: cache-build
    env:
      cache-name: cache-build
    with:
      path: .angular
      key: ${{ runner.OS }}-build${{ env.NODE_VERSION }}-${{ hashFiles('package-lock.json') }}

  - name: Create Build
    shell: bash
    run: |
      ls -la .angular/cache
      npm run build ${{ inputs.product }} -- --configuration=${{ inputs.angularConfig }} --output-path=${{ inputs.buildPath }}

But the build takes around 6 mins. Without cache it was around 7 mins. Whereas locally it'll take less than 1 min to build locally with cache and 6 mins without cache.

I tried multiple builds like this:

  - name: Create Build
    shell: bash
    run: |
      npm run build ${{ inputs.product }} -- --configuration=${{ inputs.angularConfig }} --output-path=${{ inputs.buildPath }}

  - name: Create Build 2
    shell: bash
    run: |
      npm run build ${{ inputs.product }} -- --configuration=${{ inputs.angularConfig }} --output-path=${{ inputs.buildPath }}

And the second build is around 1 minute! So I assume something is wrong. Maybe there's an Angular CLI cache in ~/ folder or somewhere else. Any ideas?

I'm expecting relatively same time for local and CI builds.