147 lines
7.9 KiB
Markdown
147 lines
7.9 KiB
Markdown
# Magic Nix Cache
|
|
|
|
> [!WARNING]
|
|
> The [Magic Nix Cache will will stop working](https://determinate.systems/posts/magic-nix-cache-free-tier-eol) on **February 1st, 2025** unless you're on [GitHub Enterprise Server](https://github.com/enterprise).
|
|
>
|
|
> You can upgrade to [FlakeHub Cache](https://flakehub.com/cache) and get **one month free** using the coupon code **`FHC`**.
|
|
>
|
|
> For more information, read [this blog post](https://determinate.systems/posts/magic-nix-cache-free-tier-eol/).
|
|
|
|
Save 30-50%+ of CI time without any effort or cost.
|
|
Use Magic Nix Cache, a totally free and zero-configuration binary cache for Nix on GitHub Actions.
|
|
|
|
Add our [GitHub Action][action] after installing Nix, in your workflow, like this:
|
|
|
|
```yaml
|
|
permissions:
|
|
contents: read
|
|
id-token: write
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: DeterminateSystems/nix-installer-action@main
|
|
- uses: DeterminateSystems/magic-nix-cache-action@main
|
|
- run: nix flake check
|
|
```
|
|
|
|
See [Usage](#usage) for a detailed example.
|
|
|
|
## Why use the Magic Nix Cache?
|
|
|
|
Magic Nix Cache uses the GitHub Actions [built-in cache][ghacache] to share builds between Workflow runs, and has many advantages over alternatives.
|
|
|
|
1. Totally free: backed by GitHub Actions' cache, there is no additional service to pay for.
|
|
1. Zero configuration: add our action to your workflow.
|
|
That's it.
|
|
Everything built in your workflow will be cached.
|
|
1. No secrets: Forks and pull requests benefit from the cache, too.
|
|
1. Secure: Magic Nix Cache follows the [same semantics as the GitHub Actions cache][semantics], and malicious pull requests cannot pollute your project.
|
|
1. Private: The cache is stored in the GitHub Actions cache, not with an additional third party.
|
|
|
|
> **Note:** the Magic Nix Cache doesn't offer a publicly available cache.
|
|
> This means the cache is only usable in CI.
|
|
> [Zero to Nix][z2n] has an article on binary caching if you want to [share Nix builds][z2ncache] with users outside of CI.
|
|
|
|
## Usage
|
|
|
|
Add it to your Linux and macOS GitHub Actions workflows, like this:
|
|
|
|
```yaml
|
|
name: CI
|
|
|
|
on:
|
|
push:
|
|
pull_request:
|
|
|
|
jobs:
|
|
check:
|
|
runs-on: ubuntu-22.04
|
|
permissions:
|
|
contents: read
|
|
id-token: write
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: DeterminateSystems/nix-installer-action@main
|
|
- uses: DeterminateSystems/magic-nix-cache-action@main
|
|
- run: nix flake check
|
|
```
|
|
|
|
That's it.
|
|
Everything built in your workflow will be cached.
|
|
|
|
## Usage Notes
|
|
|
|
The GitHub Actions Cache has a rate limit on reads and writes.
|
|
Occasionally, large projects or large rebuilds may exceed those rate-limits, and you'll see evidence of that in your logs.
|
|
The error looks like this:
|
|
|
|
```
|
|
error: unable to download 'http://127.0.0.1:37515/<...>': HTTP error 418
|
|
response body:
|
|
GitHub API error: API error (429 Too Many Requests): StructuredApiError { message: "Request was blocked due to exceeding usage of resource 'Count' in namespace ''." }
|
|
```
|
|
|
|
The caching daemon and Nix both handle this gracefully, and won't cause your CI to fail.
|
|
When the rate limit is exceeded while pulling dependencies, your workflow may perform more builds than usual.
|
|
When the rate limit is exceeded while uploading to the cache, the remainder of those store paths will be uploaded on the next run of the workflow.
|
|
|
|
## Development
|
|
|
|
This project depends on the GitHub Actions Cache API.
|
|
For local development, see `gha-cache/README.md` for more details on how to obtain the required tokens.
|
|
|
|
```shell
|
|
cargo run -- -c creds.json --upstream https://cache.nixos.org
|
|
cargo build --release --target x86_64-unknown-linux-gnu
|
|
cargo build --release --target aarch64-unknown-linux-gnu
|
|
nix copy --to 'http://127.0.0.1:3000' $(which bash)
|
|
nix-store --store $PWD/test-root --extra-substituters 'http://localhost:3000' --option require-sigs false -r $(which bash)
|
|
```
|
|
|
|
## Acknowledgement
|
|
|
|
Magic Nix Cache is a collaboration with [Zhaofeng Li][zhaofeng].
|
|
Zhaofeng is a major contributor to the Nix community, having authored [Attic][attic] and [Colmena][colmena].
|
|
We'd like to express our deep gratitude to Zhaofeng for his tremendous work on this project.
|
|
|
|
## Telemetry
|
|
|
|
The goal of Magic Nix Cache is to help teams save time in CI.
|
|
The cache daemon collects a little bit of telemetry information to help us make that true.
|
|
|
|
Here is a table of the [telemetry data we collect][telemetry]:
|
|
|
|
| Field | Use |
|
|
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
|
| `distinct_id` | An opaque string that represents your project, anonymized by sha256 hashing repository and organization details. |
|
|
| `version` | The version of Magic Nix Cache. |
|
|
| `is_ci` | Whether the Magic Nix Cache is being used in CI (i.e.: GitHub Actions). |
|
|
| `elapsed_seconds` | How long the cache daemon was running. |
|
|
| `narinfos_served` | Number of narinfos served from the cache daemon. |
|
|
| `narinfos_sent_upstream` | Number of narinfo requests forwarded to the upstream cache. |
|
|
| `narinfos_negative_cache_hits` | Effectiveness of an internal data structure which minimizes cache requests. |
|
|
| `narinfos_negative_cache_misses` | Effectiveness of an internal data structure which minimizes cache requests. |
|
|
| `narinfos_uploaded` | Number of new narinfo files cached during this run. |
|
|
| `nars_served` | Number of nars served from the cache daemon. |
|
|
| `nars_sent_upstream` | Number of nar requests forwarded to the upstream cache. |
|
|
| `nars_uploaded` | Number of nars uploaded during this run. |
|
|
| `num_original_paths` | Number of store paths that existed on startup. |
|
|
| `num_final_paths` | Number of store paths that existed on shutdown. |
|
|
| `num_new_paths` | The difference between `num_original_paths` and `num_final_paths`. |
|
|
|
|
To disable diagnostic reporting, set the diagnostics URL to an empty string by passing `--diagnostic-endpoint=""`.
|
|
|
|
You can read the full privacy policy for [Determinate Systems][detsys], the creators of this tool and the [Determinate Nix Installer][installer], [here][privacy].
|
|
|
|
[detsys]: https://determinate.systems/
|
|
[action]: https://github.com/DeterminateSystems/magic-nix-cache-action/
|
|
[installer]: https://github.com/DeterminateSystems/nix-installer/
|
|
[ghacache]: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
|
|
[privacy]: https://determinate.systems/policies/privacy
|
|
[telemetry]: https://github.com/DeterminateSystems/magic-nix-cache/blob/main/magic-nix-cache/src/telemetry.rs
|
|
[semantics]: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache
|
|
[z2ncache]: https://zero-to-nix.com/concepts/caching#binary-caches
|
|
[zhaofeng]: https://github.com/zhaofengli/
|
|
[attic]: https://github.com/zhaofengli/attic
|
|
[colmena]: https://github.com/zhaofengli/colmena
|
|
[z2n]: https://zero-to-nix.com
|