Merge pull request #6652 from abathur/doc_install_tests

doc/manual/src/contributing/hacking.md

@@ -86,9 +86,7 @@ by:
 $ nix develop
 ```
 
-## Testing
+## Running tests
-
-Nix comes with three different flavors of tests: unit, functional and integration.
 
 ### Unit-tests
 
@@ -111,3 +109,72 @@ These tests include everything that needs to interact with external services or
 Because these tests are expensive and require more than what the standard github-actions setup provides, they only run on the master branch (on <https://hydra.nixos.org/jobset/nix/master>).
 
 You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix-build -A hydraJobs.tests.{testName}`
+
+### Installer tests
+
+After a one-time setup, the Nix repository's GitHub Actions continuous integration (CI) workflow can test the installer each time you push to a branch.
+
+Creating a Cachix cache for your installer tests and adding its authorization token to GitHub enables [two installer-specific jobs in the CI workflow](https://github.com/NixOS/nix/blob/88a45d6149c0e304f6eb2efcc2d7a4d0d569f8af/.github/workflows/ci.yml#L50-L91):
+
+- The `installer` job generates installers for the platforms below and uploads them to your Cachix cache:
+  - `x86_64-linux`
+  - `armv6l-linux`
+  - `armv7l-linux`
+  - `x86_64-darwin`
+
+- The `installer_test` job (which runs on `ubuntu-latest` and `macos-latest`) will try to install Nix with the cached installer and run a trivial Nix command.
+
+#### One-time setup
+
+1. Have a GitHub account with a fork of the [Nix repository](https://github.com/NixOS/nix).
+2. At cachix.org:
+    - Create or log in to an account.
+    - Create a Cachix cache using the format `<github-username>-nix-install-tests`.
+    - Navigate to the new cache > Settings > Auth Tokens.
+    - Generate a new Cachix auth token and copy the generated value.
+3. At github.com:
+    - Navigate to your Nix fork > Settings > Secrets > Actions > New repository secret.
+    - Name the secret `CACHIX_AUTH_TOKEN`.
+    - Paste the copied value of the Cachix cache auth token.
+
+#### Using the CI-generated installer for manual testing
+
+After the CI run completes, you can check the output to extract the installer URL:
+1. Click into the detailed view of the CI run.
+2. Click into any `installer_test` run (the URL you're here to extract will be the same in all of them).
+3. Click into the `Run cachix/install-nix-action@v...` step and click the detail triangle next to the first log line (it will also be `Run cachix/install-nix-action@v...`)
+4. Copy the value of `install_url`
+5. To generate an install command, plug this `install_url` and your GitHub username into this template:
+
+    ```console
+    sh <(curl -L <install_url>) --tarball-url-prefix https://<github-username>-nix-install-tests.cachix.org/serve
+    ```
+
+<!-- #### Manually generating test installers
+
+There's obviously a manual way to do this, and it's still the only way for
+platforms that lack GA runners.
+
+I did do this back in Fall 2020 (before the GA approach encouraged here). I'll
+sketch what I recall in case it encourages someone to fill in detail, but: I
+didn't know what I was doing at the time and had to fumble/ask around a lot--
+so I don't want to uphold any of it as "right". It may have been dumb or
+the _hard_ way from the getgo. Fundamentals may have changed since.
+
+Here's the build command I used to do this on and for x86_64-darwin:
+nix build --out-link /tmp/foo ".#checks.x86_64-darwin.binaryTarball"
+
+I used the stable out-link to make it easier to script the next steps:
+link=$(readlink /tmp/foo)
+cp $link/*-darwin.tar.xz ~/somewheres
+
+I've lost the last steps and am just going from memory:
+
+From here, I think I had to extract and modify the `install` script to point
+it at this tarball (which I scped to my own site, but it might make more sense
+to just share them locally). I extracted this script once and then just
+search/replaced in it for each new build.
+
+The installer now supports a `--tarball-url-prefix` flag which _may_ have
+solved this need?
+-->