cafkafk/nixpkgs
cafkafk/nixpkgs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

Merge pull request #119942 from hercules-ci/mkDerivation-overridable-recursive-attributes

Browse source
This commit is contained in:
Artturi 2022-05-04 14:31:50 +03:00 committed by GitHub
commit f0e48dd98b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 220 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

34
doc/stdenv/meta.chapter.md
View file

@ -175,6 +175,40 @@ The NixOS tests are available as `nixosTests` in parameters of derivations. For
NixOS tests run in a VM, so they are slower than regular package tests. For more information see [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests).
Alternatively, you can specify other derivations as tests. You can make use of
the optional parameter to inject the correct package without
relying on non-local definitions, even in the presence of `overrideAttrs`.
Here that's `finalAttrs.finalPackage`, but you could choose a different name if
`finalAttrs` already exists in your scope.
`(mypkg.overrideAttrs f).passthru.tests` will be as expected, as long as the
definition of `tests` does not rely on the original `mypkg` or overrides it in
all places.
```nix
# my-package/default.nix
{ stdenv, callPackage }:
stdenv.mkDerivation (finalAttrs: {
# ...
passthru.tests.example = callPackage ./example.nix { my-package = finalAttrs.finalPackage; };
})
```
```nix
# my-package/example.nix
{ runCommand, lib, my-package, ... }:
runCommand "my-package-test" {
nativeBuildInputs = [ my-package ];
src = lib.sources.sourcesByRegex ./. [ ".*.in" ".*.expected" ];
} ''
my-package --help
my-package <example.in >example.actual
diff -U3 --color=auto example.expected example.actual
mkdir $out
''
```
### `timeout` {#var-meta-timeout}
A timeout (in seconds) for building the derivation. If the derivation takes longer than this time to build, it can fail due to breaking the timeout. However, all computers do not have the same computing power, hence some builders may decide to apply a multiplicative factor to this value. When filling this value in, try to keep it approximately consistent with other values already present in `nixpkgs`.

54
doc/stdenv/stdenv.chapter.md
View file

@ -317,6 +317,60 @@ The script will be usually run from the root of the Nixpkgs repository but you s
For information about how to run the updates, execute `nix-shell maintainers/scripts/update.nix`.
### Recursive attributes in `mkDerivation`
If you pass a function to `mkDerivation`, it will receive as its argument the final arguments, including the overrides when reinvoked via `overrideAttrs`. For example:
```nix
mkDerivation (finalAttrs: {
pname = "hello";
withFeature = true;
configureFlags =
lib.optionals finalAttrs.withFeature ["--with-feature"];
})
```
Note that this does not use the `rec` keyword to reuse `withFeature` in `configureFlags`.
The `rec` keyword works at the syntax level and is unaware of overriding.
Instead, the definition references `finalAttrs`, allowing users to change `withFeature`
consistently with `overrideAttrs`.
`finalAttrs` also contains the attribute `finalPackage`, which includes the output paths, etc.
Let's look at a more elaborate example to understand the differences between
various bindings:
```nix
# `pkg` is the _original_ definition (for illustration purposes)
let pkg =
mkDerivation (finalAttrs: {
# ...
# An example attribute
packages = [];
# `passthru.tests` is a commonly defined attribute.
passthru.tests.simple = f finalAttrs.finalPackage;
# An example of an attribute containing a function
passthru.appendPackages = packages':
finalAttrs.finalPackage.overrideAttrs (newSelf: super: {
packages = super.packages ++ packages';
});
# For illustration purposes; referenced as
# `(pkg.overrideAttrs(x)).finalAttrs` etc in the text below.
passthru.finalAttrs = finalAttrs;
passthru.original = pkg;
});
in pkg
```
Unlike the `pkg` binding in the above example, the `finalAttrs` parameter always references the final attributes. For instance `(pkg.overrideAttrs(x)).finalAttrs.finalPackage` is identical to `pkg.overrideAttrs(x)`, whereas `(pkg.overrideAttrs(x)).original` is the same as the original `pkg`.
See also the section about [`passthru.tests`](#var-meta-tests).
## Phases {#sec-stdenv-phases}
`stdenv.mkDerivation` sets the Nix [derivation](https://nixos.org/manual/nix/stable/expressions/derivations.html#derivations)'s builder to a script that loads the stdenv `setup.sh` bash library and calls `genericBuild`. Most packaging functions rely on this default builder.

8
doc/using/overrides.chapter.md
View file

@ -39,14 +39,18 @@ The function `overrideAttrs` allows overriding the attribute set passed to a `st
Example usage:
```nix
helloWithDebug = pkgs.hello.overrideAttrs (oldAttrs: rec {
helloWithDebug = pkgs.hello.overrideAttrs (finalAttrs: previousAttrs: {
separateDebugInfo = true;
});
```
In the above example, the `separateDebugInfo` attribute is overridden to be true, thus building debug info for `helloWithDebug`, while all other attributes will be retained from the original `hello` package.
The argument `oldAttrs` is conventionally used to refer to the attr set originally passed to `stdenv.mkDerivation`.
The argument `previousAttrs` is conventionally used to refer to the attr set originally passed to `stdenv.mkDerivation`.
The argument `finalAttrs` refers to the final attributes passed to `mkDerivation`, plus the `finalPackage` attribute which is equal to the result of `mkDerivation` or subsequent `overrideAttrs` calls.
If only a one-argument function is written, the argument has the meaning of `previousAttrs`.
::: {.note}
Note that `separateDebugInfo` is processed only by the `stdenv.mkDerivation` function, not the generated, raw Nix derivation. Thus, using `overrideDerivation` will not work in this case, as it overrides only the attributes of the final derivation. It is for this reason that `overrideAttrs` should be preferred in (almost) all cases to `overrideDerivation`, i.e. to allow using `stdenv.mkDerivation` to process input arguments, as well as the fact that it is easier to use (you can use the same attribute names you see in your Nix code, instead of the ones generated (e.g. `buildInputs` vs `nativeBuildInputs`), and it involves less typing).

27
nixos/doc/manual/from_md/release-notes/rl-2205.section.xml
View file

@ -43,6 +43,33 @@
Shell.
</para>
</listitem>
<listitem>
<para>
<literal>stdenv.mkDerivation</literal> now supports a
self-referencing <literal>finalAttrs:</literal> parameter
containing the final <literal>mkDerivation</literal> arguments
including overrides. <literal>drv.overrideAttrs</literal> now
supports two parameters
<literal>finalAttrs: previousAttrs:</literal>. This allows
packaging configuration to be overridden in a consistent
manner by providing an alternative to
<literal>rec {}</literal> syntax.
</para>
<para>
Additionally, <literal>passthru</literal> can now reference
<literal>finalAttrs.finalPackage</literal> containing the
final package, including attributes such as the output paths
and <literal>overrideAttrs</literal>.
</para>
<para>
New language integrations can be simplified by overriding a
<quote>prototype</quote> package containing the
language-specific logic. This removes the need for a extra
layer of overriding for the <quote>generic builder</quote>
arguments, thus removing a usability problem and source of
error.
</para>
</listitem>
<listitem>
<para>
PHP 8.1 is now available

15
nixos/doc/manual/release-notes/rl-2205.section.md
View file

@ -17,6 +17,21 @@ In addition to numerous new and upgraded packages, this release has the followin
- GNOME has been upgraded to 42. Please take a look at their [Release Notes](https://release.gnome.org/42/) for details. Notably, it replaces gedit with GNOME Text Editor, GNOME Terminal with GNOME Console (formerly King’s Cross), and GNOME Screenshot with a tool built into the Shell.
- `stdenv.mkDerivation` now supports a self-referencing `finalAttrs:` parameter
containing the final `mkDerivation` arguments including overrides.
`drv.overrideAttrs` now supports two parameters `finalAttrs: previousAttrs:`.
This allows packaging configuration to be overridden in a consistent manner by
providing an alternative to `rec {}` syntax.
Additionally, `passthru` can now reference `finalAttrs.finalPackage` containing
the final package, including attributes such as the output paths and
`overrideAttrs`.
New language integrations can be simplified by overriding a "prototype"
package containing the language-specific logic. This removes the need for a
extra layer of overriding for the "generic builder" arguments, thus removing a
usability problem and source of error.
- PHP 8.1 is now available
- Mattermost has been updated to extended support release 6.3, as the previously packaged extended support release 5.37 is [reaching its end of life](https://docs.mattermost.com/upgrade/extended-support-release.html).

13
pkgs/applications/misc/hello/default.nix
View file

@ -1,4 +1,5 @@
{ lib
{ callPackage
, lib
, stdenv
, fetchurl
, nixos
@ -6,12 +7,12 @@
, hello
}:
stdenv.mkDerivation rec {
stdenv.mkDerivation (finalAttrs: {
pname = "hello";
version = "2.12";
src = fetchurl {
url = "mirror://gnu/hello/${pname}-${version}.tar.gz";
url = "mirror://gnu/hello/hello-${finalAttrs.version}.tar.gz";
sha256 = "1ayhp9v4m4rdhjmnl2bq3cibrbqqkgjbl3s7yk2nhlh8vj3ay16g";
};
@ -27,6 +28,8 @@ stdenv.mkDerivation rec {
(nixos { environment.noXlibs = true; }).pkgs.hello;
};
passthru.tests.run = callPackage ./test.nix { hello = finalAttrs.finalPackage; };
meta = with lib; {
description = "A program that produces a familiar, friendly greeting";
longDescription = ''
@ -34,9 +37,9 @@ stdenv.mkDerivation rec {
It is fully customizable.
'';
homepage = "https://www.gnu.org/software/hello/manual/";
changelog = "https://git.savannah.gnu.org/cgit/hello.git/plain/NEWS?h=v${version}";
changelog = "https://git.savannah.gnu.org/cgit/hello.git/plain/NEWS?h=v${finalAttrs.version}";
license = licenses.gpl3Plus;
maintainers = [ maintainers.eelco ];
platforms = platforms.all;
};
}
})

8
pkgs/applications/misc/hello/test.nix Normal file
View file

@ -0,0 +1,8 @@
{ runCommand, hello }:
runCommand "hello-test-run" {
nativeBuildInputs = [ hello ];
} ''
diff -U3 --color=auto <(hello) <(echo 'Hello, world!')
touch $out
''

71
pkgs/stdenv/generic/make-derivation.nix
View file

@ -9,8 +9,72 @@ let
# to build it. This is a bit confusing for cross compilation.
inherit (stdenv) hostPlatform;
};
makeOverlayable = mkDerivationSimple:
fnOrAttrs:
if builtins.isFunction fnOrAttrs
then makeDerivationExtensible mkDerivationSimple fnOrAttrs
else makeDerivationExtensibleConst mkDerivationSimple fnOrAttrs;
# Based off lib.makeExtensible, with modifications:
makeDerivationExtensible = mkDerivationSimple: rattrs:
let
# NOTE: The following is a hint that will be printed by the Nix cli when
# encountering an infinite recursion. It must not be formatted into
# separate lines, because Nix would only show the last line of the comment.
# An infinite recursion here can be caused by having the attribute names of expression `e` in `.overrideAttrs(finalAttrs: previousAttrs: e)` depend on `finalAttrs`. Only the attribute values of `e` can depend on `finalAttrs`.
args = rattrs (args // { inherit finalPackage; });
# ^^^^
finalPackage =
mkDerivationSimple
(f0:
let
f = self: super:
# Convert f0 to an overlay. Legacy is:
# overrideAttrs (super: {})
# We want to introduce self. We follow the convention of overlays:
# overrideAttrs (self: super: {})
# Which means the first parameter can be either self or super.
# This is surprising, but far better than the confusion that would
# arise from flipping an overlay's parameters in some cases.
let x = f0 super;
in
if builtins.isFunction x
then
# Can't reuse `x`, because `self` comes first.
# Looks inefficient, but `f0 super` was a cheap thunk.
f0 self super
else x;
in
makeDerivationExtensible mkDerivationSimple
(self: let super = rattrs self; in super // f self super))
args;
in finalPackage;
# makeDerivationExtensibleConst == makeDerivationExtensible (_: attrs),
# but pre-evaluated for a slight improvement in performance.
makeDerivationExtensibleConst = mkDerivationSimple: attrs:
mkDerivationSimple
(f0:
let
f = self: super:
let x = f0 super;
in
if builtins.isFunction x
then
f0 self super
else x;
in
makeDerivationExtensible mkDerivationSimple (self: attrs // f self attrs))
attrs;
in
makeOverlayable (overrideAttrs:
# `mkDerivation` wraps the builtin `derivation` function to
# produce derivations that use this stdenv and its shell.
#
@ -70,6 +134,7 @@ in
, # TODO(@Ericson2314): Make always true and remove
strictDeps ? if config.strictDepsByDefault then true else stdenv.hostPlatform != stdenv.buildPlatform
, meta ? {}
, passthru ? {}
, pos ? # position used in error messages and for meta.position
@ -381,8 +446,6 @@ in
lib.extendDerivation
validity.handled
({
overrideAttrs = f: stdenv.mkDerivation (attrs // (f attrs));
# A derivation that always builds successfully and whose runtime
# dependencies are the original derivations build time dependencies
# This allows easy building and distributing of all derivations
@ -408,10 +471,12 @@ lib.extendDerivation
args = [ "-c" "export > $out" ];
});
inherit meta passthru;
inherit meta passthru overrideAttrs;
} //
# Pass through extra attributes that are not inputs, but
# should be made available to Nix expressions using the
# derivation (e.g., in assertions).
passthru)
(derivation derivationArg)
)