Compare commits

..

5 commits

Author SHA1 Message Date
Mel Zuser
d973508e17 bench: add benchmarks to show off string perf improvements
Change-Id: I6d032706e1e8b6fa39502276ac2018cb69de6ce5
2024-05-15 21:26:15 -07:00
Mel Zuser
be7a07d87a bench: allow passing benchmark names as args
Change-Id: Icf18ed526e64707dc6112710892a89570d54e34e
2024-05-15 21:23:28 -07:00
Mel Zuser
947c509110 libexpr: prefer Value::str() over Value::c_str()
at call sites that already cast to string_view to take advantage of the
stored length

Change-Id: I525c68e83e1c71bdc5ad2c3ed0bd0863ad20aaf2
2024-05-15 21:20:08 -07:00
Mel Zuser
a9f026f8b1 libexpr: store lazy string lengths
Splitup the internal rep of string to allow for storing string lengths
without hurting overall eval performance. If the length is not known at
the time the value is created, it will be cached after the first call to
str(). This fixes O(n) substring, O(n^2) stringToCharacters, etc.

Change-Id: I10a6f16544f6619b0726f77b617acd517ee87a00
2024-05-15 20:38:02 -07:00
Mel Zuser
1eff2adc93 libexpr: abstract string members in Value
to allow multiple internal reps similar to how lists works

Change-Id: I94139aab28b04b73b055a3dc05d496758ee72538
2024-05-15 20:03:19 -07:00
719 changed files with 14137 additions and 25131 deletions

View file

@ -16,20 +16,3 @@ Checks:
- -bugprone-unchecked-optional-access - -bugprone-unchecked-optional-access
# many warnings, seems like a questionable lint # many warnings, seems like a questionable lint
- -bugprone-branch-clone - -bugprone-branch-clone
# all thrown exceptions must derive from std::exception
- hicpp-exception-baseclass
# capturing async lambdas are dangerous
- cppcoreguidelines-avoid-capturing-lambda-coroutines
# crimes must be appropriately declared as crimes
- cppcoreguidelines-pro-type-cstyle-cast
- lix-*
# This can not yet be applied to Lix itself since we need to do source
# reorganization so that lix/ include paths work.
- -lix-fixincludes
# This lint is included as an example, but the lib function it replaces is
# already gone.
- -lix-hasprefixsuffix
CheckOptions:
bugprone-reserved-identifier.AllowedIdentifiers: '__asan_default_options'

View file

@ -24,8 +24,3 @@ indent_size = 4
# Match diffs, avoid to trim trailing whitespace # Match diffs, avoid to trim trailing whitespace
[*.{diff,patch}] [*.{diff,patch}]
trim_trailing_whitespace = false trim_trailing_whitespace = false
[*.md]
indent_style = space
indent_size = 2
max_line_length = 0

8
.envrc
View file

@ -1,5 +1,9 @@
# shellcheck shell=bash # shellcheck shell=bash
source_env_if_exists .envrc.local source_env_if_exists .envrc.local
# Use native-clangStdenvPackages to get clangd by default. # TODO: `use flake .#native-clangStdenvPackages` on macOS?
use flake ".#${LIX_SHELL_VARIANT:-native-clangStdenvPackages}" "${LIX_SHELL_EXTRA_ARGS[@]}" use flake ".#${LIX_SHELL_VARIANT:-default}" "${LIX_SHELL_EXTRA_ARGS[@]}"
export MAKEFLAGS="$MAKEFLAGS -e"
if [[ -n "$NIX_BUILD_CORES" ]]; then
export MAKEFLAGS="$MAKEFLAGS -j $NIX_BUILD_CORES"
fi
export GTEST_BRIEF=1 export GTEST_BRIEF=1

18
.github/CODEOWNERS vendored Normal file
View file

@ -0,0 +1,18 @@
# Pull requests concerning the listed files will automatically invite the respective maintainers as reviewers.
# This file is not used for denoting any kind of ownership, but is merely a tool for handling notifications.
#
# Merge permissions are required for maintaining an entry in this file.
# For documentation on this mechanism, see https://help.github.com/articles/about-codeowners/
# Default reviewers if nothing else matches
* @edolstra
# This file
.github/CODEOWNERS @edolstra
# Public documentation
/doc @fricklerhandwerk
*.md @fricklerhandwerk
# Libstore layer
/src/libstore @thufschmitt

View file

@ -9,7 +9,7 @@ assignees: ''
## Platform ## Platform
<!-- select the platform on which you tried to install Lix --> <!-- select the platform on which you tried to install Nix -->
- [ ] Linux: <!-- state your distribution, e.g. Arch Linux, Ubuntu, ... --> - [ ] Linux: <!-- state your distribution, e.g. Arch Linux, Ubuntu, ... -->
- [ ] macOS - [ ] macOS

View file

@ -19,10 +19,9 @@ assignees: ''
<!-- make sure this issue is not redundant or obsolete --> <!-- make sure this issue is not redundant or obsolete -->
- [ ] checked [latest Lix manual] \([source]\) - [ ] checked [latest Nix manual] \([source])
- [ ] checked [documentation issues] and [recent documentation changes] for possible duplicates - [ ] checked [open documentation issues and pull requests] for possible duplicates
[latest Nix manual]: https://docs.lix.systems/manual/lix/nightly [latest Nix manual]: https://nixos.org/manual/nix/unstable/
[source]: https://git.lix.systems/lix-project/lix/src/main/doc/manual/src [source]: https://github.com/NixOS/nix/tree/master/doc/manual/src
[documentation issues]: https://git.lix.systems/lix-project/lix/issues?labels=151&state=all [open documentation issues and pull requests]: https://github.com/NixOS/nix/labels/documentation
[recent documentation changes]: https://gerrit.lix.systems/q/p:lix+path:%22%5Edoc/manual/.*%22

35
.github/STALE-BOT.md vendored Normal file
View file

@ -0,0 +1,35 @@
# Stale bot information
- Thanks for your contribution!
- To remove the stale label, just leave a new comment.
- _How to find the right people to ping?_ &rarr; [`git blame`](https://git-scm.com/docs/git-blame) to the rescue! (or GitHub's history and blame buttons.)
- You can always ask for help on [our Discourse Forum](https://discourse.nixos.org/) or on [Matrix - #nix:nixos.org](https://matrix.to/#/#nix:nixos.org).
## Suggestions for PRs
1. GitHub sometimes doesn't notify people who commented / reviewed a PR previously, when you (force) push commits. If you have addressed the reviews you can [officially ask for a review](https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from those who commented to you or anyone else.
2. If it is unfinished but you plan to finish it, please mark it as a draft.
3. If you don't expect to work on it any time soon, closing it with a short comment may encourage someone else to pick up your work.
4. To get things rolling again, rebase the PR against the target branch and address valid comments.
5. If you need a review to move forward, ask in [the Discourse thread for PRs that need help](https://discourse.nixos.org/t/prs-in-distress/3604).
6. If all you need is a merge, check the git history to find and [request reviews](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/requesting-a-pull-request-review) from people who usually merge related contributions.
## Suggestions for issues
1. If it is resolved (either for you personally, or in general), please consider closing it.
2. If this might still be an issue, but you are not interested in promoting its resolution, please consider closing it while encouraging others to take over and reopen an issue if they care enough.
3. If you still have interest in resolving it, try to ping somebody who you believe might have an interest in the topic. Consider discussing the problem in [our Discourse Forum](https://discourse.nixos.org/).
4. As with all open source projects, your best option is to submit a Pull Request that addresses this issue. We :heart: this attitude!
**Memorandum on closing issues**
Don't be afraid to close an issue that holds valuable information. Closed issues stay in the system for people to search, read, cross-reference, or even reopen--nothing is lost! Closing obsolete issues is an important way to help maintainers focus their time and effort.
## Useful GitHub search queries
- [Open PRs with any stale-bot interaction](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+)
- [Open PRs with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22)
- [Open PRs with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/pulls?q=is%3Apr+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)
- [Open Issues with any stale-bot interaction](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+)
- [Open Issues with any stale-bot interaction and `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+label%3A%22stale%22+)
- [Open Issues with any stale-bot interaction and NOT `stale`](https://github.com/NixOS/nix/issues?q=is%3Aissue+is%3Aopen+commenter%3Aapp%2Fstale+-label%3A%22stale%22+)

6
.github/dependabot.yml vendored Normal file
View file

@ -0,0 +1,6 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"

23
.github/labeler.yml vendored Normal file
View file

@ -0,0 +1,23 @@
"documentation":
- doc/manual/*
- src/nix/**/*.md
"store":
- src/libstore/store-api.*
- src/libstore/*-store.*
"fetching":
- src/libfetchers/**/*
"repl":
- src/libcmd/repl.*
- src/nix/repl.*
"new-cli":
- src/nix/**/*
"with-tests":
# Unit tests
- src/*/tests/**/*
# Functional and integration tests
- tests/functional/**/*

9
.github/stale.yml vendored Normal file
View file

@ -0,0 +1,9 @@
# Configuration for probot-stale - https://github.com/probot/stale
daysUntilStale: 180
daysUntilClose: false
exemptLabels:
- "critical"
- "never-stale"
staleLabel: "stale"
markComment: false
closeComment: false

131
.gitignore vendored
View file

@ -1,5 +1,128 @@
Makefile.config
perl/Makefile.config
# /
/aclocal.m4
/autom4te.cache
/precompiled-headers.h.gch
/config.*
/configure
/stamp-h1
/svn-revision
/libtool
/config
# /doc/manual/
/doc/manual/*.1
/doc/manual/*.5
/doc/manual/*.8
/doc/manual/generated/*
/doc/manual/nix.json
/doc/manual/conf-file.json
/doc/manual/language.json
/doc/manual/xp-features.json
/doc/manual/src/command-ref/experimental-features-shortlist.md
/doc/manual/src/contributing/experimental-feature-descriptions.md
/doc/manual/src/release-notes/rl-next-generated.md
# /scripts/
/scripts/nix-profile.sh
/scripts/nix-profile-daemon.sh
/scripts/nix-profile.fish
/scripts/nix-profile-daemon.fish
# /src/libexpr/
/src/libexpr/lexer-tab.cc
/src/libexpr/lexer-tab.hh
/src/libexpr/parser-tab.cc
/src/libexpr/parser-tab.hh
/src/libexpr/parser-tab.output
/src/libexpr/nix.tbl
/src/libexpr/tests
/tests/unit/libexpr/libnixexpr-tests
# /src/libstore/
*.gen.*
/src/libstore/tests
/tests/unit/libstore/libnixstore-tests
# /src/libutil/
/src/libutil/tests
/tests/unit/libutil/libnixutil-tests
/src/nix/nix
/src/nix/doc
# /src/nix-env/
/src/nix-env/nix-env
# /src/nix-instantiate/
/src/nix-instantiate/nix-instantiate
# /src/nix-store/
/src/nix-store/nix-store
/src/nix-prefetch-url/nix-prefetch-url
/src/nix-collect-garbage/nix-collect-garbage
# /src/nix-channel/
/src/nix-channel/nix-channel
# /src/nix-build/
/src/nix-build/nix-build
/src/nix-copy-closure/nix-copy-closure
/src/error-demo/error-demo
/src/build-remote/build-remote
# /tests/functional/
/tests/functional/test-tmp
/tests/functional/common/vars-and-functions.sh
/tests/functional/result*
/tests/functional/restricted-innocent
/tests/functional/shell
/tests/functional/shell.drv
/tests/functional/config.nix
/tests/functional/ca/config.nix
/tests/functional/dyn-drv/config.nix
/tests/functional/repl-result-out
/tests/functional/debugger-test-out
/tests/functional/test-libstoreconsumer/test-libstoreconsumer
# /tests/functional/lang/
/tests/functional/lang/*.out
/tests/functional/lang/*.out.xml
/tests/functional/lang/*.err
/tests/functional/lang/*.ast
/perl/lib/Nix/Config.pm
/perl/lib/Nix/Store.cc
/misc/systemd/nix-daemon.service
/misc/systemd/nix-daemon.socket
/misc/systemd/nix-daemon.conf
/misc/upstart/nix-daemon.conf
/src/resolve-system-dependencies/resolve-system-dependencies
outputs/ outputs/
*.a
*.o
*.o.tmp
*.so
*.dylib
*.dll
*.exe
*.dep
*~
*.pc
*.plist
# GNU Global # GNU Global
GPATH GPATH
GRTAGS GRTAGS
@ -11,13 +134,15 @@ GTAGS
# auto-generated compilation database # auto-generated compilation database
compile_commands.json compile_commands.json
rust-project.json
nix-rust/target
result result
result-* result-*
.vscode/ .vscode/
.direnv/ .direnv/
.envrc.local
# clangd and possibly more # clangd and possibly more
.cache/ .cache/
@ -32,7 +157,3 @@ buildtime.bin
# We generate this with a Nix shell hook # We generate this with a Nix shell hook
/.pre-commit-config.yaml /.pre-commit-config.yaml
/.nocontribmsg /.nocontribmsg
/release
# Rust build files when using Cargo (not actually supported for building but it spews the files anyway)
/target/

View file

@ -1 +0,0 @@
This is a file used by the dev shell shellHook in package.nix to check that this is actually a Lix repo before installing git hooks. Its contents have no meaning.

1
.version Normal file
View file

@ -0,0 +1 @@
2.90.0

79
Cargo.lock generated
View file

@ -1,79 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "countme"
version = "3.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7704b5fdd17b18ae31c4c1da5a2e0305a2bf17b5249300a9ee9ed7b72114c636"
[[package]]
name = "dissimilar"
version = "1.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59f8e79d1fbf76bdfbde321e902714bf6c49df88a7dda6fc682fc2979226962d"
[[package]]
name = "expect-test"
version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e0be0a561335815e06dab7c62e50353134c796e7a6155402a64bcff66b6a5e0"
dependencies = [
"dissimilar",
"once_cell",
]
[[package]]
name = "hashbrown"
version = "0.14.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
[[package]]
name = "lix-doc"
version = "0.0.1"
dependencies = [
"expect-test",
"rnix",
"rowan",
]
[[package]]
name = "once_cell"
version = "1.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92"
[[package]]
name = "rnix"
version = "0.11.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bb35cedbeb70e0ccabef2a31bcff0aebd114f19566086300b8f42c725fc2cb5f"
dependencies = [
"rowan",
]
[[package]]
name = "rowan"
version = "0.15.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0a542b0253fa46e632d27a1dc5cf7b930de4df8659dc6e720b647fc72147ae3d"
dependencies = [
"countme",
"hashbrown",
"rustc-hash",
"text-size",
]
[[package]]
name = "rustc-hash"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2"
[[package]]
name = "text-size"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f18aa187839b2bdb1ad2fa35ead8c4c2976b64e4363c386d45ac0f7ee85c9233"

View file

@ -1,6 +0,0 @@
[workspace]
resolver = "2"
members = ["src/lix-doc"]
[workspace.package]
edition = "2021"

View file

@ -6,14 +6,14 @@ Read more about us at https://lix.systems.
## Installation ## Installation
On Linux and macOS the easiest way to install Lix is to run the following shell command On Linux and macOS the easiest way to install Nix is to run the following shell command
(as a user other than root): (as a user other than root):
```console ```console
$ curl -sSf -L https://install.lix.systems/lix | sh -s -- install $ curl -sSf -L https://install.lix.systems/lix | sh -s -- install
``` ```
For systems that **already have a Nix implementation installed**, such as NixOS systems, read our [install page](https://lix.systems/install) For systems that **already have Nix installed**, such as NixOS systems, read our [install page](https://lix.systems/install)
## Building And Developing ## Building And Developing
@ -26,4 +26,4 @@ See our [Hacking guide](https://git.lix.systems/lix-project/lix/src/branch/main/
## License ## License
Lix is released under [LGPL-2.1-or-later](./COPYING). Lix is released under the [LGPL v2.1](./COPYING).

View file

@ -6,7 +6,13 @@ shopt -s inherit_errexit
scriptdir=$(cd "$(dirname -- "$0")" ; pwd -P) scriptdir=$(cd "$(dirname -- "$0")" ; pwd -P)
cd "$scriptdir/.." cd "$scriptdir/.."
if [[ $# -lt 2 ]]; then benchSepIx=0
for arg in "${@}"; do
if [[ "$arg" == "--" ]]; then break; fi
benchSepIx=$((benchSepIx+1))
done
if [[ $# -lt 2 ]] || [[ $benchSepIx -lt 2 ]]; then
# FIXME(jade): it is a reasonable use case to want to run a benchmark run # FIXME(jade): it is a reasonable use case to want to run a benchmark run
# on just one build. However, since we are using hyperfine in comparison # on just one build. However, since we are using hyperfine in comparison
# mode, we would have to combine the JSON ourselves to support that, which # mode, we would have to combine the JSON ourselves to support that, which
@ -14,7 +20,7 @@ if [[ $# -lt 2 ]]; then
# not-bash. # not-bash.
echo "Fewer than two result dirs given, nothing to compare!" >&2 echo "Fewer than two result dirs given, nothing to compare!" >&2
echo "Pass some directories (with names indicating which alternative they are) with bin/nix in them" >&2 echo "Pass some directories (with names indicating which alternative they are) with bin/nix in them" >&2
echo "Usage: ./bench/bench.sh result-1 result-2 [result-3...]" >&2 echo "Usage: ./bench/bench.sh result-1 result-2 [result-3...] [--] [bench-1...]" >&2
exit 1 exit 1
fi fi
@ -28,7 +34,8 @@ export NIX_REMOTE="$(mktemp -d)"
_exit='rm -rfv "$NIX_REMOTE"; $_exit' _exit='rm -rfv "$NIX_REMOTE"; $_exit'
export NIX_PATH="nixpkgs=bench/nixpkgs:nixos-config=bench/configuration.nix" export NIX_PATH="nixpkgs=bench/nixpkgs:nixos-config=bench/configuration.nix"
builds=("$@") builds=("${@:1:$benchSepIx}")
benchsFromArgs=("${@:$((benchSepIx+2))}")
flake_args="--extra-experimental-features 'nix-command flakes'" flake_args="--extra-experimental-features 'nix-command flakes'"
@ -43,15 +50,22 @@ cases=(
[rebuild]="{BUILD}/bin/nix $flake_args eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'" [rebuild]="{BUILD}/bin/nix $flake_args eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'"
[rebuild-lh]="GC_INITIAL_HEAP_SIZE=10g {BUILD}/bin/nix eval $flake_args --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'" [rebuild-lh]="GC_INITIAL_HEAP_SIZE=10g {BUILD}/bin/nix eval $flake_args --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'"
[parse]="{BUILD}/bin/nix $flake_args eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix" [parse]="{BUILD}/bin/nix $flake_args eval -f bench/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix"
[substr100k]="{BUILD}/bin/nix eval --no-eval-cache -f bench/substring.nix --apply 'f: f.bench 100000'"
[substr200k]="{BUILD}/bin/nix eval --no-eval-cache -f bench/substring.nix --apply 'f: f.bench 200000'"
[substr300k]="{BUILD}/bin/nix eval --no-eval-cache -f bench/substring.nix --apply 'f: f.bench 300000'"
[substr400k]="{BUILD}/bin/nix eval --no-eval-cache -f bench/substring.nix --apply 'f: f.bench 400000'"
) )
benches=( defaultBenches=(
rebuild rebuild
rebuild-lh rebuild-lh
search search
parse parse
) )
benches=("${benchsFromArgs[@]:-${defaultBenches[@]}}")
for k in "${benches[@]}"; do for k in "${benches[@]}"; do
taskset -c 2,3 \ taskset -c 2,3 \
chrt -f 50 \ chrt -f 50 \

19
bench/substring.nix Normal file
View file

@ -0,0 +1,19 @@
# Tests stringLength and substring performance
with builtins;
{
bench =
baseStrLen:
let
# a big string
baseStr = concatStringsSep "" (genList (_: "x") baseStrLen);
# a way to force the values
sum = ns: foldl' (acc: n: acc + n) 0 ns;
forceOpTimes =
range: # Int
op: # Int -> Int
sum (genList (ix: op ix) range);
benchOp = ix: stringLength (substring ix 1 baseStr);
in
forceOpTimes baseStrLen benchOp;
}

View file

@ -0,0 +1,99 @@
diff --git a/darwin_stop_world.c b/darwin_stop_world.c
index 0468aaec..b348d869 100644
--- a/darwin_stop_world.c
+++ b/darwin_stop_world.c
@@ -356,6 +356,7 @@ GC_INNER void GC_push_all_stacks(void)
int nthreads = 0;
word total_size = 0;
mach_msg_type_number_t listcount = (mach_msg_type_number_t)THREAD_TABLE_SZ;
+ size_t stack_limit;
if (!EXPECT(GC_thr_initialized, TRUE))
GC_thr_init();
@@ -411,6 +412,19 @@ GC_INNER void GC_push_all_stacks(void)
GC_push_all_stack_sections(lo, hi, p->traced_stack_sect);
}
if (altstack_lo) {
+ // When a thread goes into a coroutine, we lose its original sp until
+ // control flow returns to the thread.
+ // While in the coroutine, the sp points outside the thread stack,
+ // so we can detect this and push the entire thread stack instead,
+ // as an approximation.
+ // We assume that the coroutine has similarly added its entire stack.
+ // This could be made accurate by cooperating with the application
+ // via new functions and/or callbacks.
+ stack_limit = pthread_get_stacksize_np(p->id);
+ if (altstack_lo >= altstack_hi || altstack_lo < altstack_hi - stack_limit) { // sp outside stack
+ altstack_lo = altstack_hi - stack_limit;
+ }
+
total_size += altstack_hi - altstack_lo;
GC_push_all_stack(altstack_lo, altstack_hi);
}
diff --git a/include/gc.h b/include/gc.h
index edab6c22..f2c61282 100644
--- a/include/gc.h
+++ b/include/gc.h
@@ -2172,6 +2172,11 @@ GC_API void GC_CALL GC_win32_free_heap(void);
(*GC_amiga_allocwrapper_do)(a,GC_malloc_atomic_ignore_off_page)
#endif /* _AMIGA && !GC_AMIGA_MAKINGLIB */
+#if !__APPLE__
+/* Patch doesn't work on apple */
+#define NIX_BOEHM_PATCH_VERSION 1
+#endif
+
#ifdef __cplusplus
} /* extern "C" */
#endif
diff --git a/pthread_stop_world.c b/pthread_stop_world.c
index b5d71e62..aed7b0bf 100644
--- a/pthread_stop_world.c
+++ b/pthread_stop_world.c
@@ -768,6 +768,8 @@ STATIC void GC_restart_handler(int sig)
/* world is stopped. Should not fail if it isn't. */
GC_INNER void GC_push_all_stacks(void)
{
+ size_t stack_limit;
+ pthread_attr_t pattr;
GC_bool found_me = FALSE;
size_t nthreads = 0;
int i;
@@ -851,6 +853,37 @@ GC_INNER void GC_push_all_stacks(void)
hi = p->altstack + p->altstack_size;
/* FIXME: Need to scan the normal stack too, but how ? */
/* FIXME: Assume stack grows down */
+ } else {
+#ifdef HAVE_PTHREAD_ATTR_GET_NP
+ if (!pthread_attr_init(&pattr)
+ || !pthread_attr_get_np(p->id, &pattr))
+#else /* HAVE_PTHREAD_GETATTR_NP */
+ if (pthread_getattr_np(p->id, &pattr))
+#endif
+ {
+ ABORT("GC_push_all_stacks: pthread_getattr_np failed!");
+ }
+ if (pthread_attr_getstacksize(&pattr, &stack_limit)) {
+ ABORT("GC_push_all_stacks: pthread_attr_getstacksize failed!");
+ }
+ if (pthread_attr_destroy(&pattr)) {
+ ABORT("GC_push_all_stacks: pthread_attr_destroy failed!");
+ }
+ // When a thread goes into a coroutine, we lose its original sp until
+ // control flow returns to the thread.
+ // While in the coroutine, the sp points outside the thread stack,
+ // so we can detect this and push the entire thread stack instead,
+ // as an approximation.
+ // We assume that the coroutine has similarly added its entire stack.
+ // This could be made accurate by cooperating with the application
+ // via new functions and/or callbacks.
+ #ifndef STACK_GROWS_UP
+ if (lo >= hi || lo < hi - stack_limit) { // sp outside stack
+ lo = hi - stack_limit;
+ }
+ #else
+ #error "STACK_GROWS_UP not supported in boost_coroutine2 (as of june 2021), so we don't support it in Nix."
+ #endif
}
GC_push_all_stack_sections(lo, hi, traced_stack_sect);
# ifdef STACK_GROWS_UP

View file

@ -0,0 +1,12 @@
diff --git a/include/gc_allocator.h b/include/gc_allocator.h
index 597c7f13..587286be 100644
--- a/include/gc_allocator.h
+++ b/include/gc_allocator.h
@@ -312,6 +312,7 @@ public:
template<>
class traceable_allocator<void> {
+public:
typedef size_t size_type;
typedef ptrdiff_t difference_type;
typedef void* pointer;

View file

@ -44,41 +44,32 @@ void FixIncludesCallbacks::LexedFileChanged(FileID, LexedFileChangeReason,
} }
void FixIncludesCallbacks::InclusionDirective( void FixIncludesCallbacks::InclusionDirective(
SourceLocation, const Token &, StringRef FileName, bool IsAngled, SourceLocation, const Token &, StringRef, bool,
CharSourceRange FilenameRange, OptionalFileEntryRef File, StringRef, CharSourceRange FilenameRange, OptionalFileEntryRef File, StringRef,
StringRef, const Module *, SrcMgr::CharacteristicKind) { StringRef, const Module *, SrcMgr::CharacteristicKind) {
if (Ignore) if (Ignore)
return; return;
// FIXME: this is kinda evil, but this is a one-time fixup // FIXME: this is kinda evil, but this is a one-time fixup
const std::vector<std::string> SourceDirs = {"src/", "include/lix/"}; const std::string SourceDir = "src/";
const auto Bracketize = [IsAngled](StringRef s) { if (File && File->getNameAsRequested().contains(SourceDir)) {
return IsAngled ? ("<" + s + ">").str() : ("\"" + s + "\"").str(); StringRef Name = File->getNameAsRequested();
}; auto Idx = Name.find(SourceDir);
assert(Idx != std::string::npos);
StringRef Suffix = Name.drop_front(Idx + SourceDir.length());
for (const auto &SourceDir : SourceDirs) { if (!Suffix.starts_with("lib")) {
const bool IsAlreadyFixed = FileName.starts_with("lix/lib"); llvm::dbgs() << "ignored: " << Suffix << "\n";
if (File && File->getNameAsRequested().contains(SourceDir) && return;
!IsAlreadyFixed) {
StringRef Name = File->getNameAsRequested();
auto Idx = Name.find(SourceDir);
assert(Idx != std::string::npos);
std::string Suffix = Name.drop_front(Idx + SourceDir.length()).str();
if (!Suffix.starts_with("lib")) {
llvm::dbgs() << "ignored: " << Suffix << "\n";
return;
}
Suffix = "lix/" + Suffix;
auto Diag = Check.diag(FilenameRange.getBegin(),
"include needs to specify the source subdir");
Diag << FilenameRange
<< FixItHint::CreateReplacement(FilenameRange, Bracketize(Suffix));
} }
auto Diag = Check.diag(FilenameRange.getBegin(),
"include needs to specify the source subdir");
Diag << FilenameRange
<< FixItHint::CreateReplacement(FilenameRange,
("\"" + Suffix + "\"").str());
} }
} }

View file

@ -2,7 +2,6 @@
#include <clang-tidy/ClangTidyModuleRegistry.h> #include <clang-tidy/ClangTidyModuleRegistry.h>
#include "FixIncludes.hh" #include "FixIncludes.hh"
#include "HasPrefixSuffix.hh" #include "HasPrefixSuffix.hh"
#include "CharPtrCast.hh"
namespace nix::clang_tidy { namespace nix::clang_tidy {
using namespace clang; using namespace clang;
@ -13,7 +12,6 @@ class NixClangTidyChecks : public ClangTidyModule {
void addCheckFactories(ClangTidyCheckFactories &CheckFactories) override { void addCheckFactories(ClangTidyCheckFactories &CheckFactories) override {
CheckFactories.registerCheck<HasPrefixSuffixCheck>("lix-hasprefixsuffix"); CheckFactories.registerCheck<HasPrefixSuffixCheck>("lix-hasprefixsuffix");
CheckFactories.registerCheck<FixIncludesCheck>("lix-fixincludes"); CheckFactories.registerCheck<FixIncludesCheck>("lix-fixincludes");
CheckFactories.registerCheck<CharPtrCastCheck>("lix-charptrcast");
} }
}; };

View file

@ -1,6 +1,6 @@
# Clang tidy lints for Lix # Clang tidy lints for Nix
This is a skeleton of a clang-tidy lints library for Lix. This is a skeleton of a clang-tidy lints library for Nix.
Currently there is one check (which is already obsolete as it has served its Currently there is one check (which is already obsolete as it has served its
goal and is there as an example), `HasPrefixSuffixCheck`. goal and is there as an example), `HasPrefixSuffixCheck`.
@ -10,13 +10,13 @@ goal and is there as an example), `HasPrefixSuffixCheck`.
One file: One file:
``` ```
ninja -C build && clang-tidy --checks='-*,lix-*' --load=build/liblix-clang-tidy.so -p ../compile_commands.json -header-filter '\.\./src/.*\.h' --fix ../src/libcmd/installables.cc ninja -C build && clang-tidy --checks='-*,nix-*' --load=build/libnix-clang-tidy.so -p ../compile_commands.json --fix ../src/libcmd/installables.cc
``` ```
Several files, in parallel: Several files, in parallel:
``` ```
ninja -C build && run-clang-tidy -checks='-*,lix-*' -load=build/liblix-clang-tidy.so -p .. -header-filter '\.\./src/.*\.h' -fix ../src | tee -a clang-tidy-result ninja -C build && run-clang-tidy -checks='-*,nix-*' -load=build/libnix-clang-tidy.so -p .. -fix ../src | tee -a clang-tidy-result
``` ```
## Resources ## Resources

13
clang-tidy/meson.build Normal file
View file

@ -0,0 +1,13 @@
project('lix-clang-tidy', ['cpp', 'c'],
version : '0.1',
default_options : ['warning_level=3', 'cpp_std=c++20'])
llvm = dependency('Clang', version: '>= 14', modules: ['libclang'])
sources = files(
'HasPrefixSuffix.cc',
'LixClangTidyChecks.cc',
'FixIncludes.cc',
)
shared_module('lix-clang-tidy', sources,
dependencies: llvm)

View file

@ -20,7 +20,7 @@ OUTPUT_DIRECTORY = @docdir@
# for a project that appears at the top of each page and should give viewer a # for a project that appears at the top of each page and should give viewer a
# quick idea about the purpose of the project. Keep the description short. # quick idea about the purpose of the project. Keep the description short.
PROJECT_BRIEF = "Lix: A modern, delicious implementation of the Nix package manager; unstable internal interfaces" PROJECT_BRIEF = "Nix, the purely functional package manager; unstable internal interfaces"
# If the GENERATE_LATEX tag is set to YES, doxygen will generate LaTeX output. # If the GENERATE_LATEX tag is set to YES, doxygen will generate LaTeX output.
# The default value is: YES. # The default value is: YES.

View file

@ -28,7 +28,6 @@ internal_api_docs = custom_target(
output : 'html', output : 'html',
install : true, install : true,
install_dir : datadir / 'doc/nix/internal-api', install_dir : datadir / 'doc/nix/internal-api',
build_always_stale : true,
) )
alias_target('internal-api-html', internal_api_docs) alias_target('internal-api-html', internal_api_docs)

View file

@ -11,10 +11,6 @@ additional-js = ["redirects.js"]
# to just submit a Gerrit CL by the web for trivial stuff. # to just submit a Gerrit CL by the web for trivial stuff.
edit-url-template = "https://github.com/lix-project/lix/tree/main/doc/manual/{path}" edit-url-template = "https://github.com/lix-project/lix/tree/main/doc/manual/{path}"
git-repository-url = "https://git.lix.systems/lix-project/lix" git-repository-url = "https://git.lix.systems/lix-project/lix"
# Folding by default would prevent things like "Ctrl+F for nix-env" from working
# trivially, but the user should be able to fold if they want to.
fold.enable = true
fold.level = 30
# Handles replacing @docroot@ with a path to ./src relative to that markdown file, # Handles replacing @docroot@ with a path to ./src relative to that markdown file,
# {{#include handlebars}}, and the @generated@ syntax used within these. it mostly # {{#include handlebars}}, and the @generated@ syntax used within these. it mostly

View file

@ -3,147 +3,61 @@
# #
# It's used for crediting people accurately in release notes. The release notes # It's used for crediting people accurately in release notes. The release notes
# script will link to forgejo, then to GitHub if forgejo is not present. # script will link to forgejo, then to GitHub if forgejo is not present.
#
# When adding someone from outside the Lix project, you generally want to simply link their GitHub profile without adding a display name unless they are well-known in the community by that display name.
#
# See doc/manual/src/contributing/hacking.md for more documentation on this file's format and typical usage.
9999years:
display_name: wiggles
forgejo: rbt
github: 9999years
Artturin:
github: Artturin
DavHau:
github: DavHau
Kha:
github: Kha
Lunaphied:
forgejo: Lunaphied
github: Lunaphied
Qyriad:
forgejo: Qyriad
github: Qyriad
SharzyL:
github: SharzyL
alois31:
forgejo: alois31
github: alois31
artemist:
display_name: Artemis Tosini
forgejo: artemist
cole-h:
display_name: Cole Helbling
github: cole-h
delan:
display_name: delan
forgejo: delan
github: delan
edolstra:
display_name: Eelco Dolstra
github: edolstra
ericson:
display_name: John Ericson
github: ericson2314
goldstein:
display_name: goldstein
forgejo: goldstein
github: GoldsteinE
horrors: horrors:
display_name: eldritch horrors display_name: eldritch horrors
forgejo: pennae forgejo: pennae
github: pennae github: pennae
iFreilicht: Qyriad:
github: iFreilicht forgejo: Qyriad
github: Qyriad
isabelroses:
forgejo: isabelroses
github: isabelroses
jade: jade:
forgejo: jade forgejo: jade
github: lf- github: lf-
kjeremy: iFreilicht:
github: kjeremy github: iFreilicht
kloenk:
forgejo: kloenk
github: kloenk
lovesegfault:
github: lovesegfault
ma27: ma27:
forgejo: ma27 forgejo: ma27
github: ma27 github: ma27
Lunaphied:
forgejo: Lunaphied
github: Lunaphied
9999years:
display_name: wiggles
github: 9999years
forgejo: rbt
matthewbauer: matthewbauer:
github: matthewbauer github: matthewbauer
midnightveil:
display_name: julia
forgejo: midnightveil
github: midnightveil
ncfavier:
github: ncfavier
piegames:
display_name: piegames
forgejo: piegames
github: piegamesde
puck:
display_name: puck
forgejo: puck
github: puckipedia
quantumjump:
display_name: Quantum Jump
github: QuantumBJump
r-vdp:
github: r-vdp
raito: raito:
display_name: Raito Bezarius display_name: Raito Bezarius
forgejo: raito
github: RaitoBezarius github: RaitoBezarius
forgejo: raito
roberth: winter:
display_name: Robert Hensing github: winterqt
github: roberth forgejo: winter
Kha:
github: Kha
Artturin:
github: Artturin
thufschmitt: thufschmitt:
display_name: Théophane Hufschmitt display_name: Théophane Hufschmitt
github: thufschmitt github: thufschmitt
tomberek: edolstra:
display_name: Tom Bereknyei display_name: Eelco Dolstra
github: tomberek github: edolstra
valentin: roberth:
display_name: Valentin Gagarin display_name: Robert Hensing
github: fricklerhandwerk github: roberth
winter:
forgejo: winter
github: winterqt
yshui:
github: yshui

View file

@ -1,18 +0,0 @@
# Usually "experimental" or "deprecated"
_kind:
# "xp" or "dp"
kindShort:
with builtins;
with import ./utils.nix;
let
showFeature =
name: doc:
squash ''
## [`${name}`]{#${kindShort}-feature-${name}}
${doc}
'';
in
xps: (concatStringsSep "\n" (attrValues (mapAttrs showFeature xps)))

View file

@ -1,14 +1,9 @@
# Usually "experimental" or "deprecated"
kind:
# "xp" or "dp"
kindShort:
with builtins; with builtins;
with import ./utils.nix; with import ./utils.nix;
let let
showExperimentalFeature = name: doc: '' showExperimentalFeature = name: doc: ''
- [`${name}`](@docroot@/contributing/${kind}-features.md#${kindShort}-feature-${name}) - [`${name}`](@docroot@/contributing/experimental-features.md#xp-feature-${name})
''; '';
in in
xps: indent " " (concatStrings (attrValues (mapAttrs showExperimentalFeature xps))) xps: indent " " (concatStrings (attrValues (mapAttrs showExperimentalFeature xps)))

View file

@ -0,0 +1,13 @@
with builtins;
with import ./utils.nix;
let
showExperimentalFeature =
name: doc:
squash ''
## [`${name}`]{#xp-feature-${name}}
${doc}
'';
in
xps: (concatStringsSep "\n" (attrValues (mapAttrs showExperimentalFeature xps)))

View file

@ -20,8 +20,6 @@ conf_file_json = custom_target(
capture : true, capture : true,
output : 'conf-file.json', output : 'conf-file.json',
env : nix_env_for_docs, env : nix_env_for_docs,
# FIXME: put the actual lib targets in here? meson have introspection challenge 2024 though.
build_always_stale : true,
) )
nix_conf_file_md_body = custom_target( nix_conf_file_md_body = custom_target(
@ -52,8 +50,6 @@ nix_exp_features_json = custom_target(
command : [ nix, '__dump-xp-features' ], command : [ nix, '__dump-xp-features' ],
capture : true, capture : true,
output : 'xp-features.json', output : 'xp-features.json',
# FIXME: put the actual lib targets in here? meson have introspection challenge 2024 though.
build_always_stale : true,
) )
language_json = custom_target( language_json = custom_target(
@ -61,8 +57,6 @@ language_json = custom_target(
output : 'language.json', output : 'language.json',
capture : true, capture : true,
env : nix_env_for_docs, env : nix_env_for_docs,
# FIXME: put the actual lib targets in here? meson have introspection challenge 2024 though.
build_always_stale : true,
) )
nix3_cli_json = custom_target( nix3_cli_json = custom_target(
@ -70,8 +64,6 @@ nix3_cli_json = custom_target(
capture : true, capture : true,
output : 'nix.json', output : 'nix.json',
env : nix_env_for_docs, env : nix_env_for_docs,
# FIXME: put the actual lib targets in here? meson have introspection challenge 2024 though.
build_always_stale : true,
) )
generate_manual_deps = files( generate_manual_deps = files(
@ -80,9 +72,9 @@ generate_manual_deps = files(
# Generates builtins.md and builtin-constants.md. # Generates builtins.md and builtin-constants.md.
subdir('src/language') subdir('src/language')
# Generates new-cli pages, {experimental,deprecated}-features-shortlist.md, and conf-file.md. # Generates new-cli pages, experimental-features-shortlist.md, and conf-file.md.
subdir('src/command-ref') subdir('src/command-ref')
# Generates {experimental,deprecated}-feature-descriptions.md. # Generates experimental-feature-descriptions.md.
subdir('src/contributing') subdir('src/contributing')
# Generates rl-next-generated.md. # Generates rl-next-generated.md.
subdir('src/release-notes') subdir('src/release-notes')
@ -114,13 +106,10 @@ manual = custom_target(
nix3_cli_files, nix3_cli_files,
experimental_features_shortlist_md, experimental_features_shortlist_md,
experimental_feature_descriptions_md, experimental_feature_descriptions_md,
deprecated_features_shortlist_md,
deprecated_feature_descriptions_md,
conf_file_md, conf_file_md,
builtins_md, builtins_md,
builtin_constants_md, builtin_constants_md,
rl_next_generated, rl_next_generated,
nix,
], ],
output : [ output : [
'manual', 'manual',
@ -197,7 +186,6 @@ foreach command : nix_nested_manpages
], ],
input : [ input : [
manual_md, manual_md,
nix,
], ],
output : command[0] + '-' + page + '.1', output : command[0] + '-' + page + '.1',
install : true, install : true,
@ -310,7 +298,6 @@ foreach page : nix3_manpages
input : [ input : [
'render-manpage.sh', 'render-manpage.sh',
manual_md, manual_md,
nix,
], ],
output : page + '.1', output : page + '.1',
install : true, install : true,
@ -354,7 +341,6 @@ foreach entry : nix_manpages
'render-manpage.sh', 'render-manpage.sh',
manual_md, manual_md,
entry.get(3, []), entry.get(3, []),
nix,
], ],
output : '@0@.@1@'.format(entry[0], entry[1]), output : '@0@.@1@'.format(entry[0], entry[1]),
install : true, install : true,

View file

@ -1,4 +1,4 @@
#!/usr/bin/env bash #!/bin/sh
set -euo pipefail set -euo pipefail

View file

@ -24,6 +24,7 @@ const redirects = {
"chap-writing-nix-expressions": "language/index.html", "chap-writing-nix-expressions": "language/index.html",
"part-command-ref": "command-ref/command-ref.html", "part-command-ref": "command-ref/command-ref.html",
"conf-allow-import-from-derivation": "command-ref/conf-file.html#conf-allow-import-from-derivation", "conf-allow-import-from-derivation": "command-ref/conf-file.html#conf-allow-import-from-derivation",
"conf-allow-new-privileges": "command-ref/conf-file.html#conf-allow-new-privileges",
"conf-allowed-uris": "command-ref/conf-file.html#conf-allowed-uris", "conf-allowed-uris": "command-ref/conf-file.html#conf-allowed-uris",
"conf-allowed-users": "command-ref/conf-file.html#conf-allowed-users", "conf-allowed-users": "command-ref/conf-file.html#conf-allowed-users",
"conf-auto-optimise-store": "command-ref/conf-file.html#conf-auto-optimise-store", "conf-auto-optimise-store": "command-ref/conf-file.html#conf-auto-optimise-store",
@ -345,7 +346,7 @@ const redirects = {
"linux": "uninstall.html#linux", "linux": "uninstall.html#linux",
"macos": "uninstall.html#macos", "macos": "uninstall.html#macos",
"uninstalling": "uninstall.html", "uninstalling": "uninstall.html",
}, }
"contributing/hacking.html": { "contributing/hacking.html": {
"nix-with-flakes": "#building-nix-with-flakes", "nix-with-flakes": "#building-nix-with-flakes",
"classic-nix": "#building-nix", "classic-nix": "#building-nix",

View file

@ -1,4 +1,4 @@
#!/usr/bin/env bash #!/bin/sh
set -euo pipefail set -euo pipefail

View file

@ -0,0 +1,15 @@
---
synopsis: Clang build timing analysis
cls: 587
---
We now have Clang build profiling available, which generates Chrome
tracing files for each compilation unit. To enable it, run `meson configure
build -Dprofile-build=enabled` then rerun the compilation.
If you want to make the build go faster, do a clang build with meson, then run
`maintainers/buildtime_report.sh build`, then contemplate how to improve the
build time.
You can also look at individual object files' traces in
<https://ui.perfetto.dev>.

View file

@ -1,21 +0,0 @@
---
synopsis: "Build failures caused by `allowSubstitutes = false` while being the wrong system now produce a decent error"
issues: [fj#484]
cls: [1841]
category: Fixes
credits: jade
---
Nix allows derivations to set `allowSubstitutes = false` in order to force them to be built locally without querying substituters for them.
This is useful for derivations that are very fast to build (especially if they produce large output).
However, this can shoot you in the foot if the derivation *has* to be substituted such as if the derivation is for another architecture, which is what `--always-allow-substitutes` is for.
Perhaps such derivations that are known to be impossible to build locally should ignore `allowSubstitutes` (irrespective of remote builders) in the future, but this at least reports the failure and solution directly.
```
$ nix build -f fail.nix
error: a 'unicornsandrainbows-linux' with features {} is required to build '/nix/store/...-meow.drv', but I am a 'x86_64-linux' with features {...}
Hint: the failing derivation has allowSubstitutes set to false, forcing it to be built rather than substituted.
Passing --always-allow-substitutes to force substitution may resolve this failure if the path is available in a substituter.
```

View file

@ -1,10 +0,0 @@
---
synopsis: "`Alt+Left` and `Alt+Right` go back/forwards by words in `nix repl`"
issues: [fj#501]
cls: [1883]
category: Fixes
credits: 9999years
---
`nix repl` now recognizes `Alt+Left` and `Alt+Right` for navigating by words
when entering input in `nix repl` on more terminals/platforms.

View file

@ -0,0 +1,42 @@
---
synopsis: Concise error printing in `nix repl`
prs: 9928
cls: 811
category: Improvements
credits: 9999years
---
Previously, if an element of a list or attribute set threw an error while
evaluating, `nix repl` would print the entire error (including source location
information) inline. This output was clumsy and difficult to parse:
```
nix-repl> { err = builtins.throw "uh oh!"; }
{ err = «error:
… while calling the 'throw' builtin
at «string»:1:9:
1| { err = builtins.throw "uh oh!"; }
| ^
error: uh oh!»; }
```
Now, only the error message is displayed, making the output much more readable.
```
nix-repl> { err = builtins.throw "uh oh!"; }
{ err = «error: uh oh!»; }
```
However, if the whole expression being evaluated throws an error, source
locations and (if applicable) a stack trace are printed, just like you'd expect:
```
nix-repl> builtins.throw "uh oh!"
error:
… while calling the 'throw' builtin
at «string»:1:1:
1| builtins.throw "uh oh!"
| ^
error: uh oh!
```

View file

@ -0,0 +1,6 @@
---
synopsis: Show all FOD errors with `nix build --keep-going`
---
`nix build --keep-going` now behaves consistently with `nix-build --keep-going`. This means
that if e.g. multiple FODs fail to build, all hash mismatches are displayed.

View file

@ -0,0 +1,11 @@
---
synopsis: "`--debugger` can now access bindings from `let` expressions"
prs: 9918
issues: 8827
category: Fixes
credits: 9999years
---
Breakpoints and errors in the bindings of a `let` expression can now access
those bindings in the debugger. Previously, only the body of `let` expressions
could access those bindings.

View file

@ -0,0 +1,11 @@
---
synopsis: Enter the `--debugger` when `builtins.trace` is called if `debugger-on-trace` is set
prs: 9914
category: Features
credits: 9999years
---
If the `debugger-on-trace` option is set and `--debugger` is given,
`builtins.trace` calls will behave similarly to `builtins.break` and will enter
the debug REPL. This is useful for determining where warnings are being emitted
from.

View file

@ -1,17 +0,0 @@
---
synopsis: Deprecated language features
issues: [fj#437]
cls: [1785, 1736, 1735, 1744]
category: Breaking Changes
credits: [piegames, horrors]
---
A system for deprecation (and then the planned removal) of undesired language features has been put into place.
It is controlled via feature flags much like experimental features, except that the deprecations are enabled default,
and can be disabled via the flags for backwards compatibility (opt-out with `--extra-deprecated-features` or the Nix configuration file).
- `url-literals`: **URL literals** have long been obsolete and discouraged of use, and now they are officially deprecated.
This means that all URLs must be properly put within quotes like all other strings.
- `rec-set-overrides`: **__overrides** is an old arcane syntax which has not been in use for more than a decade.
It is soft-deprecated with a warning only, with the plan to turn that into an error in a future release.
- `ancient-let`: **The old `let` syntax** (`let { body = …; … }`) is soft-deprecated with a warning as well. Use the regular `let … in` instead.

View file

@ -0,0 +1,9 @@
---
synopsis: Stop vendoring toml11
cls: 675
category: Packaging
credits: winter
---
We don't apply any patches to it, and vendoring it locks users into
bugs (it hasn't been updated since its introduction in late 2021).

View file

@ -0,0 +1,8 @@
---
synopsis: Fix handling of truncated `.drv` files.
prs: 9673
category: Fixes
credits: horrors
---
Previously a `.drv` that was truncated in the middle of a string would case nix to enter an infinite loop, eventually exhausting all memory and crashing.

View file

@ -0,0 +1,24 @@
---
synopsis: Duplicate attribute reports are more accurate
cls: 557
credits: horrors
category: Improvements
---
Duplicate attribute errors are now more accurate, showing the path at which an error was detected rather than the full, possibly longer, path that caused the error.
Error reports are now
```ShellSession
$ nix eval --expr '{ a.b = 1; a.b.c.d = 1; }'
error: attribute 'a.b' already defined at «string»:1:3
at «string»:1:12:
1| { a.b = 1; a.b.c.d = 1;
| ^
```
instead of
```ShellSession
$ nix eval --expr '{ a.b = 1; a.b.c.d = 1; }'
error: attribute 'a.b.c.d' already defined at «string»:1:3
at «string»:1:12:
1| { a.b = 1; a.b.c.d = 1;
| ^
```

View file

@ -0,0 +1,8 @@
---
synopsis: Disallow empty search regex in `nix search`
prs: 9481
credits: [iFreilicht, horrors]
category: Miscellany
---
[`nix search`](@docroot@/command-ref/new-cli/nix3-search.md) now requires a search regex to be passed. To show all packages, use `^`.

View file

@ -0,0 +1,27 @@
---
synopsis: The `--debugger` will start more reliably in `let` expressions and function calls
prs: 9917
issues: 6649
credits: [9999years, horrors]
category: Fixes
---
Previously, if you attempted to evaluate this file with the debugger:
```nix
let
a = builtins.trace "before inner break" (
builtins.break "hello"
);
b = builtins.trace "before outer break" (
builtins.break a
);
in
b
```
Lix would correctly enter the debugger at `builtins.break a`, but if you asked
it to `:continue`, it would skip over the `builtins.break "hello"` expression
entirely.
Now, Lix will correctly enter the debugger at both breakpoints.

View file

@ -0,0 +1,10 @@
---
synopsis: Reduce eval memory usage and wall time
prs: 9658
cls: 207
credits: horrors
category: Improvements
---
Reduce the size of the `Env` struct used in the evaluator by a pointer, or 8 bytes on most modern machines.
This reduces memory usage during eval by around 2% and wall time by around 3%.

View file

@ -0,0 +1,14 @@
---
synopsis: Add new `eval-system` setting
prs: 4093
credits: [matthewbauer, horrors]
category: Features
---
Add a new `eval-system` option.
Unlike `system`, it just overrides the value of `builtins.currentSystem`.
This is more useful than overriding `system`, because you can build these derivations on remote builders which can work on the given system.
In contrast, `system` also effects scheduling which will cause Lix to build those derivations locally even if that doesn't make sense.
`eval-system` only takes effect if it is non-empty.
If empty (the default) `system` is used as before, so there is no breakage.

View file

@ -0,0 +1,10 @@
---
synopsis: Creating setuid/setgid binaries with fchmodat2 is now prohibited by the build sandbox
prs: 10501
credits: ma27
category: Fixes
---
The build sandbox blocks any attempt to create setuid/setgid binaries, but didn't check
for the use of the `fchmodat2` syscall which was introduced in Linux 6.6 and is used by
glibc >=2.39. This is fixed now.

View file

@ -0,0 +1,24 @@
---
synopsis: Fix nested flake input `follows`
prs: 6621
cls: 994
credits: [Kha, ma27]
category: Fixes
significance: significant
---
Previously nested-input overrides were ignored; that is, the following did not
override anything, in spite of the `nix3-flake` manual documenting it working:
```
{
inputs = {
foo.url = "github:bar/foo";
foo.inputs.bar.inputs.nixpkgs = "nixpkgs";
};
}
```
This is useful to avoid the 1000 instances of nixpkgs problem without having
each flake in the dependency tree to expose all of its transitive dependencies
for modification.

View file

@ -0,0 +1,34 @@
---
synopsis: Nested debuggers are no longer supported
prs: 9920
credits: 9999years
category: Improvements
---
Previously, evaluating an expression that throws an error in the debugger would
enter a second, nested debugger:
```
nix-repl> builtins.throw "what"
error: what
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.18.1. Type :? for help.
nix-repl>
```
Now, it just prints the error message like `nix repl`:
```
nix-repl> builtins.throw "what"
error:
… while calling the 'throw' builtin
at «string»:1:1:
1| builtins.throw "what"
| ^
error: what
```

View file

@ -0,0 +1,9 @@
---
synopsis: consistent order of lambda formals in printed expressions
prs: 9874
credits: horrors
category: Fixes
---
Always print lambda formals in lexicographic order rather than the internal, creation-time based symbol order.
This makes printed formals independent of the context they appear in.

View file

@ -1,10 +0,0 @@
---
synopsis: HTTP proxy environment variables are now respected for S3 binary cache stores
issues: [fj#433]
cls: [1788]
category: Fixes
credits: jade
---
Due to "legacy reasons" (according to the AWS C++ SDK docs), the AWS SDK ignores system proxy configuration by default.
We turned it back on.

View file

@ -0,0 +1,8 @@
---
synopsis: fix duplicate attribute error positions for `inherit`
prs: 9874
credits: horrors
category: Fixes
---
When an inherit caused a duplicate attribute error, the position of the error was not reported correctly, placing the error with the inherit itself or at the start of the bindings block instead of the offending attribute name.

View file

@ -0,0 +1,9 @@
---
synopsis: "`inherit (x) ...` evaluates `x` only once"
prs: 9847
category: Fixes
credits: horrors
---
`inherit (x) a b ...` now evaluates the expression `x` only once for all inherited attributes rather than once for each inherited attribute.
This does not usually have a measurable impact, but side-effects (such as `builtins.trace`) would be duplicated and expensive expressions (such as derivations) could cause a measurable slowdown.

View file

@ -0,0 +1,12 @@
---
synopsis: Store paths are allowed to start with `.`
issues: 912
prs: [9867, 9091, 9095, 9120, 9121, 9122, 9130, 9219, 9224]
credits: [roberth, horrors]
category: Fixes
---
Leading periods were allowed by accident in Nix 2.4. The Nix team has considered this to be a bug, but this behavior has since been relied on by users, leading to unnecessary difficulties.
From now on, leading periods are officially, definitively supported. The names `.` and `..` are disallowed, as well as those starting with `.-` or `..-`.
Nix versions that denied leading periods are documented [in the issue](https://github.com/NixOS/nix/issues/912#issuecomment-1919583286).

View file

@ -0,0 +1,26 @@
---
synopsis: Lix turns more internal bugs into crashes
cls: [797, 626]
credits: jade
category: Packaging
significance: significant
---
Lix now enables build options such as trapping on signed overflow and enabling
libstdc++ assertions by default. These may find new bugs in Lix, which will
present themselves as Lix processes aborting, potentially without an error
message.
If Lix processes abort on your machine, this is a bug. Please file a bug,
ideally with the core dump (or information from it).
On Linux, run `coredumpctl list`, find the crashed process's PID at
the bottom of the list, then run `coredumpctl info THE-PID`. You can then paste
the output into a bug report.
On macOS, open the Console app from Applications/Utilities, select Crash
Reports, select the crash report in question. Right click on it, select Open In
Finder, then include that file in your bug report. [See the Apple
documentation][apple-crashreport] for more details.
[apple-crashreport]: https://developer.apple.com/documentation/xcode/acquiring-crash-reports-and-diagnostic-logs#Locate-crash-reports-and-memory-logs-on-the-device

View file

@ -0,0 +1,12 @@
---
synopsis: rename 'nix show-config' to 'nix config show'
issues: 7672
prs: 9477
cls: 993
credits: [thufschmitt, ma27]
category: Improvements
---
`nix show-config` was renamed to `nix config show` to be more consistent with the rest of the command-line interface.
Running `nix show-config` will now print a deprecation warning saying to use `nix config show` instead.

View file

@ -0,0 +1,8 @@
---
synopsis: Fix `nix-env --query --drv-path --json`
prs: 9257
credits: [Artturin, horrors]
category: Fixes
---
Fixed a bug where `nix-env --query` ignored `--drv-path` when `--json` was set.

View file

@ -0,0 +1,37 @@
---
synopsis: "`nix flake check` logs the checks"
issues: 8882
prs: 8893
cls: [259, 260, 261, 262]
credits: [9999years, raito, horrors]
category: Improvements
significance: significant
---
`nix flake check` now logs the checks it runs and the derivations it evaluates:
```
$ nix flake check -v
evaluating flake...
checking flake output 'checks'...
checking derivation 'checks.aarch64-darwin.ghciwatch-tests'...
derivation evaluated to /nix/store/nh7dlvsrhds4cxl91mvgj4h5cbq6skmq-ghciwatch-test-0.3.0.drv
checking derivation 'checks.aarch64-darwin.ghciwatch-clippy'...
derivation evaluated to /nix/store/9cb5a6wmp6kf6hidqw9wphidvb8bshym-ghciwatch-clippy-0.3.0.drv
checking derivation 'checks.aarch64-darwin.ghciwatch-doc'...
derivation evaluated to /nix/store/8brdd3jbawfszpbs7vdpsrhy80as1il8-ghciwatch-doc-0.3.0.drv
checking derivation 'checks.aarch64-darwin.ghciwatch-fmt'...
derivation evaluated to /nix/store/wjhs0l1njl5pyji53xlmfjrlya0wmz8p-ghciwatch-fmt-0.3.0.drv
checking derivation 'checks.aarch64-darwin.ghciwatch-audit'...
derivation evaluated to /nix/store/z0mps8dyj2ds7c0fn0819y5h5611033z-ghciwatch-audit-0.3.0.drv
checking flake output 'packages'...
checking derivation 'packages.aarch64-darwin.default'...
derivation evaluated to /nix/store/41abbdyglw5x9vcsvd89xan3ydjf8d7r-ghciwatch-0.3.0.drv
checking flake output 'apps'...
checking flake output 'devShells'...
checking derivation 'devShells.aarch64-darwin.default'...
derivation evaluated to /nix/store/bc935gz7dylzmcpdb5cczr8gngv8pmdb-nix-shell.drv
running 5 flake checks...
warning: The check omitted these incompatible systems: aarch64-linux, x86_64-darwin, x86_64-linux
Use '--all-systems' to check all.
```

View file

@ -0,0 +1,19 @@
---
synopsis: "Overhaul `nix flake update` and `nix flake lock` UX"
prs: 8817
credits: [iFreilicht, Lunaphied, thufschmitt]
category: Breaking Changes
---
The interface for creating and updating lock files has been overhauled:
- [`nix flake lock`](@docroot@/command-ref/new-cli/nix3-flake-lock.md) only creates lock files and adds missing inputs now.
It will *never* update existing inputs.
- [`nix flake update`](@docroot@/command-ref/new-cli/nix3-flake-update.md) does the same, but *will* update inputs.
- Passing no arguments will update all inputs of the current flake, just like it already did.
- Passing input names as arguments will ensure only those are updated. This replaces the functionality of `nix flake lock --update-input`
- To operate on a flake outside the current directory, you must now pass `--flake path/to/flake`.
- The flake-specific flags `--recreate-lock-file` and `--update-input` have been removed from all commands operating on installables.
They are superceded by `nix flake update`.

View file

@ -0,0 +1,11 @@
---
synopsis: "`nix profile` now allows referring to elements by human-readable name, and no longer accepts indices"
prs: 8678
cls: [978, 980]
category: Breaking Changes
credits: [iFreilicht, Qyriad, edolstra]
---
[`nix profile`](@docroot@/command-ref/new-cli/nix3-profile.md) now uses names to refer to installed packages when running [`list`](@docroot@/command-ref/new-cli/nix3-profile-list.md), [`remove`](@docroot@/command-ref/new-cli/nix3-profile-remove.md) or [`upgrade`](@docroot@/command-ref/new-cli/nix3-profile-upgrade.md) as opposed to indices. Indices have been removed. Profile element names are generated when a package is installed and remain the same until the package is removed.
**Warning**: The `manifest.nix` file used to record the contents of profiles has changed. Lix will automatically upgrade profiles to the new version when you modify the profile. After that, the profile can no longer be used by older versions of Lix.

View file

@ -0,0 +1,13 @@
---
synopsis: "`builtins.nixVersion` now returns a fixed value \"2.18.3-lix\""
cls: 558
credits: jade
category: Breaking Changes
---
`builtins.nixVersion` now returns a fixed value `"2.18.3-lix"`. This prevents
feature detection assuming that features that exist in Nix post-Lix-branch-off
might exist, even though the Lix version is greater than the Nix version.
In the future, check for builtins for feature detection. If a feature cannot be
detected by *those* means, please file a Lix bug.

View file

@ -0,0 +1,10 @@
---
synopsis: re-evaluate cached evaluation errors
cls: 771
credits: Qyriad
category: Fixes
---
"cached failure of [expr]" errors have been removed: expressions already in the
eval cache as a failure will now simply be re-evaluated, removing the need to
set `--no-eval-cache` or similar to see the error.

View file

@ -0,0 +1,26 @@
---
synopsis: Coercion errors include the failing value
issues: 561
prs: 9754
credits: [9999years, horrors]
category: Improvements
---
The `error: cannot coerce a <TYPE> to a string` message now includes the value
which caused the error.
Before:
```
error: cannot coerce a set to a string
```
After:
```
error: cannot coerce a set to a string: { aesSupport = «thunk»;
avx2Support = «thunk»; avx512Support = «thunk»; avxSupport = «thunk»;
canExecute = «thunk»; config = «thunk»; darwinArch = «thunk»; darwinMinVersion
= «thunk»; darwinMinVersionVariable = «thunk»; darwinPlatform = «thunk»; «84
attributes elided»}
```

View file

@ -0,0 +1,25 @@
---
synopsis: Type errors include the failing value
issues: 561
prs: 9753
credits: [9999years, horrors]
category: Improvements
---
In errors like `value is an integer while a list was expected`, the message now
includes the failing value.
Before:
```
error: value is a set while a string was expected
```
After:
```
error: expected a string but found a set: { ghc810 = «thunk»;
ghc8102Binary = «thunk»; ghc8107 = «thunk»; ghc8107Binary = «thunk»;
ghc865Binary = «thunk»; ghc90 = «thunk»; ghc902 = «thunk»; ghc92 = «thunk»;
ghc924Binary = «thunk»; ghc925 = «thunk»; «17 attributes elided»}
```

View file

@ -0,0 +1,39 @@
---
synopsis: "Visual clutter in `--debugger` is reduced"
prs: 9919
category: Improvements
credits: [9999years, horrors]
---
Before:
```
info: breakpoint reached
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
nix-repl> :continue
error: uh oh
Starting REPL to allow you to inspect the current state of the evaluator.
Welcome to Nix 2.20.0pre20231222_dirty. Type :? for help.
nix-repl>
```
After:
```
info: breakpoint reached
Nix 2.20.0pre20231222_dirty debugger
Type :? for help.
nix-repl> :continue
error: uh oh
nix-repl>
```

View file

@ -1,30 +0,0 @@
---
synopsis: Relative and tilde paths in configuration
issues: [fj#482]
cls: [1851, 1863, 1864]
category: Features
credits: [9999years]
---
[Configuration settings](@docroot@/command-ref/conf-file.md) can now refer to
files with paths relative to the file they're written in or relative to your
home directory (with `~/`).
This makes settings like
[`repl-overlays`](@docroot@/command-ref/conf-file.md#conf-repl-overlays) and
[`secret-key-files`](@docroot@/command-ref/conf-file.md#conf-repl-overlays)
much easier to set, especially if you'd like to refer to files in an existing
dotfiles repo cloned into your home directory.
If you put `repl-overlays = repl.nix` in your `~/.config/nix/nix.conf`, it'll
load `~/.config/nix/repl.nix`. Similarly, you can set `repl-overlays =
~/.dotfiles/repl.nix` to load a file relative to your home directory.
Configuration files can also
[`include`](@docroot@/command-ref/conf-file.md#file-format) paths relative to
your home directory.
Only user configuration files (like `$XDG_CONFIG_HOME/nix/nix.conf` or the
files listed in `$NIX_USER_CONF_FILES`) can use tilde paths relative to your
home directory. Configuration listed in the `$NIX_CONFIG` environment variable
may not use relative paths.

View file

@ -0,0 +1,16 @@
---
synopsis: Experimental REPL support for documentation comments using `:doc`
cls: 564
category: Features
credits: [Lunaphied, jade]
significance: significant
---
Using `:doc` in the REPL now supports showing documentation comments when defined on a function.
Previously this was only able to document builtins, however it now will show comments defined on a lambda as well.
This support is experimental and relies on an embedded version of [nix-doc](https://github.com/lf-/nix-doc).
The logic also supports limited Markdown formatting of doccomments and should easily support any [RFC 145](https://github.com/NixOS/rfcs/blob/master/rfcs/0145-doc-strings.md)
compatible documentation comments in addition to simple commented documentation.

View file

@ -0,0 +1,8 @@
---
synopsis: Interrupting builds in the REPL works more than once
cls: 1097
---
Builds in the REPL can be interrupted by pressing Ctrl+C.
Previously, this only worked once per REPL session; further attempts would be ignored.
This issue is now fixed, so that builds can be canceled consistently.

View file

@ -0,0 +1,39 @@
---
synopsis: Add `repl-overlays` option
prs: 10203
cls: 504
credits: 9999years
significance: significant
category: Features
---
A `repl-overlays` option has been added, which specifies files that can overlay
and modify the top-level bindings in `nix repl`. For example, with the
following contents in `~/.config/nix/repl.nix`:
```nix
info: final: prev: let
optionalAttrs = predicate: attrs:
if predicate
then attrs
else {};
in
optionalAttrs (prev ? legacyPackages && prev.legacyPackages ? ${info.currentSystem})
{
pkgs = prev.legacyPackages.${info.currentSystem};
}
```
We can run `nix repl` and use `pkgs` to refer to `legacyPackages.${currentSystem}`:
```ShellSession
$ nix repl --repl-overlays ~/.config/nix/repl.nix nixpkgs
Lix 2.90.0
Type :? for help.
Loading installable 'flake:nixpkgs#'...
Added 5 variables.
Loading 'repl-overlays'...
Added 6 variables.
nix-repl> pkgs.bash
«derivation /nix/store/g08b5vkwwh0j8ic9rkmd8mpj878rk62z-bash-5.2p26.drv»
```

View file

@ -0,0 +1,16 @@
---
synopsis: reintroduce shortened `-E` form for `--expr` to new CLI
cls: 605
credits: Lunaphied
category: Improvements
---
In the old CLI, it was possible to supply a shorter `-E` flag instead of fully
specifying `--expr` every time you wished to provide an expression that would
be evaluated to produce the given command's input. This was retained for the
`--file` flag when the new CLI utilities were written with `-f`, but `-E` was
dropped.
We now restore the `-E` short form for better UX. This is most useful for
`nix eval` but most any command that takes an Installable argument should benefit
from it as well.

View file

@ -0,0 +1,25 @@
---
synopsis: "In the debugger, `while evaluating the attribute` errors now include position information"
prs: 9915
credits: 9999years
category: Fixes
---
Before:
```
0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
0x600001522598
```
After:
```
0: while evaluating the attribute 'python311.pythonForBuild.pkgs'
/nix/store/hg65h51xnp74ikahns9hyf3py5mlbbqq-source/overrides/default.nix:132:27
131|
132| bootstrappingBase = pkgs.${self.python.pythonAttr}.pythonForBuild.pkgs;
| ^
133| in
```

View file

@ -0,0 +1,44 @@
---
synopsis: Source locations are printed more consistently in errors
issues: 561
prs: 9555
credits: [9999years, horrors]
category: Improvements
---
Source location information is now included in error messages more
consistently. Given this code:
```nix
let
attr = {foo = "bar";};
key = {};
in
attr.${key}
```
Previously, Nix would show this unhelpful message when attempting to evaluate
it:
```
error:
… while evaluating an attribute name
error: value is a set while a string was expected
```
Now, the error message displays where the problematic value was found:
```
error:
… while evaluating an attribute name
at bad.nix:4:11:
3| key = {};
4| in attr.${key}
| ^
5|
error: expected a string but found a set: { }
```

View file

@ -0,0 +1,35 @@
---
synopsis: Some stack overflow segfaults are fixed
issues: 9616
prs: 9617
cls: 205
category: Improvements
credits: [9999years, horrors]
---
The number of nested function calls has been restricted, to detect and report
infinite function call recursions. The default maximum call depth is 10,000 and
can be set with [the `max-call-depth`
option](@docroot@/command-ref/conf-file.md#conf-max-call-depth).
This fixes segfaults or the following unhelpful error message in many cases:
error: stack overflow (possible infinite recursion)
Before:
```
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
Segmentation fault: 11
```
After:
```
$ nix-instantiate --eval --expr '(x: x x) (x: x x)'
error: stack overflow
at «string»:1:14:
1| (x: x x) (x: x x)
| ^
```

View file

@ -0,0 +1,8 @@
---
synopsis: add `--store-path` argument to `nix upgrade-nix`, to manually specify the Nix to upgrade to
cls: 953
credits: Qyriad
category: Features
---
`nix upgrade-nix` by default downloads a manifest to find the new Nix version to upgrade to, but now you can specify `--store-path` to upgrade Nix to an arbitrary version from the Nix store.

View file

@ -0,0 +1,10 @@
---
synopsis: using `nix profile` on `/nix/var/nix/profiles/default` no longer breaks `nix upgrade-nix`
cls: 952
credits: Qyriad
category: Fixes
---
On non-NixOS, Nix is conventionally installed into a `nix-env` style profile at /nix/var/nix/profiles/default.
Like any `nix-env` profile, using `nix profile` on it automatically migrates it to a `nix profile` style profile, which is incompatible with `nix-env`.
`nix upgrade-nix` previously relied solely on `nix-env` to do the upgrade, but now will work fine with either kind of profile.

View file

@ -0,0 +1,10 @@
---
synopsis: Upstart scripts removed
cls: 574
category: Packaging
credits: jade
---
Upstart scripts have been removed from Lix, since Upstart is obsolete and has
not been shipped by any major distributions for many years. If these are
necessary to your use case, please back port them to your packaging.

View file

@ -0,0 +1,34 @@
---
synopsis: Better error reporting for `with` expressions
prs: 9658
cls: 207
credits: horrors
category: Improvements
---
`with` expressions using non-attrset values to resolve variables are now reported with proper positions.
Previously an incorrect `with` expression would report no position at all, making it hard to determine where the error originated:
```
nix-repl> with 1; a
error:
<borked>
at «none»:0: (source not available)
error: value is an integer while a set was expected
```
Now position information is preserved and reported as with most other errors:
```
nix-repl> with 1; a
error:
… while evaluating the first subexpression of a with expression
at «string»:1:1:
1| with 1; a
| ^
error: expected a set but found an integer: 1
```

View file

@ -192,61 +192,57 @@
- [Hacking](contributing/hacking.md) - [Hacking](contributing/hacking.md)
- [Testing](contributing/testing.md) - [Testing](contributing/testing.md)
- [Experimental Features](contributing/experimental-features.md) - [Experimental Features](contributing/experimental-features.md)
- [Deprecated Features](contributing/deprecated-features.md)
- [CLI guideline](contributing/cli-guideline.md) - [CLI guideline](contributing/cli-guideline.md)
- [C++ style guide](contributing/cxx.md) - [C++ style guide](contributing/cxx.md)
- [Release Notes](release-notes/release-notes.md) - [Release Notes](release-notes/release-notes.md)
- [Upcoming release](release-notes/rl-next.md) - [Upcoming release](release-notes/rl-next.md)
<!-- RELENG-AUTO-INSERTION-MARKER (see releng/release_notes.py) --> - [Release 2.18 (2023-09-20)](release-notes/rl-2.18.md)
- [Lix 2.91 (2024-08-12)](release-notes/rl-2.91.md) - [Release 2.17 (2023-07-24)](release-notes/rl-2.17.md)
- [Lix 2.90 (2024-07-10)](release-notes/rl-2.90.md) - [Release 2.16 (2023-05-31)](release-notes/rl-2.16.md)
- [Nix 2.18 (2023-09-20)](release-notes/rl-2.18.md) - [Release 2.15 (2023-04-11)](release-notes/rl-2.15.md)
- [Nix 2.17 (2023-07-24)](release-notes/rl-2.17.md) - [Release 2.14 (2023-02-28)](release-notes/rl-2.14.md)
- [Nix 2.16 (2023-05-31)](release-notes/rl-2.16.md) - [Release 2.13 (2023-01-17)](release-notes/rl-2.13.md)
- [Nix 2.15 (2023-04-11)](release-notes/rl-2.15.md) - [Release 2.12 (2022-12-06)](release-notes/rl-2.12.md)
- [Nix 2.14 (2023-02-28)](release-notes/rl-2.14.md) - [Release 2.11 (2022-08-25)](release-notes/rl-2.11.md)
- [Nix 2.13 (2023-01-17)](release-notes/rl-2.13.md) - [Release 2.10 (2022-07-11)](release-notes/rl-2.10.md)
- [Nix 2.12 (2022-12-06)](release-notes/rl-2.12.md) - [Release 2.9 (2022-05-30)](release-notes/rl-2.9.md)
- [Nix 2.11 (2022-08-25)](release-notes/rl-2.11.md) - [Release 2.8 (2022-04-19)](release-notes/rl-2.8.md)
- [Nix 2.10 (2022-07-11)](release-notes/rl-2.10.md) - [Release 2.7 (2022-03-07)](release-notes/rl-2.7.md)
- [Nix 2.9 (2022-05-30)](release-notes/rl-2.9.md) - [Release 2.6 (2022-01-24)](release-notes/rl-2.6.md)
- [Nix 2.8 (2022-04-19)](release-notes/rl-2.8.md) - [Release 2.5 (2021-12-13)](release-notes/rl-2.5.md)
- [Nix 2.7 (2022-03-07)](release-notes/rl-2.7.md) - [Release 2.4 (2021-11-01)](release-notes/rl-2.4.md)
- [Nix 2.6 (2022-01-24)](release-notes/rl-2.6.md) - [Release 2.3 (2019-09-04)](release-notes/rl-2.3.md)
- [Nix 2.5 (2021-12-13)](release-notes/rl-2.5.md) - [Release 2.2 (2019-01-11)](release-notes/rl-2.2.md)
- [Nix 2.4 (2021-11-01)](release-notes/rl-2.4.md) - [Release 2.1 (2018-09-02)](release-notes/rl-2.1.md)
- [Nix 2.3 (2019-09-04)](release-notes/rl-2.3.md) - [Release 2.0 (2018-02-22)](release-notes/rl-2.0.md)
- [Nix 2.2 (2019-01-11)](release-notes/rl-2.2.md) - [Release 1.11.10 (2017-06-12)](release-notes/rl-1.11.10.md)
- [Nix 2.1 (2018-09-02)](release-notes/rl-2.1.md) - [Release 1.11 (2016-01-19)](release-notes/rl-1.11.md)
- [Nix 2.0 (2018-02-22)](release-notes/rl-2.0.md) - [Release 1.10 (2015-09-03)](release-notes/rl-1.10.md)
- [Nix 1.11.10 (2017-06-12)](release-notes/rl-1.11.10.md) - [Release 1.9 (2015-06-12)](release-notes/rl-1.9.md)
- [Nix 1.11 (2016-01-19)](release-notes/rl-1.11.md) - [Release 1.8 (2014-12-14)](release-notes/rl-1.8.md)
- [Nix 1.10 (2015-09-03)](release-notes/rl-1.10.md) - [Release 1.7 (2014-04-11)](release-notes/rl-1.7.md)
- [Nix 1.9 (2015-06-12)](release-notes/rl-1.9.md) - [Release 1.6.1 (2013-10-28)](release-notes/rl-1.6.1.md)
- [Nix 1.8 (2014-12-14)](release-notes/rl-1.8.md) - [Release 1.6 (2013-09-10)](release-notes/rl-1.6.md)
- [Nix 1.7 (2014-04-11)](release-notes/rl-1.7.md) - [Release 1.5.2 (2013-05-13)](release-notes/rl-1.5.2.md)
- [Nix 1.6.1 (2013-10-28)](release-notes/rl-1.6.1.md) - [Release 1.5 (2013-02-27)](release-notes/rl-1.5.md)
- [Nix 1.6 (2013-09-10)](release-notes/rl-1.6.md) - [Release 1.4 (2013-02-26)](release-notes/rl-1.4.md)
- [Nix 1.5.2 (2013-05-13)](release-notes/rl-1.5.2.md) - [Release 1.3 (2013-01-04)](release-notes/rl-1.3.md)
- [Nix 1.5 (2013-02-27)](release-notes/rl-1.5.md) - [Release 1.2 (2012-12-06)](release-notes/rl-1.2.md)
- [Nix 1.4 (2013-02-26)](release-notes/rl-1.4.md) - [Release 1.1 (2012-07-18)](release-notes/rl-1.1.md)
- [Nix 1.3 (2013-01-04)](release-notes/rl-1.3.md) - [Release 1.0 (2012-05-11)](release-notes/rl-1.0.md)
- [Nix 1.2 (2012-12-06)](release-notes/rl-1.2.md) - [Release 0.16 (2010-08-17)](release-notes/rl-0.16.md)
- [Nix 1.1 (2012-07-18)](release-notes/rl-1.1.md) - [Release 0.15 (2010-03-17)](release-notes/rl-0.15.md)
- [Nix 1.0 (2012-05-11)](release-notes/rl-1.0.md) - [Release 0.14 (2010-02-04)](release-notes/rl-0.14.md)
- [Nix 0.16 (2010-08-17)](release-notes/rl-0.16.md) - [Release 0.13 (2009-11-05)](release-notes/rl-0.13.md)
- [Nix 0.15 (2010-03-17)](release-notes/rl-0.15.md) - [Release 0.12 (2008-11-20)](release-notes/rl-0.12.md)
- [Nix 0.14 (2010-02-04)](release-notes/rl-0.14.md) - [Release 0.11 (2007-12-31)](release-notes/rl-0.11.md)
- [Nix 0.13 (2009-11-05)](release-notes/rl-0.13.md) - [Release 0.10.1 (2006-10-11)](release-notes/rl-0.10.1.md)
- [Nix 0.12 (2008-11-20)](release-notes/rl-0.12.md) - [Release 0.10 (2006-10-06)](release-notes/rl-0.10.md)
- [Nix 0.11 (2007-12-31)](release-notes/rl-0.11.md) - [Release 0.9.2 (2005-09-21)](release-notes/rl-0.9.2.md)
- [Nix 0.10.1 (2006-10-11)](release-notes/rl-0.10.1.md) - [Release 0.9.1 (2005-09-20)](release-notes/rl-0.9.1.md)
- [Nix 0.10 (2006-10-06)](release-notes/rl-0.10.md) - [Release 0.9 (2005-09-16)](release-notes/rl-0.9.md)
- [Nix 0.9.2 (2005-09-21)](release-notes/rl-0.9.2.md) - [Release 0.8.1 (2005-04-13)](release-notes/rl-0.8.1.md)
- [Nix 0.9.1 (2005-09-20)](release-notes/rl-0.9.1.md) - [Release 0.8 (2005-04-11)](release-notes/rl-0.8.md)
- [Nix 0.9 (2005-09-16)](release-notes/rl-0.9.md) - [Release 0.7 (2005-01-12)](release-notes/rl-0.7.md)
- [Nix 0.8.1 (2005-04-13)](release-notes/rl-0.8.1.md) - [Release 0.6 (2004-11-14)](release-notes/rl-0.6.md)
- [Nix 0.8 (2005-04-11)](release-notes/rl-0.8.md) - [Release 0.5 and earlier](release-notes/rl-0.5.md)
- [Nix 0.7 (2005-01-12)](release-notes/rl-0.7.md)
- [Nix 0.6 (2004-11-14)](release-notes/rl-0.6.md)
- [Nix 0.5 and earlier](release-notes/rl-0.5.md)

View file

@ -1,37 +1,23 @@
xp_features_json = custom_target(
command : [nix, '__dump-xp-features'],
capture : true,
output : 'xp-features.json',
)
experimental_features_shortlist_md = custom_target( experimental_features_shortlist_md = custom_target(
command : nix_eval_for_docs + [ command : nix_eval_for_docs + [
'--expr', '--expr',
'import @INPUT0@ "experimental" "xp" (builtins.fromJSON (builtins.readFile @INPUT1@))', 'import @INPUT0@ (builtins.fromJSON (builtins.readFile @INPUT1@))',
], ],
input : [ input : [
'../../generate-features-shortlist.nix', '../../generate-xp-features-shortlist.nix',
nix_exp_features_json, xp_features_json,
], ],
capture : true, capture : true,
output : 'experimental-features-shortlist.md', output : 'experimental-features-shortlist.md',
env : nix_env_for_docs, env : nix_env_for_docs,
) )
dp_features_json = custom_target(
command : [nix, '__dump-dp-features'],
capture : true,
output : 'dp-features.json',
)
deprecated_features_shortlist_md = custom_target(
command : nix_eval_for_docs + [
'--expr',
'import @INPUT0@ "deprecated" "dp" (builtins.fromJSON (builtins.readFile @INPUT1@))',
],
input : [
'../../generate-features-shortlist.nix',
dp_features_json,
],
capture : true,
output : 'deprecated-features-shortlist.md',
env : nix_env_for_docs,
)
# Intermediate step for manpage generation. # Intermediate step for manpage generation.
# This splorks the output of generate-manpage.nix as JSON, # This splorks the output of generate-manpage.nix as JSON,
# which gets written as a directory tree below. # which gets written as a directory tree below.
@ -74,7 +60,6 @@ conf_file_md = custom_target(
'../../utils.nix', '../../utils.nix',
conf_file_json, conf_file_json,
experimental_features_shortlist_md, experimental_features_shortlist_md,
deprecated_features_shortlist_md,
], ],
output : 'conf-file.md', output : 'conf-file.md',
env : nix_env_for_docs, env : nix_env_for_docs,

View file

@ -14,8 +14,9 @@
# Disambiguation # Disambiguation
This man page describes the command `nix-build`, which is distinct from [`nix build`](./new-cli/nix3-build.md). This man page describes the command `nix-build`, which is distinct from `nix
For documentation on the latter, run `nix build --help` or see `man nix3-build`. build`. For documentation on the latter, run `nix build --help` or see `man
nix3-build`.
# Description # Description

View file

@ -78,16 +78,6 @@ Most commands in Lix accept the following command-line options:
Display the raw logs, with the progress bar at the bottom. Display the raw logs, with the progress bar at the bottom.
- `multiline`
Display a progress bar during the builds and in the lines below that one line per activity.
- `multiline-with-logs`
Displayes the raw logs, with a progress bar and activities each in a new line at the bottom.
- <span id="opt-no-build-output">[`--no-build-output`](#opt-no-build-output)</span> / `-Q` - <span id="opt-no-build-output">[`--no-build-output`](#opt-no-build-output)</span> / `-Q`
By default, output written by builders to standard output and standard error is echoed to the Lix command's standard error. By default, output written by builders to standard output and standard error is echoed to the Lix command's standard error.

View file

@ -1,37 +0,0 @@
This section describes the notion of *deprecated features*, and how it fits into the big picture of the development of Lix.
# What are deprecated features?
Deprecated features are disabled by default, with the intent to eventually remove them.
Users must explicitly enable them to keep using them, by toggling the associated [deprecated feature flags](@docroot@/command-ref/conf-file.md#conf-deprecated-features).
This allows backwards compatibility and a graceful transition away from undesired features.
# Which features can be deprecated?
Undesired features should be soft-deprecated by yielding a warning when used for a significant amount of time before the can be deprecated.
Legacy obsolete feature with little to no usage may go through this process faster.
Deprecated features should have a migration path to a preferred alternative.
# Lifecycle of a deprecated feature
This description is not normative, but a feature removal may roughly happen like this:
1. Add a warning when the feature is being used.
2. Disable the feature by default, putting it behind a deprecated feature flag.
- If disabling the feature started out as an opt-in experimental feature, turn that experimental flag into a no-op or remove it entirely.
For example, `--extra-experimental-features=no-url-literals` becomes `--extra-deprecated-features=url-literals`.
3. Decide on a time frame for how long that feature will still be supported for backwards compatibility, and clearly communicate that in the error messages.
- Sometimes, automatic migration to alternatives is possible, and such should be provided if possible
- At least one NixOS release cycle should be the minimum
4. Finally remove the feature entirely, only keeping the error message for those still using it.
# Relation to language versioning
Obviously, removing anything breaks backwards compatibility.
In an ideal world, we'd have SemVer controls over the language and its features, cleanly allowing us to make breaking changes.
See https://wiki.lix.systems/books/lix-contributors/page/language-versioning and [RFC 137](https://github.com/nixos/rfcs/pull/137) for efforts on that.
However, we do not live in such an ideal world, and currently this goal is so far away, that "just disable it with some back-compat for a couple of years" is the most realistic solution, especially for comparatively minor changes.
# Currently available deprecated features
{{#include @generated@/contributing/deprecated-feature-descriptions.md}}

View file

@ -39,19 +39,17 @@ $ nix-shell -A native-clangStdenvPackages
### Building from the development shell ### Building from the development shell
You can build and test Lix with just: As always you may run [stdenv's phases by name](https://nixos.org/manual/nixpkgs/unstable/#sec-building-stdenv-package-in-nix-shell), e.g.:
```bash ```bash
$ just setup $ configurePhase
$ just build $ buildPhase
$ just test --suite=check $ checkPhase
$ just install $ installPhase
$ just test --suite=installcheck $ installCheckPhase
``` ```
(Check and installcheck may both be done after install, allowing you to omit the --suite argument entirely, but this is the order package.nix runs them in.) To build manually, however, use the following:
You can also build Lix manually:
```bash ```bash
$ meson setup ./build "--prefix=$out" $mesonFlags $ meson setup ./build "--prefix=$out" $mesonFlags
@ -66,7 +64,9 @@ $ meson install -C build
$ meson test -C build --suite=installcheck $ meson test -C build --suite=installcheck
``` ```
In both cases, Lix will be installed to `$PWD/outputs`, the `/bin` of which is prepended to PATH in the development shells. (Check and installcheck may both be done after install, allowing you to omit the --suite argument entirely, but this is the order package.nix runs them in.)
This will install Lix to `$PWD/outputs`, the `/bin` of which is prepended to PATH in the development shells.
If the tests fail and Meson helpfully has no output for why, use the `--print-error-logs` option to `meson test`. If the tests fail and Meson helpfully has no output for why, use the `--print-error-logs` option to `meson test`.
@ -102,14 +102,14 @@ $ meson compile -C build nixexpr
All targets may be addressed as their output, relative to the build directory, e.g.: All targets may be addressed as their output, relative to the build directory, e.g.:
```bash ```bash
$ meson compile -C build src/libexpr/liblixexpr.so $ meson compile -C build src/libexpr/libnixexpr.so
``` ```
But Meson does not consider intermediate files like object files targets. But Meson does not consider intermediate files like object files targets.
To build a specific object file, use Ninja directly and specify the output file relative to the build directory: To build a specific object file, use Ninja directly and specify the output file relative to the build directory:
```bash ```bash
$ ninja -C build src/libexpr/liblixexpr.so.p/nixexpr.cc.o $ ninja -C build src/libexpr/libnixexpr.so.p/nixexpr.cc.o
``` ```
To inspect the canonical source of truth on what the state of the buildsystem configuration is, use: To inspect the canonical source of truth on what the state of the buildsystem configuration is, use:
@ -137,7 +137,7 @@ You can also build Lix for one of the [supported platforms](#platforms).
Lix can be built for various platforms, as specified in [`flake.nix`]: Lix can be built for various platforms, as specified in [`flake.nix`]:
[`flake.nix`]: https://git.lix.systems/lix-project/lix/src/branch/main/flake.nix [`flake.nix`]: https://github.com/nixos/nix/blob/master/flake.nix
- `x86_64-linux` - `x86_64-linux`
- `x86_64-darwin` - `x86_64-darwin`
@ -168,27 +168,8 @@ or for Nix with the [`flakes`] and [`nix-command`] experimental features enabled
$ nix build .#packages.aarch64-linux.default $ nix build .#packages.aarch64-linux.default
``` ```
### Cross compiling using the Lix flake Cross-compiled builds are available for ARMv6 (`armv6l-linux`) and ARMv7 (`armv7l-linux`).
Add more [system types](#system-type) to `crossSystems` in `flake.nix` to bootstrap Nix on unsupported platforms.
Lix can also be easily cross compiled to the following arbitrarily-chosen system doubles, which can be useful for bootstrapping Lix on new platforms.
These are specified in `crossSystems` in `flake.nix`; feel free to submit changes to add new ones if they are useful to you.
- `armv6l-linux`
- `armv7l-linux`
- `aarch64-linux`
- `riscv64-linux`
For example, to cross-compile Lix for `armv6l-linux` from another Linux, use the following:
```console
$ nix build .#nix-armv6l-linux
```
It's also possible to cross-compile a tarball of binaries suitable for the Lix installer, for example, for `riscv64-linux`:
```console
$ nix build .#nix-riscv64-linux.passthru.binaryTarball
```
### Building for multiple platforms at once ### Building for multiple platforms at once
@ -301,8 +282,9 @@ Regular markdown files used for the manual have a base path of their own and the
## API documentation ## API documentation
Doxygen API documentation will be available online in the future ([tracking issue](https://git.lix.systems/lix-project/lix/issues/422)). Doxygen API documentation is [available
You can also build and view it yourself: online](https://hydra.nixos.org/job/nix/master/internal-api-docs/latest/download-by-type/doc/internal-api-docs). You
can also build and view it yourself:
```console ```console
# nix build .#hydraJobs.internal-api-docs # nix build .#hydraJobs.internal-api-docs
@ -312,50 +294,44 @@ You can also build and view it yourself:
or inside a `nix develop` shell by running: or inside a `nix develop` shell by running:
```bash ```bash
$ meson configure build -Dinternal-api-docs=enabled
$ meson compile -C build internal-api-docs $ meson compile -C build internal-api-docs
$ xdg-open ./outputs/doc/share/doc/nix/internal-api/html/index.html $ xdg-open ./outputs/doc/share/doc/nix/internal-api/html/index.html
``` ```
## Coverage analysis ## Coverage analysis
A coverage analysis report will be available online in the future (FIXME(lix-hydra)). A coverage analysis report is [available
You can build it yourself: online](https://hydra.nixos.org/job/nix/master/coverage/latest/download-by-type/report/coverage). You
can build it yourself:
``` ```
# nix build .#hydraJobs.coverage # nix build .#hydraJobs.coverage
# xdg-open ./result/coverage/index.html # xdg-open ./result/coverage/index.html
``` ```
Metrics about the change in line/function coverage over time will be available in the future (FIXME(lix-hydra)). Metrics about the change in line/function coverage over time are also
[available](https://hydra.nixos.org/job/nix/master/coverage#tabs-charts).
## Add a release note ## Add a release note
`doc/manual/rl-next` contains release notes entries for all unreleased changes. `doc/manual/rl-next` contains release notes entries for all unreleased changes.
User-visible changes should come with a release note. User-visible changes should come with a release note.
Developer-facing changes should have a release note in the Development category if they are significant and if developers should know about them.
### Add an entry ### Add an entry
Here's what a complete entry looks like. Here's what a complete entry looks like. The file name is not incorporated in the document.
The file name is not incorporated in the final document, and is generally a super brief summary of the change synopsis.
```markdown ```
--- ---
synopsis: Basically a title synopsis: Basically a title
# 1234 or gh#1234 will refer to CppNix GitHub, fj#1234 will refer to a Lix forgejo issue. # 1234 or gh#1234 will refer to CppNix GitHub, fj#1234 will refer to a Lix forgejo issue.
issues: [1234, fj#1234] issues: [1234, fj#1234]
# Use this *only* if there is a CppNix pull request associated with this change. # Use this *only* if there is a CppNix pull request associated with this change
prs: 1238 prs: 1238
# List of Lix Gerrit changelist numbers. # List of Lix Gerrit changelist numbers; if there is an associated Lix GitHub
# If there is an associated Lix GitHub PR, just put in the Gerrit CL number. # PR, just put in the Gerrit CL number.
cls: [123] cls: [123]
# Heading that this release note will appear under.
category: Breaking Changes
# Add a credit mention in the bottom of the release note.
# your-name is used as a key into doc/manual/change-authors.yml for metadata
credits: [your-name]
--- ---
Here's one or more paragraphs that describe the change. Here's one or more paragraphs that describe the change.
@ -370,31 +346,6 @@ Significant changes should add the following header, which moves them to the top
significance: significant significance: significant
``` ```
The following categories of release notes are supported (see `maintainers/build-release-notes.py`):
- Breaking Changes
- Features
- Improvements
- Fixes
- Packaging
- Development
- Miscellany
The `credits` field, if present, gives credit to the author of the patch in the release notes with a message like "Many thanks to (your-name) for this" and linking to GitHub or Forgejo profiles if listed.
If you are forward-porting a change from CppNix, please credit the original author, and optionally credit yourself.
When adding credits metadata for people external to the project and deciding whether to put in a `display_name`, consider what they are generally known as in the community; even if you know their full name (e.g. from their GitHub profile), we suggest only adding it as a display name if that is what they go by in the community.
There are multiple reasons we follow this practice, but it boils down to privacy and consent: we would rather not capture full names that are not widely used in the community without the consent of the parties involved, even if they are publicly available.
As of this writing, the entries with full names as `display_name` are either members of the CppNix team or people who added them themselves.
The names specified in `credits` are used as keys to look up the authorship info in `doc/manual/change-authors.yml`.
The only mandatory part is that every key appearing in `credits` has an entry present in `change-authors.yml`.
All of the following properties are optional; you can specify `{}` as the metadata if you want a simple non-hyperlinked mention.
The following properties are supported:
- `display_name`: display name used in place of the key when showing names, if present.
- `forgejo`: Forgejo username. The name in the release notes will be a link to this, if present.
- `github`: GitHub username, used if `forgejo` is not set, again making a link.
### Build process ### Build process
Releases have a precomputed `rl-MAJOR.MINOR.md`, and no `rl-next.md`. Releases have a precomputed `rl-MAJOR.MINOR.md`, and no `rl-next.md`.

View file

@ -4,25 +4,12 @@
experimental_feature_descriptions_md = custom_target( experimental_feature_descriptions_md = custom_target(
command : nix_eval_for_docs + [ command : nix_eval_for_docs + [
'--expr', '--expr',
'import @INPUT0@ "experimental" "xp" (builtins.fromJSON (builtins.readFile @INPUT1@))', 'import @INPUT0@ (builtins.fromJSON (builtins.readFile @INPUT1@))',
], ],
input : [ input : [
'../../generate-features.nix', '../../generate-xp-features.nix',
nix_exp_features_json, xp_features_json,
], ],
capture : true, capture : true,
output : 'experimental-feature-descriptions.md', output : 'experimental-feature-descriptions.md',
) )
deprecated_feature_descriptions_md = custom_target(
command : nix_eval_for_docs + [
'--expr',
'import @INPUT0@ "deprecated" "dp" (builtins.fromJSON (builtins.readFile @INPUT1@))',
],
input : [
'../../generate-features.nix',
dp_features_json,
],
capture : true,
output : 'deprecated-feature-descriptions.md',
)

View file

@ -12,51 +12,44 @@ The unit tests are defined using the [googletest] and [rapidcheck] frameworks.
> An example of some files, demonstrating much of what is described below > An example of some files, demonstrating much of what is described below
> >
> ``` > ```
> src
> ├── libexpr
> │ ├── value/context.hh
> │ ├── value/context.cc
> │ │
> │ …
> └── tests
> │ ├── value/context.hh
> │ ├── value/context.cc
> │ │
> │ …
> │
> ├── unit-test-data
> │ ├── libstore
> │ │ ├── worker-protocol/content-address.bin
> │ │ …
> │ …
> … > …
> ├── src
> │   ├── libexpr
> │   │   ├── …
> │   │   ├── value
> │   │   │   ├── context.cc
> │   │   │   └── context.hh
> │ … …
> ├── tests
> │   …
> │   └── unit
> │   ├── libcmd
> │   │   └── args.cc
> │   ├── libexpr
> │   │   ├── …
> │   │   └── value
> │   │   ├── context.cc
> │   │   └── print.cc
> │   ├── libexpr-support
> │   │   └── tests
> │   │   ├── libexpr.hh
> │   │   └── value
> │   │   ├── context.cc
> │   │   └── context.hh
> │   ├── libstore
> │   │   ├── common-protocol.cc
> │   │   ├── data
> │   │   │   ├── libstore
> │   │   │   │   ├── common-protocol
> │   │   │   │   │   ├── content-address.bin
> │   │   │   │   │   ├── drv-output.bin
> … … … … … …
> ``` > ```
The unit tests for each Lix library (`liblixexpr`, `liblixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`). <!-- FIXME(Lix): this might get renamed to liblixexpr, etc? -->
The data is in `tests/unit/LIBNAME/data/LIBNAME`, with one subdir per library, with the same name as where the code goes. The unit tests for each Lix library (`libnixexpr`, `libnixstore`, etc..) live inside a directory `src/${library_shortname}/tests` within the directory for the library (`src/${library_shortname}`).
For example, `liblixstore` code is in `src/libstore`, and its test data is in `tests/unit/libstore/data/libstore`.
The path to the unit test data directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`. The data is in `unit-test-data`, with one subdir per library, with the same name as where the code goes.
For example, `libnixstore` code is in `src/libstore`, and its test data is in `unit-test-data/libstore`.
The path to the `unit-test-data` directory is passed to the unit test executable with the environment variable `_NIX_TEST_UNIT_DATA`.
> **Note**
> Due to the way googletest works, downstream unit test executables will actually include and re-run upstream library tests.
> Therefore it is important that the same value for `_NIX_TEST_UNIT_DATA` be used with the tests for each library.
> That is why we have the test data nested within a single `unit-test-data` directory.
### Running tests ### Running tests
You can run the whole testsuite with `just test` (see justfile for exact invocation of meson), and if you want to run just one test suite, use `just test --suite installcheck functional-init` where `installcheck` is the name of the test suite in this case and `functional-init` is the name of the test. You can run the whole testsuite with `just test` (see justfile for exact invocation of meson), and if you want to run just one test suite, use `just test --suite installcheck functional-init` where `installcheck` is the name of the test suite in this case and `functional-init` is the name of the test.
To get a list of tests, use `meson test -C build --list` (or `just test --list` for short). To get a list of tests, use `meson test -C build --list`.
For `installcheck` specifically, first run `just install` before running the test suite (this is due to meson limitations that don't let us put a dependency on installing before doing the test). For `installcheck` specifically, first run `just install` before running the test suite (this is due to meson limitations that don't let us put a dependency on installing before doing the test).
@ -81,27 +74,20 @@ See [below](#characterization-testing-1) for a broader discussion of characteriz
Like with the functional characterization, `_NIX_TEST_ACCEPT=1` is also used. Like with the functional characterization, `_NIX_TEST_ACCEPT=1` is also used.
For example: For example:
```shell-session ```shell-session
$ _NIX_TEST_ACCEPT=1 just test --suite check libstore-unit-tests $ _NIX_TEST_ACCEPT=1 make libstore-tests-exe_RUN
... ...
../tests/unit/libstore/common-protocol.cc:27: Skipped [ SKIPPED ] WorkerProtoTest.string_read
Cannot read golden master because another test is also updating it [ SKIPPED ] WorkerProtoTest.string_write
[ SKIPPED ] WorkerProtoTest.storePath_read
../tests/unit/libstore/common-protocol.cc:62: Skipped [ SKIPPED ] WorkerProtoTest.storePath_write
Updating golden master
../tests/unit/libstore/common-protocol.cc:27: Skipped
Cannot read golden master because another test is also updating it
../tests/unit/libstore/common-protocol.cc:62: Skipped
Updating golden master
... ...
``` ```
will regenerate the "golden master" expected result for the `liblixstore` characterization tests. will regenerate the "golden master" expected result for the `libnixstore` characterization tests.
The characterization tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything. The characterization tests will mark themselves "skipped" since they regenerated the expected result instead of actually testing anything.
## Functional tests ## Functional tests
The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/meson.build`. The functional tests reside under the `tests/functional` directory and are listed in `tests/functional/local.mk`.
Each test is a bash script. Each test is a bash script.
### Running the whole test suite ### Running the whole test suite
@ -199,7 +185,7 @@ edit it like so:
``` ```
<div class="warning"> <div class="warning">
FIXME(meson): the command here is incorrect for meson and this whole functionality may need rebuilding. FIXME(meson): the command here may be incorrect for meson.
</div> </div>
Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point: Then, running the test with `./mk/debug-test.sh` will drop you into GDB once the script reaches that point:
@ -223,11 +209,15 @@ This technique is to include the exact output/behavior of a former version of Ni
For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered. For example, this technique is used for the language tests, to check both the printed final value if evaluation was successful, and any errors and warnings encountered.
<div class="warning">
FIXME(meson): this is incorrect for meson. `_NIX_TEST_ACCEPT=1` is still valid but the test invocation needs to change.
</div>
It is frequently useful to regenerate the expected output. It is frequently useful to regenerate the expected output.
To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`. To do that, rerun the failed test(s) with `_NIX_TEST_ACCEPT=1`.
For example: For example:
```bash ```bash
_NIX_TEST_ACCEPT=1 just test --suite installcheck -v functional-lang _NIX_TEST_ACCEPT=1 make tests/functional/lang.sh.test
``` ```
An interesting situation to document is the case when these tests are "overfitted". An interesting situation to document is the case when these tests are "overfitted".
@ -247,6 +237,7 @@ To ensure that characterization testing doesn't make it harder to intentionally
The integration tests are defined in the Nix flake under the `hydraJobs.tests` attribute. The integration tests are defined in the Nix flake under the `hydraJobs.tests` attribute.
These tests include everything that needs to interact with external services or run Lix in a non-trivial distributed setup. These tests include everything that needs to interact with external services or run Lix in a non-trivial distributed setup.
Because these tests are expensive and require more than what the standard github-actions setup provides, they only run on the master branch (on <https://hydra.nixos.org/jobset/nix/master>).
You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix-build -A hydraJobs.tests.{testName}` You can run them manually with `nix build .#hydraJobs.tests.{testName}` or `nix-build -A hydraJobs.tests.{testName}`
@ -329,133 +320,3 @@ solved this need?
~~> ~~>
--> -->
## Magic environment variables
FIXME: maybe this section should be moved elsewhere or turned partially into user docs, but I just need a complete index for now.
I actually want to ban people calling getenv without writing documentation, and produce a comprehensive list of env-vars used by Lix and enforce it.
This is a non-exhaustive list of almost all environment variables, magic or not, accepted or used by various parts of the test suite as well as Lix itself.
Please add more if you find them.
I looked for these in the testsuite with the following bad regexes:
```
rg '(?:[^A-Za-z]|^)(_[A-Z][^-\[ }/:");$(]+)' -r '$1' --no-filename --only-matching tests | sort -u > vars.txt
rg '\$\{?([A-Z][^-\[ }/:");]+)' -r '$1' --no-filename --only-matching tests | sort -u > vars.txt
```
I grepped `src/` for `get[eE]nv\("` to find the mentions in Lix code.
### Used by Lix testing support code
- `_NIX_TEST_ACCEPT` (optional) - Writes out the result of a characterization test as the new expected value.
**Expected value**: 1
- `_NIX_TEST_UNIT_DATA` - The path to the directory for the data for a given unit test suite.
**Expected value**: `tests/unit/libstore/data/libstore` or similar
### Used by Lix
- `_NIX_FORCE_HTTP` - Forces file URIs to be treated as remote ones.
Used by `src/libfetchers/git.cc`, `src/libstore/http-binary-cache-store.cc`,
`src/libstore/local-binary-cache-store.cc`. Seems to be for forcing Git
clones of `git+file://` URLs, making the HTTP binary
cache store accept `file://` URLs (presumably passing them to curl?), and
unknown reasons for the local binary cache.
FIXME(jade): is this obscuring a bug in https://git.lix.systems/lix-project/lix/issues/200?
**Expected value**: 1
- `NIX_ATTRS_SH_FILE`, `NIX_ATTRS_JSON_FILE` (output) - Set by Lix builders; see
`structuredAttrs` documentation.
- `NIX_BIN_DIR`, `NIX_STORE_DIR` (or its inconsistently-used old alias `NIX_STORE`), `NIX_DATA_DIR`,
`NIX_LOG_DIR`, `NIX_LOG_DIR`, `NIX_STATE_DIR`, `NIX_CONF_DIR` -
Overrides compile-time configuration of various locations used by Lix. See `src/libstore/globals.cc`.
**Expected value**: a directory
- `NIX_DAEMON_SOCKET_PATH` (optional) - Overrides the daemon socket path from `$NIX_STATE_DIR/daemon-socket/socket`.
**Expected value**: path to a socket
- `NIX_LOG_FD` (output) - An FD number for logs in `internal-json` format to be sent to.
Used for, mostly, "setPhase" in nixpkgs setup.sh, but can also be creatively used to print verbose log messages from derivations.
**Provided value**: number corresponding to an FD in the builder
- `NIX_PATH` - Search path for `<whatever>`. Documented elsewhere in the manual.
**Expected value**: `:` separated list of things that are not necessarily pointing to filesystem paths
- `NIX_REMOTE` - The default value of the Lix setting `store`.
**Expected value**: "daemon", usually. Could be "auto" or any other value acceptable in `store`.
- `NIX_BUILD_SHELL` - Documented elsewhere; the shell to invoke with `nix-shell` but not `nix develop`/`nix shell`.
The latter ignoring it altogether seems like a bug.
**Expected value**: the path to an executable shell
- `PRINT_PATH` - Undocumented. Used by `nix-prefetch-url` as an alternative form of `--print-path`. Why???
- `_NIX_IN_TEST` - If present with any value, makes `fetchClosure` accept file URLs in addition to HTTP ones. Why is this not `_NIX_FORCE_HTTP`??
Not used anywhere else.
- `NIX_ALLOW_EVAL` - Used by eval-cache tests to block evaluation if set to `0`.
**Expected value**: 1 or 0
- `EDITOR` - Used by `editorFor()`, which has some extremely sketchy editor-detection code for jumping to line numbers.
- `LISTEN_FDS` and `LISTEN_PID` - Used for systemd socket activation using the systemd socket activation protocol.
- `NIX_PAGER` (alternatively, `PAGER`) - Used to select a pager for Lix output. Why does this not use libutil `getEnv()`?
- `LESS` (output) - Sets the pager settings for `less` when invoked by Lix.
- `NIX_IGNORE_SYMLINK_STORE` - When set, Lix allows the store to be a symlink. Why do we support this?
Apparently [someone was using it enough to fix it](https://github.com/NixOS/nix/pull/4038).
- `NIX_SSL_CERT_FILE` (alternatively, `SSL_CERT_FILE`) - Used to set CA certificates for libcurl.
**Expected value**: "/etc/ssl/certs/ca-certificates.crt" or similar
- `NIX_REMOTE_SYSTEMS` - Used to set `builders`. Can we please deprecate this?
- `NIX_USER_CONF_FILES` - `:` separated list of config files to load before
`/nix/nix.conf` under each of `XDG_CONFIG_DIRS`.
- `NIX_CONFIG` - Newline separated configuration to load into Lix.
- `NIX_GET_COMPLETIONS` - Returns completions.
Unsure of the exact format, someone should document it; either way my shell never had any completions.
**Expected value**: number of completions to return.
- `IN_SYSTEMD` - Used to switch the logging format so that systemd gets the correct log levels. I think.
- `NIX_HELD_LOCKS` - Not used, what is this for?? We should surely remove it right after searching github?
- `GC_INITIAL_HEAP_SIZE` - Used to set the initial heap size, processed by boehmgc.
- `NIX_COUNT_CALLS` - Documented elsewhere; prints call counts for profiling purposes.
- `NIX_SHOW_STATS` - Documented elsewhere; prints various evaluation statistics like function calls, gc info, and similar.
- `NIX_SHOW_STATS_PATH` - Writes those statistics into a file at the given path instead of stdout. Undocumented.
- `NIX_SHOW_SYMBOLS` - Dumps the symbol table into the show-stats json output.
- `TERM` - If `dumb` or unset, disables ANSI colour output.
- `FORCE_COLOR`, `CLICOLOR_FORCE` - Enables ANSI colour output if `NO_COLOR`/`NOCOLOR` not set.
- `NO_COLOR`, `NOCOLOR` - Disables ANSI colour output.
- `_NIX_DEVELOPER_SHOW_UNKNOWN_LOCATIONS` - Highlights unknown locations in errors.
- `NIX_PROFILE` - Selects which profile `nix-env` will operate on. Documented elsewhere.
- `NIX_SSHOPTS` - Options passed to `ssh(1)` when using a ssh remote store.
Incorrectly documented on `nix-copy-closure` which is *surely* not the only place they are used??
- `_NIX_TEST_GC_SYNC_1` - Path to a pipe that is used to block the GC briefly to validate invariants from the test suite.
- `_NIX_TEST_GC_SYNC_2` - Path to a pipe that is used to block the GC briefly to validate invariants from the test suite.
- `_NIX_TEST_FREE_SPACE_FILE` - Path to a file containing a decimal number with the free space that the GC is to believe it has.
- Various XDG vars
- `NIX_DEBUG_SQLITE_TRACES` - Dump all sqlite queries to the log at `notice` level.
- `_NIX_TEST_NO_SANDBOX` - Disables actually setting up the sandbox on macOS while leaving other logic the same. Unused on other platforms.
- `_NIX_TRACE_BUILT_OUTPUTS` - Dumps all the derivation paths alongside their outputs as lines into a file of the given name.
### Used by the functional test framework
- `NIX_DAEMON_PACKAGE` - Runs the test suite against an alternate Nix daemon with the current client.
**Expected value**: something like `/nix/store/...-nix-2.18.2`
- `NIX_CLIENT_PACKAGE` - Runs the test suite against an alternate Nix client with the current daemon.
**Expected value**: something like `/nix/store/...-nix-2.18.2`
- `NIX_TESTS_CA_BY_DEFAULT` - Pass `__contentAddressed`, `outputHashMode` and `outputHashAlgo` to builds of some input-addressed derivations in the test suite.
**Expected value**: 1
- `TEST_DATA` - Not an environment variable! This is used in repl characterization tests to refer to `tests/functional/repl_characterization/data`.
More specifically, that path is replaced with the string `$TEST_DATA` in output for reproducibility.
- `TEST_HOME` (output) - Set to the temporary directory that is set as `$HOME` inside the tests, underneath `$TEST_ROOT`.
- `TEST_ROOT` (output) - Set to the temporary directory that is created for each test to mess with.
- `_NIX_TEST_DAEMON_PID` (output) - Used to track the daemon pid to be able to kill it.
**Provided value**: Daemon pid as a base-10 integer, e.g. 2345

View file

@ -1,62 +1,64 @@
# Using Lix within Docker # Using Lix within Docker
Lix is available on the following two container registries: Currently the Lix project doesn't ship docker images. However, we have the infrastructure to do it, it's just not yet been done. See https://git.lix.systems/lix-project/lix/issues/252
- [ghcr.io/lix-project/lix](https://ghcr.io/lix-project/lix)
- [git.lix.systems/lix-project/lix](https://git.lix.systems/lix-project/-/packages/container/lix) <!--
To run the latest stable release of Lix with Docker run the following command: To run the latest stable release of Lix with Docker run the following command:
```console ```console
~ » sudo podman run -it ghcr.io/lix-project/lix:latest $ docker run -ti nixos/nix
Trying to pull ghcr.io/lix-project/lix:latest... Unable to find image 'nixos/nix:latest' locally
latest: Pulling from nixos/nix
bash-5.2# nix --version 5843afab3874: Pull complete
nix (Lix, like Nix) 2.90.0 b52bf13f109c: Pull complete
1e2415612aa3: Pull complete
Digest: sha256:27f6e7f60227e959ee7ece361f75d4844a40e1cc6878b6868fe30140420031ff
Status: Downloaded newer image for nixos/nix:latest
35ca4ada6e96:/# nix --version
nix (Nix) 2.3.12
35ca4ada6e96:/# exit
``` ```
# What is included in Lix's Docker image? # What is included in Lix's Docker image?
The official Docker image is created using [nix2container] The official Docker image is created using `pkgs.dockerTools.buildLayeredImage`
(and not with `Dockerfile` as it is usual with Docker images). You can still (and not with `Dockerfile` as it is usual with Docker images). You can still
base your custom Docker image on it as you would do with any other Docker base your custom Docker image on it as you would do with any other Docker
image. image.
[nix2container]: https://github.com/nlewo/nix2container The Docker image is also not based on any other image and includes minimal set
of runtime dependencies that are required to use Lix:
The Docker image is also not based on any other image and includes the nixpkgs - pkgs.nix
that Lix was built with along with a minimal set of tools in the system profile: - pkgs.bashInteractive
- pkgs.coreutils-full
- bashInteractive - pkgs.gnutar
- cacert.out - pkgs.gzip
- coreutils-full - pkgs.gnugrep
- curl - pkgs.which
- findutils - pkgs.curl
- gitMinimal - pkgs.less
- gnugrep - pkgs.wget
- gnutar - pkgs.man
- gzip - pkgs.cacert.out
- iana-etc - pkgs.findutils
- less
- libxml2
- lix
- man
- openssh
- sqlite
- wget
- which
# Docker image with the latest development version of Lix # Docker image with the latest development version of Lix
FIXME: There are not currently images of development versions of Lix. Tracking issue: https://git.lix.systems/lix-project/lix/issues/381 To get the latest image that was built by [Hydra](https://hydra.nixos.org) run
the following command:
You can build a Docker image from source yourself and copy it to either:
Podman: `nix run '.#dockerImage.copyTo' containers-storage:lix`
Docker: `nix run '.#dockerImage.copyToDockerDaemon'`
Then:
```console ```console
$ docker run -ti lix $ curl -L https://hydra.nixos.org/job/nix/master/dockerImage.x86_64-linux/latest/download/1 | docker load
$ docker run -ti nix:2.5pre20211105
``` ```
You can also build a Docker image from source yourself:
```console
$ nix build ./\#hydraJobs.dockerImage.x86_64-linux
$ docker load -i ./result/image.tar.gz
$ docker run -ti nix:2.5pre20211105
```
-->

View file

@ -1,124 +1,32 @@
# Multi-User Mode # Multi-User Mode
To allow a Nix store to be shared safely among multiple users, it is important that users cannot meaningfully influence the execution of derivation builds such that they could inject malicious code into them without changing their (either input- or output- addressed) hash. To allow a Nix store to be shared safely among multiple users, it is
If they could do so, they could install a Trojan horse in some package and compromise the accounts of other users. important that users are not able to run builders that modify the Nix
store or database in arbitrary ways, or that interfere with builds
started by other users. If they could do so, they could install a Trojan
horse in some package and compromise the accounts of other users.
To prevent this, the Nix store and database are owned by some privileged user (usually `root`) and builders are executed under unprivileged system user accounts (usually named `nixbld1`, `nixbld2`, etc.). To prevent this, the Nix store and database are owned by some privileged
When an unprivileged user runs a Nix command, actions that operate on the Nix store (such as builds) are forwarded to a *Nix daemon* running under the owner of the Nix store/database that performs the operation. user (usually `root`) and builders are executed under special user
accounts (usually named `nixbld1`, `nixbld2`, etc.). When a unprivileged
user runs a Nix command, actions that operate on the Nix store (such as
builds) are forwarded to a *Nix daemon* running under the owner of the
Nix store/database that performs the operation.
The buried lede in the above sentence is that *currently*, even in multi-user mode using a daemon, if executing as the user that owns the store, Lix directly manipulates the store unless `--store daemon` is specified. > **Note**
[We intend to change this in the future][multi-user-should-not-be-root]. >
> Multi-user mode has one important limitation: only root and a set of
> trusted users specified in `nix.conf` can specify arbitrary binary
> caches. So while unprivileged users may install packages from
> arbitrary Nix expressions, they may not get pre-built binaries.
<div class="warning"> ## Setting up the build users
The Lix team considers the goal of the sandbox to be primarily for preventing reproducibility mistakes, and does not consider multi-user mode to be a strong security boundary between users.
Do not evaluate or build untrusted, potentially-malicious, Nix language code on machines that you care deeply about maintaining user isolation on.
Although we would consider any sandbox escapes to be serious security bugs and we intend to fix them, we are not confident enough in the daemon's security to call the daemon a security boundary.
</div>
[multi-user-should-not-be-root]: https://git.lix.systems/lix-project/lix/issues/18
## Trust model
There are two categories of users of the Lix daemon: trusted users and untrusted users.
The Lix daemon only allows connections from users that are either trusted users, or are specified in, or are members of groups specified in, [`allowed-users`](../command-ref/conf-file.md#conf-allowed-users) in `nix.conf`.
Trusted users are users and users of groups specified in [`trusted-users`](../command-ref/conf-file.md#conf-trusted-users) in `nix.conf`.
All users of the Lix daemon may do the following to bring things into the Nix store:
- Users may load derivations and output-addressed files into the store with `nix-store --add` or through Nix language code.
- Users may locally build derivations, either of the output-addressed or input-addressed variety, creating output paths.
Note that [fixed-output derivations only consider name and hash](https://github.com/NixOS/nix/issues/969), so it is possible to write a fixed-output derivation for something important with a bogus hash and have it resolve to something else already built in the store.
On systems with `sandbox` enabled (default on Linux; [not *yet* on macOS][sandbox-enable-macos]), derivations are either:
- Input-addressed, so they are run in the sandbox with no network access, with the following exceptions:
- The (poorly named, since it is not *just* about chroot) property `__noChroot` is set on the derivation and `sandbox` is set to `relaxed`.
- On macOS, the derivation property `__darwinAllowLocalNetworking` allows network access to localhost from input-addressed derivations regardless of the `sandbox` setting value.
This property exists with such semantics because macOS has no network namespace equivalent to isolate individual processes' localhost networking.
- On macOS, the derivation property `__sandboxProfile` accepts extra sandbox profile S-expressions, allowing derivations to bypass arbitrary parts of the sandbox without altogether disabling it.
This is only permitted when `sandbox` is set to `relaxed`.
- Output-addressed, so they are run with network access but their result must match an expected hash.
Trusted users may set any setting, including `sandbox = false`, so the sandbox state can be different at runtime from what is described in `nix.conf` for builds invoked with such settings.
- Users may copy appropriately-signed derivation outputs into the store.
By default, any paths *copied into a store* (such as by substitution) must have signatures from [`trusted-public-keys`](../command-ref/conf-file.md#conf-trusted-public-keys) unless they are [output-addressed](../glossary.md#gloss-output-addressed-store-object).
Unsigned paths may be copied into a store if [`require-sigs`](../command-ref/conf-file.md#conf-require-sigs) is disabled in the daemon's configuration (not default), or if the client is a trusted user and passed `--no-check-sigs` to `nix copy`.
- Users may request that the daemon substitutes appropriately-signed derivation outputs from a binary cache in the daemon's [`substituters`](../command-ref/conf-file.md#conf-substituters) list.
Untrusted clients may also specify additional values for `substituters` (via e.g. `--extra-substituters` on a Nix command) that are listed in [`trusted-substituters`](../command-ref/conf-file.md#conf-trusted-substituters).
A client could in principle substitute such paths itself then copy them to the daemon (see clause above) if they are appropriately signed but are *not* from a trusted substituter, however this is not implemented in the current Lix client to our knowledge, at the time of writing.
This probably means that `trusted-substituters` is a redundant setting except insofar as such substitution would have to be done on the client rather than as root on the daemon; and it is highly defensible to not allow random usage of our HTTP client running as root.
[sandbox-enable-macos]: https://git.lix.systems/lix-project/lix/issues/386
### The Lix daemon as a security non-boundary
The Lix team and wider community does not consider the Lix daemon to be a *security boundary* against malicious Nix language code.
Although we do our best to make it secure, we do not recommend sharing a Lix daemon with potentially malicious users.
That means that public continuous integration (CI) builds of untrusted Nix code should not share builders with CI that writes into a cache used by trusted infrastructure.
For example, [hydra.nixos.org], which is the builder for [cache.nixos.org], does not execute untrusted Nix language code; a separate system, [ofborg] is used for CI of nixpkgs pull requests.
The build output of pull request CI is never pushed to [cache.nixos.org], and those systems are considered entirely untrusted.
This is because, among other things, the Lix sandbox is *more* susceptible to kernel exploits than Docker, which, unlike Lix, blocks nested user namespaces via `seccomp` in its default policy, and there have been many kernel bugs only exposed to unprivileged users via user namespaces allowing otherwise-root-only system calls.
In general, the Lix sandbox is set up to be relatively unrestricted while maintaining its goals of building useful, reproducible software; security is not its primary goal.
The Lix sandbox is a custom *non-rootless* Linux container implementation that has not been audited to nearly the same degree as Docker and similar systems.
Also, the Lix daemon is a complex and historied C++ executable running as root with very little privilege separation.
All of this means that a security hole in the Lix daemon gives immediate root access.
Systems like Docker (especially non-rootless Docker) should *themselves* probably not be used in a multi-tenant manner with mutually distrusting tenants, but the Lix daemon *especially* should not be used as such as of this writing.
The primary purpose of the sandbox is to strongly encourage packages to be reproducible, a goal which it is generally quite successful at.
[hydra.nixos.org]: https://hydra.nixos.org
[ofborg]: https://github.com/NixOS/ofborg
[cache.nixos.org]: https://cache.nixos.org
### Trusted users
Trusted users are permitted to set any setting and bypass security restrictions on the daemon.
They are currently in widespread use for a couple of reasons such as remote builds (which we [intend to fix](https://git.lix.systems/lix-project/lix/issues/171)).
Trusted users are effectively root on Nix daemons running as root (the default configuration) for *at least* the following reasons, and should be thus thought of as equivalent to passwordless sudo.
This is not a comprehensive list.
- They may copy an unsigned malicious built output into the store for `systemd` or anything else that will run as root, then when the system is upgraded, that path will be used from the local store rather than substituted.
- They may set the following settings that are commands the daemon will run as root:
- `build-hook`
- `diff-hook`
- `pre-build-hook`
- `post-build-hook`
- They may set `build-users-group`.
In particular, they may set it to empty string, which runs builds as root with respect to the rest of the system (!!).
We, too, [think that is absurd and intend to not accept such a configuration](https://git.lix.systems/lix-project/lix/issues/242).
It is then simply an exercise to the reader to find a daemon that does `SCM_CREDENTIALS` over a `unix(7)` socket and lets you run commands as root, and mount it into the sandbox with `extra-sandbox-paths`.
At the very least, the Lix daemon itself (since `root` is a trusted user by default) and probably `systemd` qualify for this.
- They may set the `builders` list, which will have ssh run as root.
We aren't sure if there is a way to abuse this for command execution but it's plausible.
Note that setting `accept-flake-config` allows arbitrary Nix flakes to set Nix settings in the `nixConfig` stanza.
Do not set this setting or pass `--accept-flake-config` while executing untrusted Nix language code as a trusted user for the reasons above!
## Build users
The *build users* are the special UIDs under which builds are performed. The *build users* are the special UIDs under which builds are performed.
A build user is selected for a build by looking in the group specified by [`build-users-group`](../command-ref/conf-file.md#conf-build-users-group), by default, `nixbld`, then a member of that group not currently executing a build is selected for the build. They should all be members of the *build users group* `nixbld`. This
The build users should not be members of any other group. group should have no other members. The build users should not be
members of any other group. On Linux, you can create the group and users
There can never be more concurrent builds than the number of build users, unless using [`auto-allocate-uids`](../command-ref/conf-file.md#conf-auto-allocate-uids) ([tracking issue][auto-allocate-uids-issue]). as follows:
[auto-allocate-uids-issue]: https://git.lix.systems/lix-project/lix/issues/387
If, for some reason, you need to create such users manually, the following command will create 10 build users on Linux:
```console ```console
$ groupadd -r nixbld $ groupadd -r nixbld
@ -127,12 +35,43 @@ $ for n in $(seq 1 10); do useradd -c "Nix build user $n" \
nixbld$n; done nixbld$n; done
``` ```
This creates 10 build users. There can never be more concurrent builds
than the number of build users, so you may want to increase this if you
expect to do many builds at the same time.
## Running the daemon ## Running the daemon
The [Nix daemon](../command-ref/nix-daemon.md) can be started manually as follows (as `root`): The [Nix daemon](../command-ref/nix-daemon.md) should be started as
follows (as `root`):
```console ```console
# nix-daemon $ nix-daemon
``` ```
In standard installations of Lix, the daemon is started by a `systemd` unit (Linux) or `launchd` service (macOS). Youll want to put that line somewhere in your systems boot scripts.
To let unprivileged users use the daemon, they should set the
[`NIX_REMOTE` environment variable](../command-ref/env-common.md) to
`daemon`. So you should put a line like
```console
export NIX_REMOTE=daemon
```
into the users login scripts.
## Restricting access
To limit which users can perform Nix operations, you can use the
permissions on the directory `/nix/var/nix/daemon-socket`. For instance,
if you want to restrict the use of Nix to the members of a group called
`nix-users`, do
```console
$ chgrp nix-users /nix/var/nix/daemon-socket
$ chmod ug=rwx,o= /nix/var/nix/daemon-socket
```
This way, users who are not in the `nix-users` group cannot connect to
the Unix domain socket `/nix/var/nix/daemon-socket/socket`, so they
cannot perform Nix operations.

View file

@ -7,8 +7,9 @@ management operations. All other users can then use the installed
packages, but they cannot perform package management operations packages, but they cannot perform package management operations
themselves. themselves.
Alternatively, you can configure Lix in “multi-user mode”. In this model, all users can perform package management operations — for instance, every user can install software for themselves without requiring root privileges. Alternatively, you can configure Lix in “multi-user mode”. In this
Lix does its best to ensure that this is secure. model, all users can perform package management operations — for
For instance, it would be considered a serious security bug for one untrusted user to be able to overwrite a package used by another user with a Trojan horse. instance, every user can install software without requiring root
privileges. Lix ensures that this is secure. For instance, its not
Nevertheless, the Lix team does not consider multi-user mode a strong security boundary, and does not recommend running untrusted user-supplied Nix language code on privileged machines, even if it is secure to the best of our knowledge at any moment in time. possible for one user to overwrite a package used by another user with a
Trojan horse.

View file

@ -1,13 +1,13 @@
# Obtaining the Source # Obtaining the Source
The most recent sources of Lix can be obtained from its [Git The most recent sources of Nix can be obtained from its [Git
repository](https://git.lix.systems/lix-project/lix). For example, the following repository](https://github.com/NixOS/nix). For example, the following
command will check out the latest revision into a directory called command will check out the latest revision into a directory called
`nix`: `nix`:
```console ```console
$ git clone https://git.lix.systems/lix-project/lix $ git clone https://github.com/NixOS/nix
``` ```
Likewise, specific releases can be obtained from the Likewise, specific releases can be obtained from the
[tags](https://git.lix.systems/lix-project/lix/tags) of the repository. [tags](https://github.com/NixOS/nix/tags) of the repository.

View file

@ -68,7 +68,10 @@ The most current alternative to this section is to read `package.nix` and see wh
may also work, but ancient versions like the ubiquitous 2.5.4a may also work, but ancient versions like the ubiquitous 2.5.4a
won't. won't.
- The `libseccomp` is used to provide syscall filtering on Linux. To get - The `libseccomp` is used to provide syscall filtering on Linux. This
is an optional dependency and can be disabled passing a
`--disable-seccomp-sandboxing` option to the `configure` script (Not
recommended unless your system doesn't support `libseccomp`). To get
the library, visit <https://github.com/seccomp/libseccomp>. the library, visit <https://github.com/seccomp/libseccomp>.
- On 64-bit x86 machines only, `libcpuid` library - On 64-bit x86 machines only, `libcpuid` library

Some files were not shown because too many files have changed in this diff Show more