cafkafk/lix
cafkafk/lix
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Merge remote-tracking branch 'origin/master' into flakes

Browse source
This commit is contained in:
Eelco Dolstra 2019-05-15 20:51:29 +02:00
commit 0f5032c5a4
12 changed files with 400 additions and 47 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
doc/manual/advanced-topics/advanced-topics.xml
View file

@ -7,5 +7,6 @@
<title>Advanced Topics</title>
<xi:include href="distributed-builds.xml" />
<xi:include href="diff-hook.xml" />
</part>

205
doc/manual/advanced-topics/diff-hook.xml Normal file
View file

@ -0,0 +1,205 @@
<chapter xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="chap-diff-hook"
version="5.0"
>
<title>Verifying Build Reproducibility with <option linkend="conf-diff-hook">diff-hook</option></title>
<subtitle>Check build reproducibility by running builds multiple times
and comparing their results.</subtitle>
<para>Specify a program with Nix's <xref linkend="conf-diff-hook" /> to
compare build results when two builds produce different results. Note:
this hook is only executed if the results are not the same, this hook
is not used for determining if the results are the same.</para>
<para>For purposes of demonstration, we'll use the following Nix file,
<filename>deterministic.nix</filename> for testing:</para>
<programlisting>
let
inherit (import &lt;nixpkgs&gt; {}) runCommand;
in {
stable = runCommand "stable" {} ''
touch $out
'';
unstable = runCommand "unstable" {} ''
echo $RANDOM > $out
'';
}
</programlisting>
<para>Additionally, <filename>nix.conf</filename> contains:
<programlisting>
diff-hook = /etc/nix/my-diff-hook
run-diff-hook = true
</programlisting>
where <filename>/etc/nix/my-diff-hook</filename> is an executable
file containing:
<programlisting>
#!/bin/sh
exec &gt;&amp;2
echo "For derivation $3:"
/run/current-system/sw/bin/diff -r "$1" "$2"
</programlisting>
</para>
<para>The diff hook is executed by the same user and group who ran the
build. However, the diff hook does not have write access to the store
path just built.</para>
<section>
<title>
Spot-Checking Build Determinism
</title>
<para>
Verify a path which already exists in the Nix store by passing
<option>--check</option> to the build command.
</para>
<para>If the build passes and is deterministic, Nix will exit with a
status code of 0:</para>
<screen>
$ nix-build ./deterministic.nix -A stable
these derivations will be built:
/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv
building '/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv'...
/nix/store/yyxlzw3vqaas7wfp04g0b1xg51f2czgq-stable
$ nix-build ./deterministic.nix -A stable --check
checking outputs of '/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv'...
/nix/store/yyxlzw3vqaas7wfp04g0b1xg51f2czgq-stable
</screen>
<para>If the build is not deterministic, Nix will exit with a status
code of 1:</para>
<screen>
$ nix-build ./deterministic.nix -A unstable
these derivations will be built:
/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv
building '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable
$ nix-build ./deterministic.nix -A unstable --check
checking outputs of '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
error: derivation '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv' may not be deterministic: output '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable' differs
</screen>
<para>In the Nix daemon's log, we will now see:
<screen>
For derivation /nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv:
1c1
&lt; 8108
---
&gt; 30204
</screen>
</para>
<para>Using <option>--check</option> with <option>--keep-failed</option>
will cause Nix to keep the second build's output in a special,
<literal>.check</literal> path:</para>
<screen>
$ nix-build ./deterministic.nix -A unstable --check --keep-failed
checking outputs of '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
note: keeping build directory '/tmp/nix-build-unstable.drv-0'
error: derivation '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv' may not be deterministic: output '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable' differs from '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable.check'
</screen>
<para>In particular, notice the
<literal>/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable.check</literal>
output. Nix has copied the build results to that directory where you
can examine it.</para>
<note xml:id="check-dirs-are-unregistered">
<title><literal>.check</literal> paths are not registered store paths</title>
<para>Check paths are not protected against garbage collection,
and this path will be deleted on the next garbage collection.</para>
<para>The path is guaranteed to be alive for the duration of
<xref linkend="conf-diff-hook" />'s execution, but may be deleted
any time after.</para>
<para>If the comparison is performed as part of automated tooling,
please use the diff-hook or author your tooling to handle the case
where the build was not deterministic and also a check path does
not exist.</para>
</note>
<para>
<option>--check</option> is only usable if the derivation has
been built on the system already. If the derivation has not been
built Nix will fail with the error:
<screen>
error: some outputs of '/nix/store/hzi1h60z2qf0nb85iwnpvrai3j2w7rr6-unstable.drv' are not valid, so checking is not possible
</screen>
Run the build without <option>--check</option>, and then try with
<option>--check</option> again.
</para>
</section>
<section>
<title>
Automatic and Optionally Enforced Determinism Verification
</title>
<para>
Automatically verify every build at build time by executing the
build multiple times.
</para>
<para>
Setting <xref linkend="conf-repeat" /> and
<xref linkend="conf-enforce-determinism" /> in your
<filename>nix.conf</filename> permits the automated verification
of every build Nix performs.
</para>
<para>
The following configuration will run each build three times, and
will require the build to be deterministic:
<programlisting>
enforce-determinism = true
repeat = 2
</programlisting>
</para>
<para>
Setting <xref linkend="conf-enforce-determinism" /> to false as in
the following configuration will run the build multiple times,
execute the build hook, but will allow the build to succeed even
if it does not build reproducibly:
<programlisting>
enforce-determinism = false
repeat = 1
</programlisting>
</para>
<para>
An example output of this configuration:
<screen>
$ nix-build ./test.nix -A unstable
these derivations will be built:
/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv
building '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' (round 1/2)...
building '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' (round 2/2)...
output '/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable' of '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' differs from '/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable.check' from previous round
/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable
</screen>
</para>
</section>
</chapter>

88
doc/manual/command-ref/conf-file.xml
View file

@ -1,7 +1,9 @@
<?xml version="1.0" encoding="utf-8"?>
<refentry xmlns="http://docbook.org/ns/docbook"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xi="http://www.w3.org/2001/XInclude"
xml:id="sec-conf-file">
xml:id="sec-conf-file"
version="5">
<refmeta>
<refentrytitle>nix.conf</refentrytitle>
@ -240,6 +242,71 @@ false</literal>.</para>
</varlistentry>
<varlistentry xml:id="conf-diff-hook"><term><literal>diff-hook</literal></term>
<listitem>
<para>
Absolute path to an executable capable of diffing build results.
The hook executes if <xref linkend="conf-run-diff-hook" /> is
true, and the output of a build is known to not be the same.
This program is not executed to determine if two results are the
same.
</para>
<para>
The diff hook is executed by the same user and group who ran the
build. However, the diff hook does not have write access to the
store path just built.
</para>
<para>The diff hook program receives three parameters:</para>
<orderedlist>
<listitem>
<para>
A path to the previous build's results
</para>
</listitem>
<listitem>
<para>
A path to the current build's results
</para>
</listitem>
<listitem>
<para>
The path to the build's derivation
</para>
</listitem>
<listitem>
<para>
The path to the build's scratch directory. This directory
will exist only if the build was run with
<option>--keep-failed</option>.
</para>
</listitem>
</orderedlist>
<para>
The stderr and stdout output from the diff hook will not be
displayed to the user. Instead, it will print to the nix-daemon's
log.
</para>
<para>When using the Nix daemon, <literal>diff-hook</literal> must
be set in the <filename>nix.conf</filename> configuration file, and
cannot be passed at the command line.
</para>
</listitem>
</varlistentry>
<varlistentry xml:id="conf-enforce-determinism">
<term><literal>enforce-determinism</literal></term>
<listitem><para>See <xref linkend="conf-repeat" />.</para></listitem>
</varlistentry>
<varlistentry xml:id="conf-extra-sandbox-paths">
<term><literal>extra-sandbox-paths</literal></term>
@ -595,9 +662,9 @@ password <replaceable>my-password</replaceable>
they are deterministic. The default value is 0. If the value is
non-zero, every build is repeated the specified number of
times. If the contents of any of the runs differs from the
previous ones, the build is rejected and the resulting store paths
are not registered as “valid” in Nix’s database.</para></listitem>
previous ones and <xref linkend="conf-enforce-determinism" /> is
true, the build is rejected and the resulting store paths are not
registered as “valid” in Nix’s database.</para></listitem>
</varlistentry>
<varlistentry xml:id="conf-require-sigs"><term><literal>require-sigs</literal></term>
@ -628,6 +695,19 @@ password <replaceable>my-password</replaceable>
</varlistentry>
<varlistentry xml:id="conf-run-diff-hook"><term><literal>run-diff-hook</literal></term>
<listitem>
<para>
If true, enable the execution of <xref linkend="conf-diff-hook" />.
</para>
<para>
When using the Nix daemon, <literal>run-diff-hook</literal> must
be set in the <filename>nix.conf</filename> configuration file,
and cannot be passed at the command line.
</para>
</listitem>
</varlistentry>
<varlistentry xml:id="conf-sandbox"><term><literal>sandbox</literal></term>

2
scripts/install-nix-from-closure.sh
View file

@ -26,7 +26,7 @@ fi
if [ "$(uname -s)" = "Darwin" ]; then
macos_major=$(sw_vers -productVersion | cut -d '.' -f 2)
macos_minor=$(sw_vers -productVersion | cut -d '.' -f 3)
if [ "$macos_major" -lt 12 ] || ([ "$macos_major" -eq 12 ] && [ "$macos_minor" -lt 6 ]); then
if [ "$macos_major" -lt 12 ] || { [ "$macos_major" -eq 12 ] && [ "$macos_minor" -lt 6 ]; }; then
echo "$0: macOS $(sw_vers -productVersion) is not supported, upgrade to 10.12.6 or higher"
exit 1
fi

7
scripts/nix-profile.sh.in
View file

@ -51,10 +51,9 @@ if [ -n "$HOME" ] && [ -n "$USER" ]; then
unset __nix_defexpr
fi
# Append ~/.nix-defexpr/channels/nixpkgs to $NIX_PATH so that
# <nixpkgs> paths work when the user has fetched the Nixpkgs
# channel.
export NIX_PATH="${NIX_PATH:+$NIX_PATH:}nixpkgs=$HOME/.nix-defexpr/channels/nixpkgs"
# Append ~/.nix-defexpr/channels to $NIX_PATH so that <nixpkgs>
# paths work when the user has fetched the Nixpkgs channel.
export NIX_PATH=${NIX_PATH:+$NIX_PATH:}$HOME/.nix-defexpr/channels
# Set up environment.
# This part should be kept in sync with nixpkgs:nixos/modules/programs/environment.nix

75
src/libstore/build.cc
View file

@ -461,6 +461,28 @@ static void commonChildInit(Pipe & logPipe)
close(fdDevNull);
}
void handleDiffHook(uid_t uid, uid_t gid, Path tryA, Path tryB, Path drvPath, Path tmpDir)
{
auto diffHook = settings.diffHook;
if (diffHook != "" && settings.runDiffHook) {
try {
RunOptions diffHookOptions(diffHook,{tryA, tryB, drvPath, tmpDir});
diffHookOptions.searchPath = true;
diffHookOptions.uid = uid;
diffHookOptions.gid = gid;
diffHookOptions.chdir = "/";
auto diffRes = runProgram(diffHookOptions);
if (!statusOk(diffRes.first))
throw ExecError(diffRes.first, fmt("diff-hook program '%1%' %2%", diffHook, statusToString(diffRes.first)));
if (diffRes.second != "")
printError(chomp(diffRes.second));
} catch (Error & error) {
printError("diff hook execution failed: %s", error.what());
}
}
}
//////////////////////////////////////////////////////////////////////
@ -803,9 +825,6 @@ private:
/* Whether we're currently doing a chroot build. */
bool useChroot = false;
/* Whether we need to perform hash rewriting if there are valid output paths. */
bool needsHashRewrite;
Path chrootRootDir;
/* RAII object to delete the chroot directory. */
@ -885,6 +904,9 @@ public:
Worker & worker, BuildMode buildMode = bmNormal);
~DerivationGoal();
/* Whether we need to perform hash rewriting if there are valid output paths. */
bool needsHashRewrite();
void timedOut() override;
string key() override
@ -997,13 +1019,6 @@ DerivationGoal::DerivationGoal(const Path & drvPath, const StringSet & wantedOut
, wantedOutputs(wantedOutputs)
, buildMode(buildMode)
{
#if __linux__
needsHashRewrite = !useChroot;
#else
/* Darwin requires hash rewriting even when sandboxing is enabled. */
needsHashRewrite = true;
#endif
state = &DerivationGoal::getDerivation;
name = (format("building of '%1%'") % drvPath).str();
trace("created");
@ -1044,6 +1059,17 @@ DerivationGoal::~DerivationGoal()
}
inline bool DerivationGoal::needsHashRewrite()
{
#if __linux__
return !useChroot;
#else
/* Darwin requires hash rewriting even when sandboxing is enabled. */
return true;
#endif
}
void DerivationGoal::killChild()
{
if (pid != -1) {
@ -2083,7 +2109,7 @@ void DerivationGoal::startBuilder()
#endif
}
if (needsHashRewrite) {
if (needsHashRewrite()) {
if (pathExists(homeDir))
throw Error(format("directory '%1%' exists; please remove it") % homeDir);
@ -3039,8 +3065,7 @@ void DerivationGoal::registerOutputs()
InodesSeen inodesSeen;
Path checkSuffix = ".check";
bool runDiffHook = settings.runDiffHook;
bool keepPreviousRound = settings.keepFailed || runDiffHook;
bool keepPreviousRound = settings.keepFailed || settings.runDiffHook;
std::exception_ptr delayedException;
@ -3067,7 +3092,7 @@ void DerivationGoal::registerOutputs()
if (buildMode != bmCheck) actualPath = worker.store.toRealPath(path);
}
if (needsHashRewrite) {
if (needsHashRewrite()) {
Path redirected = redirectedOutputs[path];
if (buildMode == bmRepair
&& redirectedBadOutputs.find(path) != redirectedBadOutputs.end()
@ -3185,11 +3210,17 @@ void DerivationGoal::registerOutputs()
if (!worker.store.isValidPath(path)) continue;
auto info = *worker.store.queryPathInfo(path);
if (hash.first != info.narHash) {
if (settings.keepFailed) {
if (settings.runDiffHook || settings.keepFailed) {
Path dst = worker.store.toRealPath(path + checkSuffix);
deletePath(dst);
if (rename(actualPath.c_str(), dst.c_str()))
throw SysError(format("renaming '%1%' to '%2%'") % actualPath % dst);
handleDiffHook(
buildUser ? buildUser->getUID() : getuid(),
buildUser ? buildUser->getGID() : getgid(),
path, dst, drvPath, tmpDir);
throw Error(format("derivation '%1%' may not be deterministic: output '%2%' differs from '%3%'")
% drvPath % path % dst);
} else
@ -3254,16 +3285,10 @@ void DerivationGoal::registerOutputs()
? fmt("output '%1%' of '%2%' differs from '%3%' from previous round", i->second.path, drvPath, prev)
: fmt("output '%1%' of '%2%' differs from previous round", i->second.path, drvPath);
auto diffHook = settings.diffHook;
if (prevExists && diffHook != "" && runDiffHook) {
try {
auto diff = runProgram(diffHook, true, {prev, i->second.path});
if (diff != "")
printError(chomp(diff));
} catch (Error & error) {
printError("diff hook execution failed: %s", error.what());
}
}
handleDiffHook(
buildUser ? buildUser->getUID() : getuid(),
buildUser ? buildUser->getGID() : getgid(),
prev, i->second.path, drvPath, tmpDir);
if (settings.enforceDeterminism)
throw NotDeterministic(msg);

11
src/libutil/util.cc
View file

@ -16,6 +16,7 @@
#include <future>
#include <fcntl.h>
#include <grp.h>
#include <limits.h>
#include <pwd.h>
#include <sys/ioctl.h>
@ -1038,6 +1039,16 @@ void runProgram2(const RunOptions & options)
if (source && dup2(in.readSide.get(), STDIN_FILENO) == -1)
throw SysError("dupping stdin");
if (options.chdir && chdir((*options.chdir).c_str()) == -1)
throw SysError("chdir failed");
if (options.gid && setgid(*options.gid) == -1)
throw SysError("setgid failed");
/* Drop all other groups if we're setgid. */
if (options.gid && setgroups(0, 0) == -1)
throw SysError("setgroups failed");
if (options.uid && setuid(*options.uid) == -1)
throw SysError("setuid failed");
Strings args_(options.args);
args_.push_front(options.program);

4
src/libutil/util.hh
View file

@ -271,6 +271,9 @@ string runProgram(Path program, bool searchPath = false,
struct RunOptions
{
std::optional<uid_t> uid;
std::optional<uid_t> gid;
std::optional<Path> chdir;
Path program;
bool searchPath = true;
Strings args;
@ -427,6 +430,7 @@ void ignoreException();
/* Some ANSI escape sequences. */
#define ANSI_NORMAL "\e[0m"
#define ANSI_BOLD "\e[1m"
#define ANSI_FAINT "\e[2m"
#define ANSI_RED "\e[31;1m"
#define ANSI_GREEN "\e[32;1m"
#define ANSI_BLUE "\e[34;1m"

10
src/nix/main.cc
View file

@ -20,6 +20,8 @@ std::string programPath;
struct NixArgs : virtual MultiCommand, virtual MixCommonArgs
{
bool printBuildLogs = false;
NixArgs() : MultiCommand(*RegisterCommand::commands), MixCommonArgs("nix")
{
mkFlag()
@ -41,6 +43,11 @@ struct NixArgs : virtual MultiCommand, virtual MixCommonArgs
throw Exit();
});
mkFlag()
.longName("print-build-logs")
.description("print full build logs on stderr")
.set(&printBuildLogs, true);
mkFlag()
.longName("version")
.description("show version information")
@ -109,8 +116,7 @@ void mainWrapped(int argc, char * * argv)
Finally f([]() { stopProgressBar(); });
if (isatty(STDERR_FILENO))
startProgressBar();
startProgressBar(args.printBuildLogs);
args.command->prepare();
args.command->run();

39
src/nix/progress-bar.cc
View file

@ -2,6 +2,7 @@
#include "util.hh"
#include "sync.hh"
#include "store-api.hh"
#include "names.hh"
#include <atomic>
#include <map>
@ -38,6 +39,7 @@ private:
std::map<ActivityType, uint64_t> expectedByType;
bool visible = true;
ActivityId parent;
std::optional<std::string> name;
};
struct ActivitiesByType
@ -68,10 +70,16 @@ private:
std::condition_variable quitCV, updateCV;
bool printBuildLogs;
bool isTTY;
public:
ProgressBar()
ProgressBar(bool printBuildLogs, bool isTTY)
: printBuildLogs(printBuildLogs)
, isTTY(isTTY)
{
state_.lock()->active = isTTY;
updateThread = std::thread([&]() {
auto state(state_.lock());
while (state->active) {
@ -109,8 +117,14 @@ public:
void log(State & state, Verbosity lvl, const std::string & s)
{
writeToStderr("\r\e[K" + s + ANSI_NORMAL "\n");
draw(state);
if (state.active) {
writeToStderr("\r\e[K" + s + ANSI_NORMAL "\n");
draw(state);
} else {
auto s2 = s + ANSI_NORMAL "\n";
if (!isTTY) s2 = filterANSIEscapes(s2, true);
writeToStderr(s2);
}
}
void startActivity(ActivityId act, Verbosity lvl, ActivityType type,
@ -141,6 +155,7 @@ public:
auto nrRounds = getI(fields, 3);
if (nrRounds != 1)
i->s += fmt(" (round %d/%d)", curRound, nrRounds);
i->name = DrvName(name).name;
}
if (type == actSubstitute) {
@ -217,11 +232,15 @@ public:
auto i = state->its.find(act);
assert(i != state->its.end());
ActInfo info = *i->second;
state->activities.erase(i->second);
info.lastLine = lastLine;
state->activities.emplace_back(info);
i->second = std::prev(state->activities.end());
update();
if (printBuildLogs) {
log(*state, lvlInfo, ANSI_FAINT + info.name.value_or("unnamed") + "> " + ANSI_NORMAL + lastLine);
} else {
state->activities.erase(i->second);
info.lastLine = lastLine;
state->activities.emplace_back(info);
i->second = std::prev(state->activities.end());
update();
}
}
}
@ -395,9 +414,9 @@ public:
}
};
void startProgressBar()
void startProgressBar(bool printBuildLogs)
{
logger = new ProgressBar();
logger = new ProgressBar(printBuildLogs, isatty(STDERR_FILENO));
}
void stopProgressBar()

2
src/nix/progress-bar.hh
View file

@ -4,7 +4,7 @@
namespace nix {
void startProgressBar();
void startProgressBar(bool printBuildLogs = false);
void stopProgressBar();

3
tests/linux-sandbox.sh
View file

@ -25,3 +25,6 @@ nix path-info -r $outPath | grep input-2
nix ls-store -R -l $outPath | grep foobar
nix cat-store $outPath/foobar | grep FOOBAR
# Test --check without hash rewriting.
nix-build dependencies.nix --no-out-link --check --sandbox-paths /nix/store