cafkafk/lix
cafkafk/lix
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Update nix-push manpage and document the binary cache format

Browse source
This commit is contained in:
Eelco Dolstra 2012-11-06 15:49:53 +01:00
parent 82951e5582
commit 3a95e1a17c
2 changed files with 340 additions and 76 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

12
doc/manual/glossary.xml
View file

@ -160,6 +160,18 @@
</glossentry>
<glossentry xml:id="gloss-nar"><glossterm>NAR</glossterm>
<glossdef><para>A <emphasis>N</emphasis>ix
<emphasis>AR</emphasis>chive. This is a serialisation of a path in
the Nix store. It can contain regular files, directories and
symbolic links. NARs are generated and unpacked using
<command>nix-store --dump</command> and <command>nix-store
--restore</command>.</para></glossdef>
</glossentry>
</glosslist>

392
doc/manual/nix-push.xml
View file

@ -12,24 +12,19 @@
<refnamediv>
<refname>nix-push</refname>
<refpurpose>push store paths onto a network cache</refpurpose>
<refpurpose>generate a binary cache</refpurpose>
</refnamediv>
<refsynopsisdiv>
<cmdsynopsis>
<command>nix-push</command>
<group choice='req'>
<arg choice='req'>
<arg choice='plain'><replaceable>archivesPutURL</replaceable></arg>
<arg choice='plain'><replaceable>archivesGetURL</replaceable></arg>
<arg choice='plain'><replaceable>manifestPutURL</replaceable></arg>
</arg>
<arg choice='req'>
<arg choice='plain'><option>--copy</option></arg>
<arg choice='plain'><replaceable>archivesDir</replaceable></arg>
<arg choice='plain'><replaceable>manifestFile</replaceable></arg>
</arg>
</group>
<arg choice='plain'><option>--dest</option> <replaceable>dest-dir</replaceable></arg>
<arg><option>--bzip2</option></arg>
<arg><option>--force</option></arg>
<arg><option>--link</option></arg>
<arg><option>--manifest</option></arg>
<arg><option>--manifest-path</option> <replaceable>filename</replaceable></arg>
<arg><option>--url-prefix</option> <replaceable>url</replaceable></arg>
<arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
</cmdsynopsis>
</refsynopsisdiv>
@ -37,93 +32,350 @@
<refsection><title>Description</title>
<para>The command <command>nix-push</command> builds a set of store
paths (if necessary), and then packages and uploads all store paths in
the resulting closures to a server. A network cache thus populated
can subsequently be used to speed up software deployment on other
machines using the <command>nix-pull</command> command.</para>
<para>The command <command>nix-push</command> produces a
<emphasis>binary cache</emphasis>, a directory containing compressed
Nix archives (NARs) plus some metadata of the closure of the specified
store paths. This directory can then be made available through a web
server to other Nix installations, allowing them to skip building from
source and instead download binaries from the cache
automatically.</para>
<para><command>nix-push</command> performs the following actions.
<orderedlist>
<listitem><para>Each path in <replaceable>paths</replaceable> is
realised (using <link
linkend='rsec-nix-store-realise'><literal>nix-store
--realise</literal></link>).</para></listitem>
built (using <link
linkend='rsec-nix-store-realise'><command>nix-store
--realise</command></link>).</para></listitem>
<listitem><para>All paths in the closure of the store expressions
stored in <replaceable>paths</replaceable> are determined (using
<literal>nix-store --query --requisites
--include-outputs</literal>). It should be noted that since the
<option>--include-outputs</option> flag is used, you get a combined
source/binary distribution.</para></listitem>
<listitem><para>All paths in the closure of
<replaceable>paths</replaceable> are determined (using
<command>nix-store --query --requisites
--include-outputs</command>). Note that since the
<option>--include-outputs</option> flag is used, if
<replaceable>paths</replaceable> includes a store derivation, you
get a combined source/binary distribution (e.g., source tarballs
will be included).</para></listitem>
<listitem><para>All store paths determined in the previous step are
packaged and compressed into a <command>bzip</command>ped NAR
archive (extension <filename>.nar.bz2</filename>).</para></listitem>
packaged into a NAR (using <command>nix-store --dump</command>) and
compressed using <command>xz</command> or <command>bzip2</command>.
The resulting files have the extension <filename>.nar.xz</filename>
or <filename>.nar.bz2</filename>. Also for each store path, Nix
generates a file with extension <filename>.narinfo</filename>
containing metadata such as the references, cryptographic hash and
size of each path.</para></listitem>
<listitem><para>A <emphasis>manifest</emphasis> is created that
contains information on the store paths, their eventual URLs in the
cache, and cryptographic hashes of the contents of the NAR
archives.</para></listitem>
<listitem><para>Optionally, a single <emphasis>manifest</emphasis>
file is created that contains the same metadata as the
<filename>.narinfo</filename> files. This is for compatibility with
Nix versions prior to 1.2 (see <command>nix-pull</command> for
details).</para></listitem>
<listitem><para>Each store path is uploaded to the remote directory
specified by <replaceable>archivesPutURL</replaceable>. HTTP PUT
requests are used to do this. However, before a file
<varname>x</varname> is uploaded to
<literal><replaceable>archivesPutURL</replaceable>/</literal><varname>x</varname>,
<command>nix-push</command> first determines whether this upload is
unnecessary by issuing a HTTP HEAD request on
<literal><replaceable>archivesGetURL</replaceable>/</literal><varname>x</varname>.
This allows a cache to be shared between many partially overlapping
<command>nix-push</command> invocations. (We use two URLs because
the upload URL typically refers to a CGI script, while the download
URL just refers to a file system directory on the
server.)</para></listitem>
<listitem><para>The manifest is uploaded using an HTTP PUT request
to <replaceable>manifestPutURL</replaceable>. The corresponding
URL to download the manifest can then be used by
<command>nix-pull</command>.</para></listitem>
<listitem><para>A file named <option>nix-cache-info</option> is
placed in the destination directory. The existence of this file
marks the directory as a binary cache.</para></listitem>
</orderedlist>
</para>
<!--
<para>TODO: <option>- -copy</option></para>
-->
</refsection>
<refsection><title>Options</title>
<variablelist>
<varlistentry><term><option>--dest</option> <replaceable>dest-dir</replaceable></term>
<listitem><para>Set the destination directory to
<replaceable>dir</replaceable>, which is created if it does not
exist. This flag is required.</para></listitem>
</varlistentry>
<varlistentry><term><option>--bzip2</option></term>
<listitem><para>Compress NARs using <command>bzip2</command>
instead of <command>xz -9</command>. The latter compresses about
30% better on typical archives, decompresses about twice as fast,
but compresses a lot slower and is not supported by Nix prior to
version 1.2.</para></listitem>
</varlistentry>
<varlistentry><term><option>--force</option></term>
<listitem><para>Overwrite <filename>.narinfo</filename> files if
they already exist.</para></listitem>
</varlistentry>
<varlistentry><term><option>--link</option></term>
<listitem><para>By default, NARs are generated in the Nix store
and then copied to <replaceable>dest-dir</replaceable>. If this
option is given, hard links are used instead. This only works if
<replaceable>dest-dir</replaceable> is on the same filesystem as
the Nix store.</para></listitem>
</varlistentry>
<varlistentry><term><option>--manifest</option></term>
<listitem><para>Force the generation of a manifest suitable for
use by <command>nix-pull</command>. The manifest is stored as
<filename><replaceable>dest-dir</replaceable>/MANIFEST</filename>.</para></listitem>
</varlistentry>
<varlistentry><term><option>--manifest-path</option> <replaceable>filename</replaceable></term>
<listitem><para>Like <option>--manifest</option>, but store the
manifest in <replaceable>filename</replaceable>.</para></listitem>
</varlistentry>
<varlistentry><term><option>--url-prefix</option> <replaceable>url</replaceable></term>
<listitem><para>Manifests are expected to contain the absolute
URLs of NARs. For generating these URLs, the prefix
<replaceable>url</replaceable> is used. It defaults to
<uri>file://<replaceable>dest-dir</replaceable></uri>.</para></listitem>
</varlistentry>
</variablelist>
</refsection>
<refsection><title>Examples</title>
<para>To upload files there typically is some CGI script on the server
side. This script should be be protected with a password. The
following example uploads the store paths resulting from building the
Nix expressions in <filename>foo.nix</filename>, passing appropriate
authentication information:
<para>To add the closure of Thunderbird to a binary cache:
<screen>
$ nix-push \
http://foo@bar:server.domain/cgi-bin/upload.pl/cache \
http://server.domain/cache \
http://foo@bar:server.domain/cgi-bin/upload.pl/MANIFEST \
$(nix-instantiate foo.nix)</screen>
$ nix-push --dest /tmp/cache $(nix-build -A thunderbird)
</screen>
This will push both sources and binaries (and any build-time
dependencies used in the build, such as compilers).</para>
<para>If we just want to push binaries, not sources and build-time
dependencies, we can do:
Assuming that <filename>/tmp/cache</filename> is exported by a web
server as <uri>http://example.org/cache</uri>, you can then use this
cache on another machine to speed up the installation of Thunderbird:
<screen>
$ nix-push <replaceable>urls</replaceable> $(nix-store -r $(nix-instantiate foo.nix))</screen>
$ nix-build -A thunderbird --option binary-caches http://example.org/cache
</screen>
Alternatively, you could add <literal>binary-caches =
http://example.org/cache</literal> to
<filename>nix.conf</filename>.</para>
<para>To also include build-time dependencies (such as source
tarballs):
<screen>
$ nix-push --dest /tmp/cache $(nix-instantiate -A thunderbird)
</screen>
</para>
<para>To generate a manifest suitable for <command>nix-pull</command>:
<screen>
$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) --manifest
</screen>
On another machine you can then do:
<screen>
$ nix-pull http://example.org/cache
</screen>
to cause the binaries to be used by subsequent Nix operations.</para>
</refsection>
<refsection><title>Binary cache format and operation</title>
<para>A binary cache with URL <replaceable>url</replaceable> only
denotes a valid binary cache if the file
<uri><replaceable>url</replaceable>/nix-cache-info</uri> exists. If
this file does not exist (or cannot be downloaded), the cache is
ignored. If it does exist, it must be a text file containing cache
properties. Here’s an example:
<screen>
StoreDir: /nix/store
WantMassQuery: 1
</screen>
The properties that are currently supported are:
<variablelist>
<varlistentry><term><literal>StoreDir</literal></term>
<listitem><para>The path of the Nix store to which this binary
cache applies. Binaries are not relocatable — a binary built for
<filename>/nix/store</filename> won’t generally work in
<filename>/home/alice/store</filename> — so to prevent binaries
from being used in a wrong store, a binary cache is only used if
its <literal>StoreDir</literal> matches the local Nix
configuration. The default is
<filename>/nix/store</filename>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>WantMassQuery</literal></term>
<listitem><para>Query operations such as <command>nix-env
-qas</command> can cause thousands of cache queries, and thus
thousands of HTTP requests, to determine which packages are
available in binary form. While these requests are small, not
every server may appreciate a potential onslaught of queries. If
<literal>WantMassQuery</literal> is set to <literal>0</literal>
(default), “mass queries” such as <command>nix-env -qas</command>
will skip this cache. Thus a package may appear not to have a
binary substitute. However, the binary will still be used when
you actually install the package. If
<literal>WantMassQuery</literal> is set to <literal>1</literal>,
mass queries will use this cache.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>Every time Nix needs to build some store path
<replaceable>p</replaceable>, it will check each configured binary
cache to see if it has a NAR file for <replaceable>p</replaceable>,
until it finds one. If no cache has a NAR, Nix will fall back to
building the path from source (if applicable). To see if a cache with
URL <replaceable>url</replaceable> has a binary for
<replaceable>p</replaceable>, Nix fetches
<replaceable>url/h</replaceable>, where <replaceable>h</replaceable>
is the hash part of <replaceable>p</replaceable>. Thus, if we have a
cache <uri>http://nixos.org/binary-cache</uri> and we want to obtain
the store path
<screen>
/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13
</screen>
then Nix will attempt to fetch
<screen>
http://nixos.org/binary-cache/cj7a81wsm1ijwwpkks3725661h3263p5.narinfo
</screen>
(Commands such as <command>nix-env -qas</command> will issue an HTTP
HEAD request, since it only needs to know if the
<filename>.narinfo</filename> file exists.) The
<filename>.narinfo</filename> file is a simple text file that looks
like this:
<screen>
StorePath: /nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13
URL: nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2
Compression: bzip2
FileHash: sha256:0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21
FileSize: 10119014
NarHash: sha256:120mhshkcgvn48xvxz0fjbd47k615v0viv7jy4gy7dwhmqapgd9d
NarSize: 33702192
References: cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14
Deriver: xs32zzf2d58f6l5iz5fgwxrds2q5pnvn-glibc-2.13.drv
System: x86_64-linux
</screen>
The fields are as follows:
<variablelist>
<varlistentry><term><literal>StorePath</literal></term>
<listitem><para>The full store path, including the name part
(e.g., <literal>glibc-2.13</literal>). It must match the
requested store path.</para></listitem>
</varlistentry>
<varlistentry><term><literal>URL</literal></term>
<listitem><para>The URL of the NAR, relative to the binary cache
URL.</para></listitem>
</varlistentry>
<varlistentry><term><literal>Compression</literal></term>
<listitem><para>The compression method; either
<literal>xz</literal> or
<literal>bzip2</literal>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>FileHash</literal></term>
<listitem><para>The SHA-256 hash of the compressed
NAR.</para></listitem>
</varlistentry>
<varlistentry><term><literal>FileSize</literal></term>
<listitem><para>The size of the compressed NAR.</para></listitem>
</varlistentry>
<varlistentry><term><literal>NarHash</literal></term>
<listitem><para>The SHA-256 hash of the uncompressed NAR. This is
equal to the hash of the store path as returned by
<command>nix-store -q --hash
<replaceable>p</replaceable></command>.</para></listitem>
</varlistentry>
<varlistentry><term><literal>NarSize</literal></term>
<listitem><para>The size of the uncompressed NAR.</para></listitem>
</varlistentry>
<varlistentry><term><literal>References</literal></term>
<listitem><para>The references of the store path, without the Nix
store prefix.</para></listitem>
</varlistentry>
<varlistentry><term><literal>Deriver</literal></term>
<listitem><para>The deriver of the store path, without the Nix
store prefix. This field is optional.</para></listitem>
</varlistentry>
<varlistentry><term><literal>System</literal></term>
<listitem><para>The Nix platform type of this binary, if known.
This field is optional.</para></listitem>
</varlistentry>
</variablelist>
</para>
<para>Thus, in our example, after recursively ensuring that the
references exist (e.g.,
<filename>/nix/store/jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14</filename>),
Nix will fetch <screen>
http://nixos.org/binary-cache/nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2
</screen> and decompress and unpack it to
<filename>/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13</filename>.</para>
</refsection>
</refentry>