Commit graph

114 commits

Author SHA1 Message Date
Eelco Dolstra
42c5774e78
Sort substituters by priority
Fixes .
2017-07-04 16:34:53 +02:00
Eelco Dolstra
b7203e853e
getDefaultSubstituters(): Simplify initialisation
As shlevy pointed out, static variables in C++11 have thread-safe
initialisation.
2017-07-04 16:26:48 +02:00
Eelco Dolstra
c0015e87af
Support base-64 hashes
Also simplify the Hash API.

Fixes .
2017-07-04 15:07:41 +02:00
Eelco Dolstra
fcca702a96
Replace a few bool flags with enums
Functions like copyClosure() had 3 bool arguments, which creates a
severe risk of mixing up arguments.

Also, implement copyClosure() using copyPaths().
2017-07-03 11:38:08 +02:00
Eelco Dolstra
6e01ecd112
Fix nix-copy-closure test
Fixes

  client# error: size mismatch importing path ‘/nix/store/ywf5fihjlxwijm6ygh6s0a353b5yvq4d-libidn2-0.16’; expected 0, got 120264

This is mostly an artifact of the NixOS VM test environment, where the
Nix database doesn't contain hashes/sizes.

http://hydra.nixos.org/build/53537471
2017-05-29 16:08:56 +02:00
Eelco Dolstra
b01d62285c
Improve progress indicator 2017-05-16 16:09:57 +02:00
Eelco Dolstra
62d476c7ee
Fix typo 2017-05-11 14:02:03 +02:00
Eelco Dolstra
7689181e4f
Minor cleanup 2017-05-08 15:56:52 +02:00
Eelco Dolstra
3a5f04f48c
build-remote: Don't require signatures
This restores the old behaviour.
2017-05-01 20:03:25 +02:00
Eelco Dolstra
031d70e500
Support arbitrary store URIs in nix.machines
For backwards compatibility, if the URI is just a hostname, ssh://
(i.e. LegacySSHStore) is prepended automatically.

Also, all fields except the URI are now optional. For example, this is
a valid nix.machines file:

  local?root=/tmp/nix

This is useful for testing the remote build machinery since you don't
have to mess around with ssh.
2017-05-01 17:35:30 +02:00
Eelco Dolstra
0dddcf867a
Add a dummy Store::buildPaths() method
This default implementation of buildPaths() does nothing if all
requested paths are already valid, and throws an "unsupported
operation" error otherwise. This fixes a regression introduced by
c30330df6f in binary cache and legacy
SSH stores.
2017-05-01 13:43:34 +02:00
Eelco Dolstra
ba9ad29fdb
Convert Settings to the new config system
This makes all config options self-documenting.

Unknown or unparseable config settings and --option flags now cause a
warning.
2017-04-13 20:53:23 +02:00
Eelco Dolstra
1860070548
Merge branch 'rework-options' of https://github.com/copumpkin/nix 2017-04-13 16:15:51 +02:00
Eelco Dolstra
2040240e23
Add a Config class to simplify adding configuration settings
The typical use is to inherit Config and add Setting<T> members:

  class MyClass : private Config
  {
    Setting<int> foo{this, 123, "foo", "the number of foos to use"};
    Setting<std::string> bar{this, "blabla", "bar", "the name of the bar"};

    MyClass() : Config(readConfigFile("/etc/my-app.conf"))
    {
      std::cout << foo << "\n"; // will print 123 unless overriden
    }
  };

Currently, this is used by Store and its subclasses for store
parameters. You now get a warning if you specify a non-existant store
parameter in a store URI.
2017-04-13 16:03:31 +02:00
Eelco Dolstra
95295482ea
Allow "auto" as a store URI
Using the empty string is likely to be ambiguous in some contexts.
2017-04-10 11:27:29 +02:00
Eelco Dolstra
256940fc48
nix-daemon: Disable path info cache
This is useless because the client also caches path info, and can
cause problems for long-running clients like hydra-queue-runner
(i.e. it may return cached info about paths that have been
garbage-collected).
2017-04-06 14:30:31 +02:00
Eelco Dolstra
e1e49c58e1
Only use cache.nixos.org when the store is /nix/store
This is consistent with the behaviour of the old
download-from-binary-cache substituter.
2017-03-21 17:59:18 +01:00
Eelco Dolstra
4bb38591e5
Restore cache.nixos.org as the default substituter
Fixes .
2017-03-21 15:06:46 +01:00
Eelco Dolstra
c5b83d8913
copyPaths(): Use queryValidPaths() to reduce SSH latency 2017-03-16 13:50:01 +01:00
Eelco Dolstra
91d67692cf
copyPaths(): Don't query path info for a path the target already has
For example, this cuts "nix-copy-closure --from" on a NixOS system
closure from 15.9s to 0.5s.
2017-03-16 12:05:51 +01:00
Eelco Dolstra
d3eb1cf3bb
build-remote: Don't use a SSH master
This is unnecessary because we make only one connection.
2017-03-03 19:23:20 +01:00
Dan Peebles
e7cb2847ab Explicitly model all settings and fail on unrecognized ones
Previously, the Settings class allowed other code to query for string
properties, which led to a proliferation of code all over the place making
up new options without any sort of central registry of valid options. This
commit pulls all those options back into the central Settings class and
removes the public get() methods, to discourage future abuses like that.

Furthermore, because we know the full set of options ahead of time, we
now fail loudly if someone enters an unrecognized option, thus preventing
subtle typos. With some template fun, we could probably also dump the full
set of options (with documentation, defaults, etc.) to the command line,
but I'm not doing that yet here.
2017-02-22 20:19:25 -05:00
Eelco Dolstra
f023f64f40
RemoteStore::addToStore(): Pass content-addressability assertion
... and use this in Downloader::downloadCached(). This fixes

  $ nix-build https://nixos.org/channels/nixos-16.09-small/nixexprs.tar.xz -A hello
  error: cannot import path ‘/nix/store/csfbp1s60dkgmk9f8g0zk0mwb7hzgabd-nixexprs.tar.xz’ because it lacks a valid signature
2017-02-22 16:58:00 +01:00
Eelco Dolstra
6f4682ad36
Merge branch 'nix-copy-closure-c++' of https://github.com/shlevy/nix 2017-02-07 20:47:45 +01:00
Eelco Dolstra
f38224e924
copyStorePath(): Don't require signatures for "trusted" stores
For example, SSH stores could be trusted.
2017-02-07 19:29:21 +01:00
Eelco Dolstra
fa07558a06
Provide default implementations for a couple of Store methods 2017-02-07 19:29:21 +01:00
Eelco Dolstra
c2b0d8749f
exportReferencesGraph: Export more complete info in JSON format
This writes info about every path in the closure in the same format as
‘nix path-info --json’. Thus it also includes NAR hashes and sizes.

Example:

  [
    {
      "path": "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
      "narHash": "sha256:0ckdc4z20kkmpqdilx0wl6cricxv90lh85xpv2qljppcmz6vzcxl",
      "narSize": 197648,
      "references": [
        "/nix/store/10h6li26i7g6z3mdpvra09yyf10mmzdr-hello-2.10",
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20939776
    },
    {
      "path": "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24",
      "narHash": "sha256:1nfn3m3p98y1c0kd0brp80dn9n5mycwgrk183j17rajya0h7gax3",
      "narSize": 20742128,
      "references": [
        "/nix/store/27binbdy296qvjycdgr1535v8872vz3z-glibc-2.24"
      ],
      "closureSize": 20742128
    }
  ]

Fixes .
2017-01-26 20:41:08 +01:00
Shea Levy
bfa41eb671 nix-copy-closure: Implement in C++.
Tests fail currently because the database is not given proper hashes in the VM
2017-01-20 09:47:58 -05:00
Shea Levy
167d12b02c build-remote: Implement in C++ 2016-11-10 11:09:15 -05:00
Eelco Dolstra
e4655b166c copyClosure() / copyStorePath(): Expose dontCheckSigs
Needed by Hydra.
2016-11-09 19:08:20 +01:00
Eelco Dolstra
4b8f1b0ec0 Merge branch 'ssh-store' of https://github.com/shlevy/nix 2016-11-09 18:57:22 +01:00
Eelco Dolstra
542ae5c8f8 BinaryCacheStore: Optionally write a NAR listing
The store parameter "write-nar-listing=1" will cause BinaryCacheStore
to write a file ‘<store-hash>.ls.xz’ for each ‘<store-hash>.narinfo’
added to the binary cache. This file contains an XZ-compressed JSON
file describing the contents of the NAR, excluding the contents of
regular files.

E.g.

  {
    "version": 1,
    "root": {
      "type": "directory",
      "entries": {
        "lib": {
          "type": "directory",
          "entries": {
            "Mcrt1.o": {
              "type": "regular",
              "size": 1288
            },
            "Scrt1.o": {
              "type": "regular",
              "size": 3920
            },
          }
        }
      }
      ...
    }
  }

(The actual file has no indentation.)

This is intended to speed up the NixOS channels programs index
generator [1], since fetching gazillions of large NARs from
cache.nixos.org is currently a bottleneck for updating the regular
(non-small) channel.

[1] https://github.com/NixOS/nixos-channel-scripts/blob/master/generate-programs-index.cc
2016-10-21 16:50:28 +02:00
Eelco Dolstra
844219f364 Store::queryValidPaths(): Use async queryPathInfo()
This allows the binary cache substituter to pipeline requests.
2016-10-07 19:43:36 +02:00
Eelco Dolstra
c663b84573 Implement generic Store::queryValidPaths() 2016-10-07 19:20:47 +02:00
Eelco Dolstra
35db4f65a0 Add copyClosure utility function for Hydra 2016-10-07 19:15:25 +02:00
Eelco Dolstra
c55bf085eb printMsg(lvlError, ...) -> printError(...) etc. 2016-09-21 16:54:53 +02:00
Eelco Dolstra
75989bdca7 Make computeFSClosure() single-threaded again
The fact that queryPathInfo() is synchronous meant that we needed a
thread for every concurrent binary cache lookup, even though they end
up being handled by the same download thread. Requiring hundreds of
threads is not a good idea. So now there is an asynchronous version of
queryPathInfo() that takes a callback function to process the
result. Similarly, enqueueDownload() now takes a callback rather than
returning a future.

Thus, a command like

  nix path-info --store https://cache.nixos.org/ -r /nix/store/slljrzwmpygy1daay14kjszsr9xix063-nixos-16.09beta231.dccf8c5

that returns 4941 paths now takes 1.87s using only 2 threads (the main
thread and the downloader thread). (This is with a prewarmed
CloudFront.)
2016-09-16 18:54:14 +02:00
Shea Levy
0f39633290 Factor out the unix domain socket-specific code from RemoteStore 2016-09-02 14:15:04 -04:00
Shea Levy
53b27ddce2 Factor a function to get the store type from a URI out of the main RegisterStoreImplementation 2016-09-02 06:39:29 -04:00
Shea Levy
a91954f0c6 Merge openStore and openStoreAt with default arguments 2016-09-02 06:35:48 -04:00
Eelco Dolstra
d961c29c9c Mark content-addressed paths in the Nix database and in .narinfo
This allows such paths to be imported without signatures.
2016-08-10 18:05:35 +02:00
Eelco Dolstra
ee22a91ab8 makeFixedOutputPath(): Drop superfluous HashType argument 2016-07-26 21:25:52 +02:00
Eelco Dolstra
74dd603495 Re-implement negative binary cache lookup caching 2016-06-20 17:39:05 +02:00
Eelco Dolstra
812c0dfbe2 Allow setting the state directory as a store parameter
E.g. "local?store=/tmp/store&state=/tmp/var".
2016-06-02 16:02:48 +02:00
Eelco Dolstra
7850d3d279 Make the store directory a member variable of Store 2016-06-01 16:24:17 +02:00
Eelco Dolstra
b66ab6cdbc Fix repair during substitution 2016-05-30 15:18:12 +02:00
Eelco Dolstra
538a64e8c3 Add a Store::addToStore() variant that accepts a NAR
As a side effect, this ensures that signatures are propagated when
copying paths between stores.

Also refactored import/export to make use of this.
2016-05-04 16:15:54 +02:00
Eelco Dolstra
f435f82475 Remove OpenSSL-based signing 2016-05-04 11:01:48 +02:00
Eelco Dolstra
dfebfc835f Add a copyStorePath() utility function 2016-05-04 11:01:48 +02:00
Eelco Dolstra
95d20dfde9 Allow parameters in store URIs
This is to allow store-specific configuration,
e.g. s3://my-cache?compression=bzip2&secret-key=/path/to/key.
2016-04-29 16:26:16 +02:00