When building an image with multiple layers, files
already included in an underlying layer are supposed to
be excluded from the current layer. However, some subtleties
in the way filepaths are compared seem to be blocking this.
Specifically:
* tar generates relative filepaths with directories ending in '/'
* find generates absolute filepaths with no trailing slashes on directories
That is, paths extracted from the underlying tarball look like:
nix/store/.../foobar/
whereas the layer being generated uses paths like:
/nix/store/.../foobar
This patch modifies the output of "tar -t" to match the latter format.
They now go to devman, devdoc, or $outputMan, in that order. This is
to prevent cases such as the man-pages package quietly losing its
section 3 pages.
This builds elisp to setup an emacs buffer with the packages given
available. See shlevy/nix-buffer for more information.
Currently only modifies $PATH.
This one was already merged into release-16.09, so let's not have the
stable branch is ahead of master and confuse things. In addition to
that, currently we have an odd situation that master has less things
actually finished building than in staging.
Conflicts:
pkgs/data/documentation/man-pages/default.nix
This was one of the ways to build packages, we are trying
hard to minimize different ways so it's easier for newcomers
to learn only one way.
This also:
- removes texLive (old), fixes#14807
- removed upstream-updater, if that code is still used it should be in
separate repo
- changes a few packages like gitit/mit-scheme to use new texlive
Previously, features.grsecurity wasn't actually set due to a bug in the
grsec builder. We now rely on the generic kernel builder to set features
from kernelPatches.
This reverts commit da68127737, reversing
changes made to 2ba494b728.
These changes are reverted because they cause an incompatibility which
they should not.
"emacsWrapper" replaces emacsWithPackages. In addition to "packagesFun",
emacsWrapper has an optional variable called "execStart". execStart can
be used to append elisp to the default site-start.el script. This is
useful for providing a way to load a user's .emacs.d/init.el
file. "emacsWithPackages" is implemented with emacsWrapper for
convenience and compatability.
Sometimes patches start without a leading prefix. We default to strip
one prefix or path component from patches (-p1) in the patchPhase in
stdenv.
As all patches should therefore be in this format, fetchpatch should
have an option to normalize patch paths. This commit introduces a new
argument to fetchpatch called addPrefixes that adds one patch prefix to
the old and new paths in a patch before putting it into the store.
fetchMavenArtifact downloads a Maven artifact given a group id, an artifact id,
and a version.
Example usage:
org_apache_httpcomponents_httpclient_4_5_2 = fetchMavenArtifact {
groupId = "org.apache.httpcomponents";
artifactId = "httpclient";
version = "4.5.2";
sha256 = "0ms00zc28pwqk83nwwbafhq6p8zci9mrjzbqalpn6v0d80hwdzqd";
# Optionally: repos = [ ... urls to some Maven repos to use ... ];
# Optionally: url, urls - pointing directly to a specific jar url.
};
Now `org_apache_httpcomponents_httpclient_4_5_2.jar` points to the downloaded
JAR file, while `org_apache_httpcomponents_httpclient_4_5_2` refers to a
derivation that when used used in `buildInputs` will be automatically added to
the Java classpath.
downloads.sourceforge.net is the official way to download tarballs from
SourceForge. However, it is reported as unreliable due to SF's weird
load balancing system.
This commit gives the official mirror utmost priority, and will use
other configured mirrors (which may be temporary) as a fallback only
when the official one can't be reached/download fails/hangs.
References: NixOs/nixpkgs#16900
This patch fixes#16614 and #16741.
The first issue was caused by the fact that both `/share` and
`/share/fish/vendor_completions.d` end in the `pathsToLink`. The
`pkgs/build-support/buildenv/builder.pl` creates `/share`, then links
`/share/fish` under `/share` and then tries to create the directory
`/share/fish/vendor_completions.d` and fails because it already exists.
The simplest way to reproduce the issue is to build the next Nix
expression:
```nix
let pkgs = import <nixpkgs> { };
in pkgs.buildEnv {
name = "buildenv-issue";
paths = [
pkgs.fish
pkgs.vim
];
pathsToLink = [
"/share"
"/share/fish/vendor_completions.d"
];
}
```
The second issue is more critical and was caused by the fact findFiles
doesn't recurse deep enough. It stops at first unique directory for the
package (e.g., "/share" or even "/") and later the scripts decides it
shouldn't link it as it doesn't match pathsToLink (e.g., "/share/fish"),
so the result is empty.
The test:
```nix
let pkgs = import <nixpkgs> { };
in pkgs.buildEnv {
name = "buildenv-issue";
paths = [
pkgs.fish
pkgs.vim
];
pathsToLink = [
"/share/fish/functions"
];
}
```
or
```nix
let pkgs = import <nixpkgs> { };
in pkgs.buildEnv {
name = "buildenv-issue";
paths = [
pkgs.vim
];
pathsToLink = [
"/share"
];
}
```
Prior to this commit, trailing whitespace would be introduced when
modifying '#!' lines with no arguments. For example (whitespace added):
/nix/store/.../foo: interpreter directive changed
from "/bin/bash"
to "/nix/store/...-bash-4.3-p42/bin/bash "
/nix/store/.../bar: interpreter directive changed
from "/bin/baz wef"
to "/nix/store/...-baz wef "
We add a sed command to strip trailing whitespace, so the above commands
would drop the two spaces after "bash", or the one space after "baz wef".
abbradar: fixed commit title
Closes#16785.
This fixes a regression caused by commit f56ab9e
("nix-prefetch-git: Include the date in the machine-readable [...]")
where a couple of directory paths printed by pushd/popd appeared before
the JSON output on stdout (thus breaking it). Fix it by redirecting the
extraneous output to /dev/null.
Reported by Michael Alan Dorman <mdorman@ironicdesign.com>.
stdout, in strict ISO 8601 format.
This will be helpful for automatically updating fetchgit expressions
and the dates in version numbers associated with them.
stripHash uses a global variable to communicate it's computation
results, but it's not necessary. You can just pipe to stdout in a
subshell. A function mostly behaves like just another command.
baseHash() also introduces a suffix-stripping capability since it's
something the users of the function tend to use.
The tests need to expand passed variable and very carefully.
I could see no other easy way than to change single-quoting in
makeWrapper to double-quoting.
The tests now fail with the same problem as on master...
This patch replaces the old grsecurity kernels with a single NixOS
specific grsecurity kernel. This kernel is intended as a general
purpose kernel, tuned for casual desktop use.
Providing only a single kernel may seem like a regression compared to
offering a multitude of flavors. It is impossible, however, to
effectively test and support that many options. This is amplified by
the reality that very few seem to actually use grsecurity on NixOS,
meaning that bugs go unnoticed for long periods of time, simply because
those code paths end up never being exercised. More generally, it is
hopeless to anticipate imagined needs. It is better to start from a
solid foundation and possibly add more flavours on demand.
While the generic kernel is intended to cover a wide range of use cases,
it cannot cover everything. For some, the configuration will be either
too restrictive or too lenient. In those cases, the recommended
solution is to build a custom kernel --- this is *strongly* recommended
for security sensitive deployments.
Building a custom grsec kernel should be as simple as
```nix
linux_grsec_nixos.override {
extraConfig = ''
GRKERNSEC y
PAX y
# and so on ...
'';
}
```
The generic kernel should be usable both as a KVM guest and host. When
running as a host, the kernel assumes hardware virtualisation support.
Virtualisation systems other than KVM are *unsupported*: users of
non-KVM systems are better served by compiling a custom kernel.
Unlike previous Grsecurity kernels, this configuration disables `/proc`
restrictions in favor of `security.hideProcessInformation`.
Known incompatibilities:
- ZFS: can't load spl and zfs kernel modules; claims incompatibility
with KERNEXEC method `or` and RAP; changing to `bts` does not fix the
problem, which implies we'd have to disable RAP as well for ZFS to
work
- `kexec()`: likely incompatible with KERNEXEC (unverified)
- Xen: likely incompatible with KERNEXEC and UDEREF (unverified)
- Virtualbox: likely incompatible with UDEREF (unverified)
Close#15803. This avoids the error:
while setting up the build environment: executing
‘/nix/store/7sb42axk5lrxqz45nldrb2pchlys14s1-bash-4.3-p42/bin/bash’:
Argument list too long
Note: I wanted to make it optional based on buildCommand length,
but that seems pointless as I'm sure it's less performant.
Amended by vcunat:
https://github.com/NixOS/nixpkgs/pull/15803#issuecomment-224841225
This takes another approach at binding FHS directory structure. We
now bind-mount all the root filesystem to directory "/host" in the target tree.
From that we symlink all the directories into the tree if they do not already
exist in FHS structure.
This probably makes `CHROOTENV_EXTRA_BINDS` unnecessary -- its main usecase was
to add bound directories from the host to the sandbox, and we not just symlink
all of them. I plan to get some feedback on its usage and maybe deprecate it.
This also drops old `buildFHSChrootEnv` infrastructure. The main problem with it
is it's very difficult to unmount a recursive-bound directory when mount is not
sandboxed. This problem is a bug even without these changes -- if
you have for example `/home/alice` mounted to somewhere, you wouldn't see
it in `buildFHSChrootEnv` now. With the new directory structure, it's
impossible to use regular bind at all. After some tackling with this I realized
that the fix would be brittle and dangerous (if you don't unmount everything
clearly and proceed to removing the temporary directory, bye-bye fs!). It also
probably doesn't worth it because I haven't heard that someone actually uses it
for a long time, and `buildFHSUserEnv` should cover most cases while being much
more maintainable and safe for the end-user.
1. When multiple versions of the same package are required
$revs is an array.
2. When cargo fetch is run it usually doesn't need a network
connection. But when it does SSL_CERT_FILE isn't set.
find fails when called with an inexistent search path.
That situation may arise when the output is created after by a postFixup hook.
vcunat amended the PR by clarifying one more `return` to `return 0`.
Checkinstall had two problems:
1. when it was called without a version (e.g. with a derivation created
by fetchFromGitHub) it would use `src` as debian version, which caused
dpkg to fail
2. when dpkg failed, it would invoke the pager with the log, which hangs
the build
So now
1. the default version is the dummy `0.0.0`
2. the used pager is `cat`
Source of this change goes back to 2009 and original version of
fetchgit at 205fb0c87e.
The nondeterminism is really caused by changing .git so leave other
files alone as they might be interesting.
Note: this causes a hash mismatch with Hydra's version of Git Plugin
which we should fix to comply.
Move all the dependencies to their own derivation, so that we don't publish all
of them if the wrapper is installed in a profile.
The previous solution just moved them to a custom directory to avoid conflicts,
this refactors that and completely hides them, while preserving the desired
improvement of adding only one directory to each of the emacs search paths
The chroot caps restriction disallows chroot'ed processes from running
any command that requires `CAP_SYS_ADMIN`, breaking `nixos-rebuild`. See
e.g., https://github.com/NixOS/nixpkgs/issues/15293
This significantly weakens chroot protections, but to break
nixos-rebuild out of the box is too severe.
There are no users of it in main tree and recent merge
of multiple outputs branch makes it obsolete for private trees
too.
At the time hook was created, recently merged multiple output
branch was relying on passing flags to autotools to split
outputs, which obviously wasn't working for other build systems
Scatter output was taking different approach where files were
moved out from a build tree based on known paths, which is more
or less what current multiple-outputs.sh hook is able to do too.
A nix specific set of tools for converting icon files
that are not in a freedesktop ready format.
I plan on using these tools for both `keepass` and
`retroarch` packages. It may benifit many other packages.
For every *.{exe,dll} in $output/bin/ we try to find all (potential)
transitive dependencies and symlink those DLLs into $output/bin
so they are found on invocation.
(DLLs are first searched in the directory of the running exe file.)
The links are relative, so relocating whole /nix/store won't break them.
The hook is activated on cygwin and when cross-compiling to mingw.
- the default --docdir is typically DATAROOTDIR/doc/pkgName
- I saw no other way than to employ some magic to guess this `pkgName`
- user can override it by setting $shareDocName
I supplied meta.outputsToInstall automatically in all
mkDerivation products, but some packages still don't use it.
The reported case: jekyll -> bundlerEnv -> buildEnv -> runCommand.
The /nix path in 4d200538 of the layer tar didn't exist for some
packages, such as cacert. This is because cacert just creates an /etc
directory and doesn't depend on any other /nix paths. If we tried
putting this directory in the tar and using overlayfs with it, we'd get
"Invalid argument" when trying to remove the directory.
We now check whether the closure is non-empty before telling tar to
store the /nix directory.
Fixes#14710.
There were two sources of non-determinisim coming into the images. The
first was tar mtimes, the second was pigz/gzip times.
An example image now passes with the --check flag.
I think the intention of this functionality was to provide a simple
alternative to the "runAsRoot" and "contents" attributes.
The implementation caused very slow builds of Docker images. Almost all
of the build time was spent in IO for tar, due to tarballs being
created, immediately extracted, then recreated. I had 30 minute builds
on some of my images which are now down to less than 2 minutes. A couple
of other users on #nix IRC have observed similar improvements.
The implementation also mutated the produced Docker layers without
changing their hashes. Using non-empty tarballs would produce images
which got cached incorrectly in Docker.
I have a commit which just fixes the performance problem but I opted to
completely remove the tarball feature after I found out that it didn't
correctly implement the Docker Image Specification due to the broken
hashing.
This is to avoid unwanted side effects when installing a wrapped emacs in the environment:
* All executables in the dependencies become available in the user environment
* All site-lisp binaries in the dependencies become accessible to unwrapped emacs
Also, both bin and site-lisp would generate conflicts so installing a wrapped emacs becomes really cumbersome
1. Update bower2nix version and add new/updated dependencies into
node-packages-generated.nix. This was done manually, with npm2nix
generating the initial set of derivations. In future, it would be
nice to have an automatic process (see #10358, #9332).
2. Add an override to nodePackages.bower2nix wrapping the commands so
that git is on the PATH.
3. Update fetchbower to support new command-line options of bower2nix,
and to allow github URL tag versions.
Previously, nix-prefetch-git would report the same JSON whether submodules were being fetched or not; with this change, the --fetch-submodules option will cause the JSON output to include "fetchSubmodules": true, so that fetchgit (builtins.fromJSON (builtins.readFile ./path/to/output.json)) will work.
