cafkafk/nixpkgs
cafkafk/nixpkgs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2

doc: update content on dockerTools layered images and follow doc conventions

Browse source
This commit is contained in:
DS 2024-01-15 14:22:51 -08:00 committed by Silvan Mosberger
parent 14bbc48cc6
commit 2004205b7a
1 changed files with 338 additions and 103 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

441
doc/build-helpers/images/dockertools.section.md
View file

@ -12,9 +12,9 @@ This function will create a single layer for all files (and dependencies) that a
Only new dependencies that are not already in the existing layers will be copied. Only new dependencies that are not already in the existing layers will be copied.
If you prefer to create multiple layers for the files and dependencies you want to add to the image, see [](#ssec-pkgs-dockerTools-buildLayeredImage) or [](#ssec-pkgs-dockerTools-streamLayeredImage) instead. If you prefer to create multiple layers for the files and dependencies you want to add to the image, see [](#ssec-pkgs-dockerTools-buildLayeredImage) or [](#ssec-pkgs-dockerTools-streamLayeredImage) instead.
`buildImage` allows scripts to be run during the layer generation process, allowing custom behaviour to affect the contents of the image (see the documentation of the `runAsRoot` and `extraCommands` attributes). This function allows a script to be run during the layer generation process, allowing custom behaviour to affect the final results of the image (see the documentation of the `runAsRoot` and `extraCommands` attributes).
The resulting repository tarball will only list a single image as specified by the `name` and `tag` attributes. The resulting repository tarball will list a single image as specified by the `name` and `tag` attributes.
By default, that image will use a static creation date (see documentation for the `created` attribute). By default, that image will use a static creation date (see documentation for the `created` attribute).
This allows `buildImage` to produce reproducible images. This allows `buildImage` to produce reproducible images.
@ -77,7 +77,6 @@ Similarly, if you encounter errors similar to `Error_Protocol ("certificate has
`copyToRoot` (Path, List of Paths, or Null; _optional_) `copyToRoot` (Path, List of Paths, or Null; _optional_)
: Files to add to the generated image. : Files to add to the generated image.
This can be either a path or a list of paths.
Anything that coerces to a path (e.g. a derivation) can also be used. Anything that coerces to a path (e.g. a derivation) can also be used.
This can be seen as an equivalent of `ADD contents/ /` in a `Dockerfile`. This can be seen as an equivalent of `ADD contents/ /` in a `Dockerfile`.
@ -322,7 +321,6 @@ dockerTools.buildImage {
::: :::
:::{.example #ex-dockerTools-buildImage-creatednow} :::{.example #ex-dockerTools-buildImage-creatednow}
# Building a Docker image with a creation date set to the current time # Building a Docker image with a creation date set to the current time
Note that using a value of `"now"` in the `created` attribute will break reproducibility. Note that using a value of `"now"` in the `created` attribute will break reproducibility.
@ -352,132 +350,369 @@ $ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE REPOSITORY TAG IMAGE ID CREATED SIZE
hello latest de2bf4786de6 About a minute ago 25.2MB hello latest de2bf4786de6 About a minute ago 25.2MB
``` ```
::: :::
## buildLayeredImage {#ssec-pkgs-dockerTools-buildLayeredImage} ## buildLayeredImage {#ssec-pkgs-dockerTools-buildLayeredImage}
Create a Docker image with many of the store paths being on their own layer to improve sharing between images. The image is realized into the Nix store as a gzipped tarball. Depending on the intended usage, many users might prefer to use `streamLayeredImage` instead, which this function uses internally. `buildLayeredImage` uses [`streamLayeredImage`](#ssec-pkgs-dockerTools-streamLayeredImage) underneath to build a compressed Docker-compatible repository tarball.
Basically, `buildLayeredImage` runs the script created by `streamLayeredImage` to save the compressed image in the Nix store.
`buildLayeredImage` supports the same options as `streamLayeredImage`, see [`streamLayeredImage`](#ssec-pkgs-dockerTools-streamLayeredImage) for details.
`name` :::{.note}
Despite the similar name, [`buildImage`](#ssec-pkgs-dockerTools-buildImage) works completely differently from `buildLayeredImage` and `streamLayeredImage`.
: The name of the resulting image. Even though some of the arguments may seem related, they cannot be interchanged.
:::
`tag` _optional_ You can use this function to load an image in Docker with `docker load`.
See [](#ex-dockerTools-buildLayeredImage-hello) to see how to do that.
: Tag of the generated image. ### Examples {#ssec-pkgs-dockerTools-buildLayeredImage-examples}
*Default:* the output path's hash :::{.example #ex-dockerTools-buildLayeredImage-hello}
# Building a layered Docker image
`fromImage` _optional_ The following package builds a layered Docker image that runs the `hello` executable from the `hello` package.
The Docker image will have name `hello` and tag `latest`.
: The repository tarball containing the base image. It must be a valid Docker image, such as one exported by `docker save`.
*Default:* `null`, which can be seen as equivalent to `FROM scratch` of a `Dockerfile`.
`contents` _optional_
: Top-level paths in the container. Either a single derivation, or a list of derivations.
*Default:* `[]`
`config` _optional_
`architecture` is _optional_ and used to specify the image architecture, this is useful for multi-architecture builds that don't need cross compiling. If not specified it will default to `hostPlatform`.
: Run-time configuration of the container. A full list of the options available is in the [Docker Image Specification v1.2.0](https://github.com/moby/moby/blob/master/image/spec/v1.2.md#image-json-field-descriptions).
*Default:* `{}`
`created` _optional_
: Date and time the layers were created. Follows the same `now` exception supported by `buildImage`.
*Default:* `1970-01-01T00:00:01Z`
`maxLayers` _optional_
: Maximum number of layers to create.
*Default:* `100`
*Maximum:* `125`
`extraCommands` _optional_
: Shell commands to run while building the final layer, without access to most of the layer contents. Changes to this layer are "on top" of all the other layers, so can create additional directories and files.
`fakeRootCommands` _optional_
: Shell commands to run while creating the archive for the final layer in a fakeroot environment. Unlike `extraCommands`, you can run `chown` to change the owners of the files in the archive, changing fakeroot's state instead of the real filesystem. The latter would require privileges that the build user does not have. Static binaries do not interact with the fakeroot environment. By default all files in the archive will be owned by root.
`enableFakechroot` _optional_
: Whether to run in `fakeRootCommands` in `fakechroot`, making programs behave as though `/` is the root of the image being created, while files in the Nix store are available as usual. This allows scripts that perform installation in `/` to work as expected. Considering that `fakechroot` is implemented via the same mechanism as `fakeroot`, the same caveats apply.
*Default:* `false`
### Behavior of `contents` in the final image {#dockerTools-buildLayeredImage-arg-contents}
Each path directly listed in `contents` will have a symlink in the root of the image.
For example:
```nix ```nix
pkgs.dockerTools.buildLayeredImage { { dockerTools, hello }:
dockerTools.buildLayeredImage {
name = "hello"; name = "hello";
contents = [ pkgs.hello ]; tag = "latest";
contents = [ hello ];
config.Cmd = [ "/bin/hello" ];
} }
``` ```
will create symlinks for all the paths in the `hello` package: The result of building this package is a `.tar.gz` file that can be loaded into Docker:
```ShellSession ```shell
/bin/hello -> /nix/store/h1zb1padqbbb7jicsvkmrym3r6snphxg-hello-2.10/bin/hello $ nix-build
/share/info/hello.info -> /nix/store/h1zb1padqbbb7jicsvkmrym3r6snphxg-hello-2.10/share/info/hello.info (some output removed for clarity)
/share/locale/bg/LC_MESSAGES/hello.mo -> /nix/store/h1zb1padqbbb7jicsvkmrym3r6snphxg-hello-2.10/share/locale/bg/LC_MESSAGES/hello.mo building '/nix/store/bk8bnrbw10nq7p8pvcmdr0qf57y6scha-hello.tar.gz.drv'...
No 'fromImage' provided
Creating layer 1 from paths: ['/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1']
Creating layer 2 from paths: ['/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4']
Creating layer 3 from paths: ['/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc']
Creating layer 4 from paths: ['/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27']
Creating layer 5 from paths: ['/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1']
Creating layer 6 with customisation...
Adding manifests...
Done.
/nix/store/hxcz7snvw7f8rzhbh6mv8jq39d992905-hello.tar.gz
$ docker load -i /nix/store/hxcz7snvw7f8rzhbh6mv8jq39d992905-hello.tar.gz
(some output removed for clarity)
Loaded image: hello:latest
``` ```
:::
### Automatic inclusion of `config` references {#dockerTools-buildLayeredImage-arg-config}
The closure of `config` is automatically included in the closure of the final image.
This allows you to make very simple Docker images with very little code. This container will start up and run `hello`:
```nix
pkgs.dockerTools.buildLayeredImage {
name = "hello";
config.Cmd = [ "${pkgs.hello}/bin/hello" ];
}
```
### Adjusting `maxLayers` {#dockerTools-buildLayeredImage-arg-maxLayers}
Increasing the `maxLayers` increases the number of layers which have a chance to be shared between different images.
Modern Docker installations support up to 128 layers, but older versions support as few as 42.
If the produced image will not be extended by other Docker builds, it is safe to set `maxLayers` to `128`. However, it will be impossible to extend the image further.
The first (`maxLayers-2`) most "popular" paths will have their own individual layers, then layer \#`maxLayers-1` will contain all the remaining "unpopular" paths, and finally layer \#`maxLayers` will contain the Image configuration.
Docker's Layers are not inherently ordered, they are content-addressable and are not explicitly layered until they are composed in to an Image.
## streamLayeredImage {#ssec-pkgs-dockerTools-streamLayeredImage} ## streamLayeredImage {#ssec-pkgs-dockerTools-streamLayeredImage}
Builds a script which, when run, will stream an uncompressed tarball of a Docker image to stdout. The arguments to this function are as for `buildLayeredImage`. This method of constructing an image does not realize the image into the Nix store, so it saves on IO and disk/cache space, particularly with large images. `streamLayeredImage` builds a **script** which, when run, will stream to stdout a Docker-compatible repository tarball containing a single image, using multiple layers to improve sharing between images.
This means that `streamLayeredImage` does not output an image into the Nix store, but only a script that builds the image, saving on IO and disk/cache space, particularly with large images.
The image produced by running the output script can be piped directly into `docker load`, to load it into the local docker daemon: You can use this function to load an image in Docker with `docker load`.
See [](#ex-dockerTools-streamLayeredImage-hello) to see how to do that.
```ShellSession For this function, you specify a [store path](https://nixos.org/manual/nix/stable/store/store-path) or a list of store paths to be added to the image, and the functions will automatically include any dependencies of those paths in the image.
$(nix-build) | docker load The function will attempt to create one layer per object in the Nix store that needs to be added to the image.
In case there are more objects to include than available layers, the function will put the most ["popular"](https://github.com/NixOS/nixpkgs/tree/release-23.11/pkgs/build-support/references-by-popularity) objects in their own layers, and group all remaining objects into a single layer.
An additional layer will be created with symlinks to the store paths you specified to be included in the image.
These symlinks are built with [`symlinkJoin`](#trivial-builder-symlinkJoin), so they will be included in the root of the image.
See [](#ex-dockerTools-streamLayeredImage-exploringlayers) to understand how these symlinks are laid out in the generated image.
`streamLayeredImage` allows scripts to be run when creating the additional layer with symlinks, allowing custom behaviour to affect the final results of the image (see the documentation of the `extraCommands` and `fakeRootCommands` attributes).
The resulting repository tarball will list a single image as specified by the `name` and `tag` attributes.
By default, that image will use a static creation date (see documentation for the `created` attribute).
This allows the function to produce reproducible images.
### Inputs {#ssec-pkgs-dockerTools-streamLayeredImage-inputs}
`streamLayeredImage` expects one argument with the following attributes:
`name` (String)
: The name of the generated image.
`tag` (String; _optional_)
: Tag of the generated image.
If `null`, the hash of the nix derivation will be used as the tag.
_Default value:_ `null`.
`fromImage`(Path or Null; _optional_)
: The repository tarball of an image to be used as the base for the generated image.
It must be a valid Docker image, such as one exported by `docker save`, or another image built with the `dockerTools` utility functions.
This can be seen as an equivalent of `FROM fromImage` in a `Dockerfile`.
A value of `null` can be seen as an equivalent of `FROM scratch`.
If specified, the created layers will be appended to the layers defined in the base image.
_Default value:_ `null`.
`contents` (Path or List of Paths; _optional_) []{#dockerTools-buildLayeredImage-arg-contents}
: Directories whose contents will be added to the generated image.
Things that coerce to paths (e.g. a derivation) can also be used.
This can be seen as an equivalent of `ADD contents/ /` in a `Dockerfile`.
All the contents specified by `contents` will be added as a final layer in the generated image.
They will be added as links to the actual files (e.g. links to the store paths).
The actual files will be added in previous layers.
_Default value:_ `[]`
`config` (Attribute Set; _optional_) []{#dockerTools-buildLayeredImage-arg-config}
: Used to specify the configuration of the containers that will be started off the generated image.
Must be an attribute set, with each attribute as listed in the [Docker Image Specification v1.3.0](https://github.com/moby/moby/blob/46f7ab808b9504d735d600e259ca0723f76fb164/image/spec/spec.md#image-json-field-descriptions).
If any packages are used directly in `config`, they will be automatically included in the generated image.
See [](#ex-dockerTools-streamLayeredImage-configclosure) for an example.
_Default value:_ `null`.
`architecture` (String; _optional_)
: Used to specify the image architecture.
This is useful for multi-architecture builds that don't need cross compiling.
If specified, its value should follow the [OCI Image Configuration Specification](https://github.com/opencontainers/image-spec/blob/main/config.md#properties), which should still be compatible with Docker.
According to the linked specification, all possible values for `$GOARCH` in [the Go docs](https://go.dev/doc/install/source#environment) should be valid, but will commonly be one of `386`, `amd64`, `arm`, or `arm64`.
_Default value:_ the same value from `pkgs.go.GOARCH`.
`created` (String; _optional_)
: Specifies the time of creation of the generated image.
This should be either a date and time formatted according to [ISO-8601](https://en.wikipedia.org/wiki/ISO_8601) or `"now"`, in which case the current date will be used.
:::{.caution}
Using `"now"` means that the generated image will not be reproducible anymore (because the date will always change whenever it's built).
:::
_Default value:_ `"1970-01-01T00:00:01Z"`.
`maxLayers` (Number; _optional_) []{#dockerTools-buildLayeredImage-arg-maxLayers}
: The maximum number of layers that will be used by the generated image.
If a `fromImage` was specified, the number of layers used by `fromImage` will be subtracted from `maxLayers` to ensure that the image generated will have at most `maxLayers`.
:::{.caution}
Depending on the tool/runtime where the image will be used, there might be a limit to the number of layers that an image can have.
For Docker, see [this issue on GitHub](https://github.com/docker/docs/issues/8230).
:::
_Default value:_ 100.
`extraCommands` (String; _optional_)
: A bash script that will run in the context of the layer created with the contents specified by `contents`.
At the moment this script runs, only the contents directly specified by `contents` will be available as links.
_Default value:_ `""`.
`fakeRootCommands` (String; _optional_)
: A bash script that will run in the context of the layer created with the contents specified by `contents`.
During the process to generate that layer, the script in `extraCommands` will be run first, if specified.
After that, a {manpage}`fakeroot(1)` environment will be entered.
The script specified in `fakeRootCommands` runs inside the fakeroot environment, and the layer is then generated from the view of the files inside the fakeroot environment.
This is useful to change the owners of the files in the layer (by running `chown`, for example), or performing any other privileged operations related to file manipulation (by default, all files in the layer will be owned by root, and the build environment doesn't have enough privileges to directly perform privileged operations on these files).
For more details, see the manpage for {manpage}`fakeroot(1)`.
:::{.caution}
Due to how fakeroot works, static binaries cannot perform privileged file operations in `fakeRootCommands`, unless `enableFakechroot` is set to `true`.
:::
_Default value:_ `""`.
`enableFakechroot` (Boolean; _optional_)
: By default, the script specified in `fakeRootCommands` only runs inside a fakeroot environment.
If `enableFakechroot` is `true`, a more complete chroot environment will be created using [`proot`](https://proot-me.github.io/) before running the script in `fakeRootCommands`.
Files in the Nix store will be available.
This allows scripts that perform installation in `/` to work as expected.
This can be seen as an equivalent of `RUN ...` in a `Dockerfile`.
_Default value:_ `false`
`includeStorePaths` (Boolean; _optional_)
: The files specified in `contents` are put into layers in the generated image.
If `includeStorePaths` is `false`, the actual files will not be included in the generated image, and only links to them will be added instead.
It is **not recommended** to set this to `false` unless you have other tooling to insert the store paths via other means (such as bind mounting the host store) when running containers with the generated image.
If you don't provide any extra tooling, the generated image won't run properly.
See [](#ex-dockerTools-streamLayeredImage-exploringlayers) to understand the impact of setting `includeStorePaths` to `false`.
_Default value:_ `true`
`passthru` (Attribute Set; _optional_)
: Use this to pass any attributes as [passthru](#var-stdenv-passthru) for the resulting derivation.
_Default value:_ `{}`
### Passthru outputs {#ssec-pkgs-dockerTools-streamLayeredImage-passthru-outputs}
`streamLayeredImage` also defines its own [`passthru`](#var-stdenv-passthru) attributes:
`imageTag` (String)
: The tag of the generated image.
This is useful if no tag was specified in the attributes of the argument to the function, because an automatic tag will be used instead.
`imageTag` allows you to retrieve the value of the tag used in this case.
### Examples {#ssec-pkgs-dockerTools-streamLayeredImage-examples}
:::{.example #ex-dockerTools-streamLayeredImage-hello}
# Streaming a layered Docker image
The following package builds a **script** which, when run, will stream a layered Docker image that runs the `hello` executable from the `hello` package.
The Docker image will have name `hello` and tag `latest`.
```nix
{ dockerTools, hello }:
dockerTools.streamLayeredImage {
name = "hello";
tag = "latest";
contents = [ hello ];
config.Cmd = [ "/bin/hello" ];
}
``` ```
Alternatively, the image be piped via `gzip` into `skopeo`, e.g., to copy it into a registry: The result of building this package is a script.
Running this script and piping it into `docker load` gives you the same image that was built in [](#ex-dockerTools-buildLayeredImage-hello).
Note that in this case, the image is never added to the Nix store, but instead streamed directly into Docker.
```ShellSession ```shell
$(nix-build) | gzip --fast | skopeo copy docker-archive:/dev/stdin docker://some_docker_registry/myimage:tag $ nix-build
(output removed for clarity)
/nix/store/wsz2xl8ckxnlb769irvq6jv1280dfvxd-stream-hello
$ /nix/store/wsz2xl8ckxnlb769irvq6jv1280dfvxd-stream-hello | docker load
No 'fromImage' provided
Creating layer 1 from paths: ['/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1']
Creating layer 2 from paths: ['/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4']
Creating layer 3 from paths: ['/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc']
Creating layer 4 from paths: ['/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27']
Creating layer 5 from paths: ['/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1']
Creating layer 6 with customisation...
Adding manifests...
Done.
(some output removed for clarity)
Loaded image: hello:latest
``` ```
:::
:::{.example #ex-dockerTools-streamLayeredImage-exploringlayers}
# Exploring the layers in an image built with `streamLayeredImage`
Assume the following package, which builds a layered Docker image with the `hello` package.
```nix
{ dockerTools, hello }:
dockerTools.streamLayeredImage {
name = "hello";
contents = [ hello ];
}
```
The `hello` package depends on 4 other packages:
```shell
$ nix-store --query -R $(nix-build -A hello)
/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1
/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4
/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc
/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27
/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1
```
This means that all these packages will be included in the image generated by `streamLayeredImage`.
It will put each package in its own layer, for a total of 5 layers with actual files in them.
A final layer will be created only with symlinks for the `hello` package.
The image generated will have the following directory structure (some directories were collapsed for readability):
```
├── bin
│ └── hello → /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1/bin/hello
├── nix
│ └── store
│ ├─⊕ 9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27
│ ├─⊕ i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1
│ ├─⊕ ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4
│ ├─⊕ ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc
│ └─⊕ zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1
└── share
├── info
│ └── hello.info → /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1/share/info/hello.info
├─⊕ locale
└── man
└── man1
└── hello.1.gz → /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1/share/man/man1/hello.1.gz
```
Each of the packages in `/nix/store` comes from a layer in the image.
The final layer adds the `/bin` and `/share` directories, but they only contain links to the actual files in `/nix/store`.
If our package sets `includeStorePaths` to `false`, we'll end up with only the final layer with the links, but the actual files won't exist in the image:
```nix
{ dockerTools, hello }:
dockerTools.streamLayeredImage {
name = "hello";
contents = [ hello ];
}
```
After building this package, the image will have the following directory structure:
```
├── bin
│ └── hello → /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1/bin/hello
└── share
├── info
│ └── hello.info → /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1/share/info/hello.info
├─⊕ locale
└── man
└── man1
└── hello.1.gz → /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1/share/man/man1/hello.1.gz
```
Note how the links point to paths in `/nix/store`, but they're not included in the image itself.
This is why you need extra tooling when using `includeStorePaths`:
a container created from such image won't find any of the files it needs to run otherwise.
:::
::: {.example #ex-dockerTools-streamLayeredImage-configclosure}
# Building a layered Docker image with packages directly in `config`
The closure of `config` is automatically included in the generated image.
The following package shows a more compact way to create the same output generated in [](#ex-dockerTools-streamLayeredImage-hello).
```nix
{ dockerTools, hello, lib }:
dockerTools.streamLayeredImage {
name = "hello";
tag = "latest";
config.Cmd = [ "${lib.getExe hello}" ];
}
```
:::
## pullImage {#ssec-pkgs-dockerTools-fetchFromRegistry} ## pullImage {#ssec-pkgs-dockerTools-fetchFromRegistry}