cafkafk/nixpkgs
cafkafk/nixpkgs
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
nixpkgs/pkgs/build-support/docker/store-path-to-layer.sh

55 lines
1.6 KiB
Bash
Raw Normal View History

treewide: fixup bcf54ce fix the executable bit for scripts installed with substituteAll and some remaining shebangs.
2019-03-06 00:10:34 +01:00
#!@shell@
dockerTools.buildLayeredImage: init Create a many-layered Docker Image. Implements much less than buildImage: - Doesn't support specific uids/gids - Doesn't support runninng commands after building - Doesn't require qemu - Doesn't create mutable copies of the files in the path - Doesn't support parent images If you want those feature, I recommend using buildLayeredImage as an input to buildImage. Notably, it does support: - Caching low level, common paths based on a graph traversial algorithm, see referencesByPopularity in 0a80233487993256e811f566b1c80a40394c03d6 - Configurable number of layers. If you're not using AUFS or not extending the image, you can specify a larger number of layers at build time: pkgs.dockerTools.buildLayeredImage { name = "hello"; maxLayers = 128; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; - Parallelized creation of the layers, improving build speed. - The contents of the image includes the closure of the configuration, so you don't have to specify paths in contents and config. With buildImage, paths referred to by the config were not included automatically in the image. Thus, if you wanted to call Git, you had to specify it twice: pkgs.dockerTools.buildImage { name = "hello"; contents = [ pkgs.gitFull ]; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; buildLayeredImage on the other hand includes the runtime closure of the config when calculating the contents of the image: pkgs.dockerTools.buildImage { name = "hello"; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; Minor Problems - If any of the store paths change, every layer will be rebuilt in the nix-build. However, beacuse the layers are bit-for-bit reproducable, when these images are loaded in to Docker they will match existing layers and not be imported or uploaded twice. Common Questions - Aren't Docker layers ordered? No. People who have used a Dockerfile before assume Docker's Layers are inherently ordered. However, this is not true -- Docker layers are content-addressable and are not explicitly layered until they are composed in to an Image. - What happens if I have more than maxLayers of store paths? The first (maxLayers-2) most "popular" paths will have their own individual layers, then layer #(maxLayers-1) will contain all the remaining "unpopular" paths, and finally layer #(maxLayers) will contain the Image configuration.
2018-09-25 16:53:42 +02:00
set -eu
layerNumber=$1
shift
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
storePath="$1"
shift
dockerTools.buildLayeredImage: init Create a many-layered Docker Image. Implements much less than buildImage: - Doesn't support specific uids/gids - Doesn't support runninng commands after building - Doesn't require qemu - Doesn't create mutable copies of the files in the path - Doesn't support parent images If you want those feature, I recommend using buildLayeredImage as an input to buildImage. Notably, it does support: - Caching low level, common paths based on a graph traversial algorithm, see referencesByPopularity in 0a80233487993256e811f566b1c80a40394c03d6 - Configurable number of layers. If you're not using AUFS or not extending the image, you can specify a larger number of layers at build time: pkgs.dockerTools.buildLayeredImage { name = "hello"; maxLayers = 128; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; - Parallelized creation of the layers, improving build speed. - The contents of the image includes the closure of the configuration, so you don't have to specify paths in contents and config. With buildImage, paths referred to by the config were not included automatically in the image. Thus, if you wanted to call Git, you had to specify it twice: pkgs.dockerTools.buildImage { name = "hello"; contents = [ pkgs.gitFull ]; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; buildLayeredImage on the other hand includes the runtime closure of the config when calculating the contents of the image: pkgs.dockerTools.buildImage { name = "hello"; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; Minor Problems - If any of the store paths change, every layer will be rebuilt in the nix-build. However, beacuse the layers are bit-for-bit reproducable, when these images are loaded in to Docker they will match existing layers and not be imported or uploaded twice. Common Questions - Aren't Docker layers ordered? No. People who have used a Dockerfile before assume Docker's Layers are inherently ordered. However, this is not true -- Docker layers are content-addressable and are not explicitly layered until they are composed in to an Image. - What happens if I have more than maxLayers of store paths? The first (maxLayers-2) most "popular" paths will have their own individual layers, then layer #(maxLayers-1) will contain all the remaining "unpopular" paths, and finally layer #(maxLayers) will contain the Image configuration.
2018-09-25 16:53:42 +02:00
layerPath="./layers/$layerNumber"
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
echo "Creating layer #$layerNumber for $storePath"
dockerTools.buildLayeredImage: init Create a many-layered Docker Image. Implements much less than buildImage: - Doesn't support specific uids/gids - Doesn't support runninng commands after building - Doesn't require qemu - Doesn't create mutable copies of the files in the path - Doesn't support parent images If you want those feature, I recommend using buildLayeredImage as an input to buildImage. Notably, it does support: - Caching low level, common paths based on a graph traversial algorithm, see referencesByPopularity in 0a80233487993256e811f566b1c80a40394c03d6 - Configurable number of layers. If you're not using AUFS or not extending the image, you can specify a larger number of layers at build time: pkgs.dockerTools.buildLayeredImage { name = "hello"; maxLayers = 128; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; - Parallelized creation of the layers, improving build speed. - The contents of the image includes the closure of the configuration, so you don't have to specify paths in contents and config. With buildImage, paths referred to by the config were not included automatically in the image. Thus, if you wanted to call Git, you had to specify it twice: pkgs.dockerTools.buildImage { name = "hello"; contents = [ pkgs.gitFull ]; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; buildLayeredImage on the other hand includes the runtime closure of the config when calculating the contents of the image: pkgs.dockerTools.buildImage { name = "hello"; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; Minor Problems - If any of the store paths change, every layer will be rebuilt in the nix-build. However, beacuse the layers are bit-for-bit reproducable, when these images are loaded in to Docker they will match existing layers and not be imported or uploaded twice. Common Questions - Aren't Docker layers ordered? No. People who have used a Dockerfile before assume Docker's Layers are inherently ordered. However, this is not true -- Docker layers are content-addressable and are not explicitly layered until they are composed in to an Image. - What happens if I have more than maxLayers of store paths? The first (maxLayers-2) most "popular" paths will have their own individual layers, then layer #(maxLayers-1) will contain all the remaining "unpopular" paths, and finally layer #(maxLayers) will contain the Image configuration.
2018-09-25 16:53:42 +02:00
mkdir -p "$layerPath"
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
dockerTools.buildLayeredImage: fix typo in comments
2020-01-11 09:02:30 +01:00
# Make sure /nix and /nix/store appear first in the archive.
# We create the directories here and use them because
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
# when there are other things being added to the
# nix store, tar could fail, saying,
# "tar: /nix/store: file changed as we read it"
mkdir -p nix/store
tar -cf "$layerPath/layer.tar" \
dockerTools.buildLayeredImage: restore layer sharing PR #58431 added /nix/store to each layer.tar. However, the timestamp was not explicitly set while adding /nix and /nix/store to the archive. This resulted in different SHA256 hashes of layer.tar between image builds. This change sets time and owner when tar'ing /nix/store.
2019-06-05 11:10:00 +02:00
--mtime="@$SOURCE_DATE_EPOCH" \
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
--owner=0 --group=0 \
--transform='s,nix,/nix,' \
nix
dockerTools.buildLayeredImage: fix typo in comments
2020-01-11 09:02:30 +01:00
# We change into the /nix/store in order to avoid a similar
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
# "file changed as we read it" error as above. Namely,
# if we use the absolute path of /nix/store/123-pkg
dockerTools.buildLayeredImage: fix typo in comments
2020-01-11 09:02:30 +01:00
# and something new is added to the nix store while tar
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
# is running, it will detect a change to /nix/store and
# fail. Instead, if we cd into the nix store and copy
# the relative nix store path, tar will ignore changes
# to /nix/store. In order to create the correct structure
# in the tar file, we transform the relative nix store
dockerTools.buildLayeredImage: fix typo in comments
2020-01-11 09:02:30 +01:00
# path to the absolute store path.
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
n=$(basename "$storePath")
tar -C /nix/store -rpf "$layerPath/layer.tar" \
--hard-dereference --sort=name \
dockerTools.buildLayeredImage: init Create a many-layered Docker Image. Implements much less than buildImage: - Doesn't support specific uids/gids - Doesn't support runninng commands after building - Doesn't require qemu - Doesn't create mutable copies of the files in the path - Doesn't support parent images If you want those feature, I recommend using buildLayeredImage as an input to buildImage. Notably, it does support: - Caching low level, common paths based on a graph traversial algorithm, see referencesByPopularity in 0a80233487993256e811f566b1c80a40394c03d6 - Configurable number of layers. If you're not using AUFS or not extending the image, you can specify a larger number of layers at build time: pkgs.dockerTools.buildLayeredImage { name = "hello"; maxLayers = 128; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; - Parallelized creation of the layers, improving build speed. - The contents of the image includes the closure of the configuration, so you don't have to specify paths in contents and config. With buildImage, paths referred to by the config were not included automatically in the image. Thus, if you wanted to call Git, you had to specify it twice: pkgs.dockerTools.buildImage { name = "hello"; contents = [ pkgs.gitFull ]; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; buildLayeredImage on the other hand includes the runtime closure of the config when calculating the contents of the image: pkgs.dockerTools.buildImage { name = "hello"; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; Minor Problems - If any of the store paths change, every layer will be rebuilt in the nix-build. However, beacuse the layers are bit-for-bit reproducable, when these images are loaded in to Docker they will match existing layers and not be imported or uploaded twice. Common Questions - Aren't Docker layers ordered? No. People who have used a Dockerfile before assume Docker's Layers are inherently ordered. However, this is not true -- Docker layers are content-addressable and are not explicitly layered until they are composed in to an Image. - What happens if I have more than maxLayers of store paths? The first (maxLayers-2) most "popular" paths will have their own individual layers, then layer #(maxLayers-1) will contain all the remaining "unpopular" paths, and finally layer #(maxLayers) will contain the Image configuration.
2018-09-25 16:53:42 +02:00
--mtime="@$SOURCE_DATE_EPOCH" \
dockerTools.buildLayeredImage: fix building layered images in parallel when tar'ing store paths into layered archives when building layered images, don't use the absolute nix store path so that tar won't complain if something new is added to the nix store when building the final docker image, ignore any file changes tar detects in the layers. they are all immutable and the only thing that might change is the number of hard links due to store optimization
2019-12-30 22:45:10 +01:00
--owner=0 --group=0 \
--transform="s,$n,/nix/store/$n," \
$n
dockerTools.buildLayeredImage: init Create a many-layered Docker Image. Implements much less than buildImage: - Doesn't support specific uids/gids - Doesn't support runninng commands after building - Doesn't require qemu - Doesn't create mutable copies of the files in the path - Doesn't support parent images If you want those feature, I recommend using buildLayeredImage as an input to buildImage. Notably, it does support: - Caching low level, common paths based on a graph traversial algorithm, see referencesByPopularity in 0a80233487993256e811f566b1c80a40394c03d6 - Configurable number of layers. If you're not using AUFS or not extending the image, you can specify a larger number of layers at build time: pkgs.dockerTools.buildLayeredImage { name = "hello"; maxLayers = 128; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; - Parallelized creation of the layers, improving build speed. - The contents of the image includes the closure of the configuration, so you don't have to specify paths in contents and config. With buildImage, paths referred to by the config were not included automatically in the image. Thus, if you wanted to call Git, you had to specify it twice: pkgs.dockerTools.buildImage { name = "hello"; contents = [ pkgs.gitFull ]; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; buildLayeredImage on the other hand includes the runtime closure of the config when calculating the contents of the image: pkgs.dockerTools.buildImage { name = "hello"; config.Cmd = [ "${pkgs.gitFull}/bin/git" ]; }; Minor Problems - If any of the store paths change, every layer will be rebuilt in the nix-build. However, beacuse the layers are bit-for-bit reproducable, when these images are loaded in to Docker they will match existing layers and not be imported or uploaded twice. Common Questions - Aren't Docker layers ordered? No. People who have used a Dockerfile before assume Docker's Layers are inherently ordered. However, this is not true -- Docker layers are content-addressable and are not explicitly layered until they are composed in to an Image. - What happens if I have more than maxLayers of store paths? The first (maxLayers-2) most "popular" paths will have their own individual layers, then layer #(maxLayers-1) will contain all the remaining "unpopular" paths, and finally layer #(maxLayers) will contain the Image configuration.
2018-09-25 16:53:42 +02:00
# Compute a checksum of the tarball.
tarhash=$(tarsum < $layerPath/layer.tar)
# Add a 'checksum' field to the JSON, with the value set to the
# checksum of the tarball.
cat ./generic.json | jshon -s "$tarhash" -i checksum > $layerPath/json
# Indicate to docker that we're using schema version 1.0.
echo -n "1.0" > $layerPath/VERSION
Reference in a new issue Copy permalink