cafkafk/lix
cafkafk/lix
1
0
Fork 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

Merge pull request #2456 from grahamc/s3-substituters

Browse source 
Update docs to describe how s3:// URLS does in fact support endpoint, region, and profile for upload
This commit is contained in:
Eelco Dolstra 2018-10-01 22:47:00 +02:00 committed by GitHub
commit aa5e47b2f4
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 104 additions and 92 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

196
doc/manual/packages/s3-substituter.xml
View file

@ -12,8 +12,49 @@ from Amazon S3 and S3 compatible services. This uses the same
<emphasis>binary</emphasis> cache mechanism that Nix usually uses to <emphasis>binary</emphasis> cache mechanism that Nix usually uses to
fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para> fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
<para>The following options can be specified as URL parameters to
the S3 URL:</para>
<variablelist>
<varlistentry><term><literal>profile</literal></term>
<listitem>
<para>
The name of the AWS configuration profile to use. By default
Nix will use the <literal>default</literal> profile.
</para>
</listitem>
</varlistentry>
<varlistentry><term><literal>region</literal></term>
<listitem>
<para>
The region of the S3 bucket. <literal>us–east-1</literal> by
default.
</para>
<para>
If your bucket is not in <literal>us–east-1</literal>, you
should always explicitly specify the region parameter.
</para>
</listitem>
</varlistentry>
<varlistentry><term><literal>endpoint</literal></term>
<listitem>
<para>
The URL to your S3-compatible service, for when not using
Amazon S3. Do not specify this value if you're using Amazon
S3.
</para>
<note><para>This endpoint must support HTTPS and will use
path-based addressing instead of virtual host based
addressing.</para></note>
</listitem>
</varlistentry>
</variablelist>
<para>In this example we will use the bucket named <para>In this example we will use the bucket named
<literal>example-bucket</literal>.</para> <literal>example-nix-cache</literal>.</para>
<section xml:id="ssec-s3-substituter-anonymous-reads"> <section xml:id="ssec-s3-substituter-anonymous-reads">
<title>Anonymous Reads to your S3-compatible binary cache</title> <title>Anonymous Reads to your S3-compatible binary cache</title>
@ -24,65 +65,56 @@ fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
cache.</para> cache.</para>
<para>For AWS S3 the binary cache URL for example bucket will be <para>For AWS S3 the binary cache URL for example bucket will be
exactly <uri>https://example-bucket.s3.amazonaws.com</uri>. For S3 exactly <uri>https://example-nix-cache.s3.amazonaws.com</uri> or
compatible binary caches ago have to consult your software's <uri>s3://example-nix-cache</uri>. For S3 compatible binary caches,
documentation.</para> consult that cache's documentation.</para>
<para>Your bucket will need the following bucket policy:</para> <para>Your bucket will need the following bucket policy:</para>
<programlisting> <programlisting><![CDATA[
<![CDATA[
{ {
"Id": "DirectReads", "Id": "DirectReads",
"Version": "2012-10-17", "Version": "2012-10-17",
"Statement": [ "Statement": [
{ {
"Sid": "AlowDirectReads", "Sid": "AlowDirectReads",
"Action": [ "Action": [
"s3:GetObject" "s3:GetObject",
], "s3:GetBucketLocation"
"Effect": "Allow", ],
"Resource": "arn:aws:s3:::example-bucket/*", "Effect": "Allow",
"Principal": "*" "Resource": [
} "arn:aws:s3:::example-nix-cache",
] "arn:aws:s3:::example-nix-cache/*"
],
"Principal": "*"
}
]
} }
]]> ]]></programlisting>
</programlisting>
</section> </section>
<section xml:id="ssec-s3-substituter-authenticated-reads"> <section xml:id="ssec-s3-substituter-authenticated-reads">
<title>Authenticated Reads to your S3 binary cache</title> <title>Authenticated Reads to your S3 binary cache</title>
<para>For AWS S3 the binary cache URL for example bucket will be <para>For AWS S3 the binary cache URL for example bucket will be
exactly <uri>s3://example-bucket</uri>.</para> exactly <uri>s3://example-nix-cache</uri>.</para>
<para>Nix will use the <link <para>Nix will use the <link
xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default
credential provider chain</link> for authenticating requests to credential provider chain</link> for authenticating requests to
Amazon S3.</para> Amazon S3.</para>
<para>Nix supports authenticated writes to S3 compatible binary <para>Nix supports authenticated reads from Amazon S3 and S3
caches but only supports Authenticated reads from Amazon S3. compatible binary caches.</para>
Additionally, the following limitations are in place for
authenticated reads:</para>
<itemizedlist>
<listitem><para>The bucket must actually be hosted by Amazon S3 and
<emphasis>not</emphasis> an S3 compatible
service.</para></listitem>
<listitem><para>The bucket must be within the
<literal>us-east-1</literal> region.</para></listitem>
<listitem><para>The Amazon credentials, if stored in a credential
profile, must be stored in the <literal>default</literal>
profile.</para></listitem>
</itemizedlist>
<para>Your bucket will need a bucket policy allowing the desired <para>Your bucket will need a bucket policy allowing the desired
users to perform the <literal>s3:GetObject</literal> action on all users to perform the <literal>s3:GetObject</literal> and
objects in the bucket.</para> <literal>s3:GetBucketLocation</literal> action on all objects in the
bucket. The anonymous policy in <xref
linkend="ssec-s3-substituter-anonymous-reads" /> can be updated to
have a restricted <literal>Principal</literal> to support
this.</para>
</section> </section>
@ -91,69 +123,49 @@ fetch prebuilt binaries from <uri>cache.nixos.org</uri>.</para>
<para>Nix support fully supports writing to Amazon S3 and S3 <para>Nix support fully supports writing to Amazon S3 and S3
compatible buckets. The binary cache URL for our example bucket will compatible buckets. The binary cache URL for our example bucket will
be <uri>s3://example-bucket</uri>.</para> be <uri>s3://example-nix-cache</uri>.</para>
<para>Nix will use the <link <para>Nix will use the <link
xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default xlink:href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default.">default
credential provider chain</link> for authenticating requests to credential provider chain</link> for authenticating requests to
Amazon S3.</para> Amazon S3.</para>
<para>The following options can be specified as URL parameters to <para>Your account will need the following IAM policy to
the S3 URL:</para> upload to the cache:</para>
<variablelist>
<varlistentry><term><literal>profile</literal></term>
<listitem>
<para>
The name of the AWS configuration profile to use. By default
Nix will use the <literal>default</literal> profile.
</para>
</listitem>
</varlistentry>
<varlistentry><term><literal>region</literal></term> <programlisting><![CDATA[
<listitem> {
<para> "Version": "2012-10-17",
The region of the S3 bucket. <literal>us–east-1</literal> by "Statement": [
default. {
</para> "Sid": "UploadToCache",
</listitem> "Effect": "Allow",
</varlistentry> "Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:ListBucketMultipartUploads",
"s3:ListMultipartUploadParts",
"s3:ListObjects",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::example-nix-cache",
"arn:aws:s3:::example-nix-cache/*"
]
}
]
}
]]></programlisting>
<varlistentry><term><literal>endpoint</literal></term>
<listitem>
<para>
The URL to your S3-compatible service, for when not using
Amazon S3. Do not specify this value if you're using Amazon
S3.
</para>
<note><para>This endpoint must support HTTPS and will use
path-based addressing instead of virtual host based
addressing.</para></note>
</listitem>
</varlistentry>
</variablelist>
<example><title>Uploading with non-default credential profile for Amazon S3</title> <example><title>Uploading with a specific credential profile for Amazon S3</title>
<para><command>nix copy --to ssh://machine nixpkgs.hello s3://example-bucket?profile=cache-upload</command></para> <para><command>nix copy --to 's3://example-nix-cache?profile=cache-upload&amp;region=eu-west-2' nixpkgs.hello</command></para>
</example> </example>
<example><title>Uploading to an S3-Compatible Binary Cache</title> <example><title>Uploading to an S3-Compatible Binary Cache</title>
<para><command>nix copy --to ssh://machine nixpkgs.hello s3://example-bucket?profile=cache-upload&amp;endpoint=minio.example.com</command></para> <para><command>nix copy --to 's3://example-nix-cache?profile=cache-upload&amp;endpoint=minio.example.com' nixpkgs.hello</command></para>
</example> </example>
<para>The user writing to the bucket will need to perform the
following actions against the bucket:</para>
<itemizedlist>
<listitem><para><literal>s3:ListBucket</literal></para></listitem>
<listitem><para><literal>s3:GetBucketLocation</literal></para></listitem>
<listitem><para><literal>s3:ListObjects</literal></para></listitem>
<listitem><para><literal>s3:GetObject</literal></para></listitem>
<listitem><para><literal>s3:PutObject</literal></para></listitem>
<listitem><para><literal>s3:ListBucketMultipartUploads</literal></para></listitem>
<listitem><para><literal>s3:CreateMultipartUpload</literal></para></listitem>
<listitem><para><literal>s3:ListMultipartUploadParts</literal></para></listitem>
<listitem><para><literal>s3:AbortMultipartUpload</literal></para></listitem>
</itemizedlist>
</section> </section>
</section> </section>