From 2af79133b91d62389acf34ffbd4353c21cc23dea Mon Sep 17 00:00:00 2001 From: Akihiro Suda Date: Wed, 7 Feb 2024 15:12:43 +0900 Subject: [PATCH] docs/rootless.md: move auxiliary information to the bottom Move "Distribution-specific hint" and "Troubleshooting" to the bottom of the document to improve readability. Signed-off-by: Akihiro Suda --- docs/rootless.md | 165 +++++++++++++++++++++++------------------------ 1 file changed, 82 insertions(+), 83 deletions(-) diff --git a/docs/rootless.md b/docs/rootless.md index 39b77ec6e38e..b2a05883e20e 100644 --- a/docs/rootless.md +++ b/docs/rootless.md @@ -2,58 +2,6 @@ Rootless mode allows running BuildKit daemon as a non-root user. -## Distribution-specific hint -Using Ubuntu kernel is recommended. -### Container-Optimized OS from Google -Make sure to have an `emptyDir` volume below: -```yaml -spec: - containers: - - name: buildkitd - volumeMounts: - # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too, - # but the default VOLUME does not work with rootless on Google's Container-Optimized OS - # as it is mounted with `nosuid,nodev`. - # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038 - - mountPath: /home/user/.local/share/buildkit - name: buildkitd - volumes: - - name: buildkitd - emptyDir: {} -``` - -See also the [example manifests](#Kubernetes). - -### Bottlerocket OS - -Needs to set the max user namespaces to a positive integer, through the [API settings](https://github.com/bottlerocket-os/bottlerocket#kernel-settings): - -```toml -[settings.kernel.sysctl] -"user.max_user_namespaces" = "16384" -``` - -See [`../examples/eksctl/bottlerocket.yaml`](../examples/eksctl/bottlerocket.yaml) for an example to configure a Node Group in EKS. - -
-Old distributions - -

- -### Debian GNU/Linux 10 -Add `kernel.unprivileged_userns_clone=1` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`. -This step is not needed for Debian GNU/Linux 11 and later. - -### RHEL/CentOS 7 -Add `user.max_user_namespaces=28633` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`. -This step is not needed for RHEL/CentOS 8 and later. - -### Fedora, before kernel 5.13 -You may have to disable SELinux, or run BuildKit with `--oci-worker-snapshotter=fuse-overlayfs`. - -

-
- ## Known limitations * Using the `overlayfs` snapshotter requires kernel >= 5.11 or Ubuntu kernel. On kernel >= 4.18, the `fuse-overlayfs` snapshotter is used instead of `overlayfs`. @@ -93,37 +41,6 @@ Then let buildkitd join the same namespace as containerd. $ containerd-rootless-setuptool.sh nsenter -- buildkitd --oci-worker=false --containerd-worker=true --containerd-worker-snapshotter=native ``` -## Troubleshooting - -### Error related to `overlayfs` -Try running `buildkitd` with `--oci-worker-snapshotter=fuse-overlayfs`: - -```console -$ rootlesskit buildkitd --oci-worker-snapshotter=fuse-overlayfs -``` - -### Error related to `fuse-overlayfs` -Try running `buildkitd` with `--oci-worker-snapshotter=native`: - -```console -$ rootlesskit buildkitd --oci-worker-snapshotter=native -``` - -### Error related to `newuidmap` or `/etc/subuid` -See https://rootlesscontaine.rs/getting-started/common/subuid/ - -### Error `Options:[rbind ro]}]: operation not permitted` -Make sure to mount an `emptyDir` volume on `/home/user/.local/share/buildkit` . - -### Error `fork/exec /proc/self/exe: no space left on device` with `level=warning msg="/proc/sys/user/max_user_namespaces needs to be set to non-zero."` -Run `sysctl -w user.max_user_namespaces=N` (N=positive integer, like 63359) on the host nodes. - -See [`../examples/kubernetes/sysctl-userns.privileged.yaml`](../examples/kubernetes/sysctl-userns.privileged.yaml). - -### Error `mount proc:/proc (via /proc/self/fd/6), flags: 0xe: operation not permitted` -This error is known to happen when BuildKit is executed in a container without the `--oci-worker-no-sandbox` flag. -Make sure that `--oci-worker-no-process-sandbox` is specified (See [below](#docker)). - ## Containerized deployment ### Kubernetes @@ -193,3 +110,85 @@ $ make images $ docker run ... moby/buildkit:local-rootless ... ``` +## Troubleshooting + +### Error related to `overlayfs` +Try running `buildkitd` with `--oci-worker-snapshotter=fuse-overlayfs`: + +```console +$ rootlesskit buildkitd --oci-worker-snapshotter=fuse-overlayfs +``` + +### Error related to `fuse-overlayfs` +Try running `buildkitd` with `--oci-worker-snapshotter=native`: + +```console +$ rootlesskit buildkitd --oci-worker-snapshotter=native +``` + +### Error related to `newuidmap` or `/etc/subuid` +See https://rootlesscontaine.rs/getting-started/common/subuid/ + +### Error `Options:[rbind ro]}]: operation not permitted` +Make sure to mount an `emptyDir` volume on `/home/user/.local/share/buildkit` . + +### Error `fork/exec /proc/self/exe: no space left on device` with `level=warning msg="/proc/sys/user/max_user_namespaces needs to be set to non-zero."` +Run `sysctl -w user.max_user_namespaces=N` (N=positive integer, like 63359) on the host nodes. + +See [`../examples/kubernetes/sysctl-userns.privileged.yaml`](../examples/kubernetes/sysctl-userns.privileged.yaml). + +### Error `mount proc:/proc (via /proc/self/fd/6), flags: 0xe: operation not permitted` +This error is known to happen when BuildKit is executed in a container without the `--oci-worker-no-sandbox` flag. +Make sure that `--oci-worker-no-process-sandbox` is specified (See [below](#docker)). + +## Distribution-specific hint +Using Ubuntu kernel is recommended. +### Container-Optimized OS from Google +Make sure to have an `emptyDir` volume below: +```yaml +spec: + containers: + - name: buildkitd + volumeMounts: + # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too, + # but the default VOLUME does not work with rootless on Google's Container-Optimized OS + # as it is mounted with `nosuid,nodev`. + # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038 + - mountPath: /home/user/.local/share/buildkit + name: buildkitd + volumes: + - name: buildkitd + emptyDir: {} +``` + +See also the [example manifests](#Kubernetes). + +### Bottlerocket OS + +Needs to set the max user namespaces to a positive integer, through the [API settings](https://github.com/bottlerocket-os/bottlerocket#kernel-settings): + +```toml +[settings.kernel.sysctl] +"user.max_user_namespaces" = "16384" +``` + +See [`../examples/eksctl/bottlerocket.yaml`](../examples/eksctl/bottlerocket.yaml) for an example to configure a Node Group in EKS. + +
+Old distributions + +

+ +### Debian GNU/Linux 10 +Add `kernel.unprivileged_userns_clone=1` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`. +This step is not needed for Debian GNU/Linux 11 and later. + +### RHEL/CentOS 7 +Add `user.max_user_namespaces=28633` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`. +This step is not needed for RHEL/CentOS 8 and later. + +### Fedora, before kernel 5.13 +You may have to disable SELinux, or run BuildKit with `--oci-worker-snapshotter=fuse-overlayfs`. + +

+