rootless: support Google Container-Optimized OS

Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
but the default VOLUME does not work with rootless on Google's Container-Optimized OS
as it is mounted with `nosuid,nodev`.

So the volume has to be explicitly mounted as an `emptyDir` volume.

Tested with GKE Autopilot 1.24.3-gke.200 (kernel 5.10.123+, containerd 1.6.6).

Fix issue 879

Thanks to Andrew Grigorev (ei-grad) and Ben Cressey (bcressey).

Signed-off-by: Akihiro Suda <[email protected]>

docs/rootless.md

@@ -4,6 +4,30 @@ Rootless mode allows running BuildKit daemon as a non-root user.
 
 ## Distribution-specific hint
 Using Ubuntu kernel is recommended.
+### Container-Optimized OS from Google
+Make sure to have an `emptyDir` volume below:
+```yaml
+spec:
+  containers:
+    - name: buildkitd
+      volumeMounts:
+        # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+        # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+        # as it is mounted with `nosuid,nodev`.
+        # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+        - mountPath: /home/user/.local/share/buildkit
+          name: buildkitd
+  volumes:
+    - name: buildkitd
+      emptyDir: {}
+```
+
+See also the [example manifests](#Kubernetes).
+
+<details>
+<summary>Old distributions</summary>
+
+<p>
 
 ### Debian GNU/Linux 10
 Add `kernel.unprivileged_userns_clone=1` to `/etc/sysctl.conf` (or `/etc/sysctl.d`) and run `sudo sysctl -p`.
@@ -16,8 +40,8 @@ This step is not needed for RHEL/CentOS 8 and later.
 ### Fedora, before kernel 5.13
 You may have to disable SELinux, or run BuildKit with `--oci-worker-snapshotter=fuse-overlayfs`.
 
-### Container-Optimized OS from Google
-:warning: Currently unsupported. See [#879](https://github.com/moby/buildkit/issues/879).
+</p>
+</details>
 
 ## Known limitations
 * Using the `overlayfs` snapshotter requires kernel >= 5.11 or Ubuntu kernel.
@@ -77,6 +101,9 @@ $ rootlesskit buildkitd --oci-worker-snapshotter=native
 ### Error related to `newuidmap` or `/etc/subuid`
 See https://rootlesscontaine.rs/getting-started/common/subuid/
 
+### Error `Options:[rbind ro]}]: operation not permitted`
+Make sure to mount an `emptyDir` volume on `/home/user/.local/share/buildkit` .
+
 ## Containerized deployment
 
 ### Kubernetes

examples/kubernetes/README.md

@@ -7,8 +7,7 @@ This directory contains Kubernetes manifests for `Pod`, `Deployment` (with `Serv
 * `Job`: good if you don't want to have daemon pods
 
 Using Rootless mode (`*.rootless.yaml`) is recommended because Rootless mode image is executed as non-root user (UID 1000) and doesn't need `securityContext.privileged`.
-
-:warning: Rootless mode may not work on some host kernels. See [`../../docs/rootless.md`](../../docs/rootless.md).
+See [`../../docs/rootless.md`](../../docs/rootless.md).
 
 See also ["Building Images Efficiently And Securely On Kubernetes With BuildKit" (KubeCon EU 2019)](https://kccnceu19.sched.com/event/MPX5).
 

examples/kubernetes/deployment+service.rootless.yaml

@@ -63,11 +63,19 @@ spec:
             - name: certs
               readOnly: true
               mountPath: /certs
+            # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+            # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+            # as it is mounted with `nosuid,nodev`.
+            # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+            - mountPath: /home/user/.local/share/buildkit
+              name: buildkitd
       volumes:
         # buildkit-daemon-certs must contain ca.pem, cert.pem, and key.pem
         - name: certs
           secret:
             secretName: buildkit-daemon-certs
+        - name: buildkitd
+          emptyDir: {}
 ---
 apiVersion: v1
 kind: Service

examples/kubernetes/job.rootless.yaml

@@ -52,8 +52,16 @@ spec:
             - name: workspace
               readOnly: true
               mountPath: /workspace
+            # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+            # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+            # as it is mounted with `nosuid,nodev`.
+            # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+            - mountPath: /home/user/.local/share/buildkit
+              name: buildkitd
       # To push the image, you also need to create `~/.docker/config.json` secret
       # and set $DOCKER_CONFIG to `/path/to/.docker` directory.
       volumes:
         - name: workspace
           emptyDir: {}
+        - name: buildkitd
+          emptyDir: {}

examples/kubernetes/pod.rootless.yaml

@@ -34,3 +34,13 @@ spec:
         # To change UID/GID, you need to rebuild the image
         runAsUser: 1000
         runAsGroup: 1000
+      volumeMounts:
+        # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+        # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+        # as it is mounted with `nosuid,nodev`.
+        # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+        - mountPath: /home/user/.local/share/buildkit
+          name: buildkitd
+  volumes:
+    - name: buildkitd
+      emptyDir: {}

examples/kubernetes/statefulset.rootless.yaml

@@ -47,3 +47,13 @@ spec:
             # To change UID/GID, you need to rebuild the image
             runAsUser: 1000
             runAsGroup: 1000
+          volumeMounts:
+            # Dockerfile has `VOLUME /home/user/.local/share/buildkit` by default too,
+            # but the default VOLUME does not work with rootless on Google's Container-Optimized OS
+            # as it is mounted with `nosuid,nodev`.
+            # https://github.com/moby/buildkit/issues/879#issuecomment-1240347038
+            - mountPath: /home/user/.local/share/buildkit
+              name: buildkitd
+      volumes:
+        - name: buildkitd
+          emptyDir: {}