processes can remain active after build finishes with --oci-worker-no-process-sandbox

[jampy]

I'm not really sure if this is a bug or by-design. If it is expected, I could not find any hints about it online.

I found a situation where a Dockerfile can start processes on the buildkitd host that keep running even after the build job terminates. This only happens when running Buildkit in rootless mode.

In essence, this allows anyone that can start container builds to install a permanent daemon on the buildkitd host, although probably/hopefully restricted.

Testcase

The following testcase triggers the problem consistently and easily on Kubernetes k3s v1.32.1, but I don't think the problem is specific to Kubernetes.

Start a Kubernetes pod with Buildkitd in rootless mode (based on the official Buildkit rootless deployment example):

cat <<EOF | kubectl apply -f -

apiVersion: v1                                           
kind: Pod                                                
metadata:                                                
  name: buildkitd-rootless-test                         
spec:                                                    

  containers:
    - name: main
      image: moby/buildkit:v0.19.0-rootless
      args:
        - --addr
        - unix:///run/user/1000/buildkit/buildkitd.sock
        - --oci-worker-no-process-sandbox
      securityContext:
        appArmorProfile:
          type: Unconfined
        seccompProfile:
          type: Unconfined
        runAsUser: 1000
        runAsGroup: 1000
      volumeMounts:
        - mountPath: /home/user/.local/share/buildkit
          name: userdir
  volumes:
    - name: userdir
      emptyDir: {}

EOF

Then exec into the container:

That's just to make the testcase simpler. The same end result can be observed when binding buildkitd to a TCP socket and run buildctl on a different host that can contact said TCP socket.

kubectl exec -it buildkitd-rootless-test -- /bin/sh

Within the container prepare the problematic Dockerfile:

cd /tmp

cat >Dockerfile <<EOF

FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get -y install libncurses5 libicu66 libtommath-dev wget
RUN ln -s /usr/lib/x86_64-linux-gnu/libtommath.so.1 /usr/lib/x86_64-linux-gnu/libtommath.so.0
RUN cd /opt/ \
 && wget -O - https://github.com/FirebirdSQL/firebird/releases/download/v3.0.11/Firebird-3.0.11.33703-0.amd64.tar.gz | tar xz \
 && cd Firebird* \
 && ./install.sh -silent

EOF

Build that Dockerfile:

  buildctl \
    --addr unix:///run/user/1000/buildkit/buildkitd.sock \
    build \
    --frontend dockerfile.v0 \
    --local "context=$PWD" \
    --local "dockerfile=$PWD"

The build should complete with no errors.

Now, still in the buildkitd-rootless-test pod, when running ps auxw shows two firebird processes left from the build, which should not be there. At least that's the case for me and I would except that this should not be possible for a non-malicious build:

$  ps auxw
PID   USER     TIME  COMMAND
    1 user      0:00 rootlesskit buildkitd --addr unix:///run/user/1000/buildkit/buildkitd.sock --oci-worker-no-process-sandbox
   13 user      0:00 /proc/self/exe buildkitd --addr unix:///run/user/1000/buildkit/buildkitd.sock --oci-worker-no-process-sandbox
   32 user      0:01 buildkitd --addr unix:///run/user/1000/buildkit/buildkitd.sock --oci-worker-no-process-sandbox
   74 user      0:00 /bin/sh
 5526 100083    0:00 /opt/firebird/bin/fbguard -pidfile /var/run/firebird/firebird.pid -daemon -forever
 5527 100083    0:00 /opt/firebird/bin/firebird
 5543 user      0:00 ps auxw

(here PIDs 5526 and 5527)

Notes

I guess this has something to do with --oci-worker-no-process-sandbox and the Buildkit docs mention a warning:

Note that --oci-worker-no-process-sandbox allows build executor containers to kill (and potentially ptrace depending on the seccomp configuration) an arbitrary process in the BuildKit daemon container.

Anyway, it does not mention that processes can also survive the build process and live on the buildkitd host. I don't expect build containers to be unbreakable, but at least provide a isolation good enough to prevent accidental process leftovers.

If that's "by-design" or expected, I wonder if there is a way to prevent it.

[AkihiroSuda]

This is by design, as it does not create a PID namespace.

On Docker, a workaround is to remove --oci-worker-no-process-sandbox (buildkitd flag) and set --security-opt systempaths=unconfined (docker run flag) instead.
This is not really different from --privileged though, but the processes still work as an unprivileged user.

However, it is still hard to set its equivalent (procMount: Unmasked) on Kubernetes.

Will open a PR to update the example manifests and the docs.

[AkihiroSuda]

• rootless: update docs and examples #5765

[jampy]

Thanks a lot for your response and clarification.

I tried procMount: Unmasked but it needs two feature gates to be enabled and and did still not work in my cluster.

So, I guess on Kubernetes the way to gon is using --privileged