Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Buildkit 0.20.0 may add a "variant" field, which alters the image digest #5774

Open
4 tasks done
apyrgio opened this issue Feb 24, 2025 · 1 comment · May be fixed by #5776
Open
4 tasks done

Buildkit 0.20.0 may add a "variant" field, which alters the image digest #5774

apyrgio opened this issue Feb 24, 2025 · 1 comment · May be fixed by #5776
Milestone
v0.20.1

Comments

@apyrgio
Copy link

apyrgio commented Feb 24, 2025 •
edited
Loading

Contributing guidelines and issue reporting guide

Well-formed report checklist

  • I have found a bug that the documentation does not mention anything about my problem
  • I have found a bug that there are no open or closed issues that are related to my problem
  • I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

Images built with Buildkit 0.20.0 may differ from the ones built with Buildkit 0.19.0, because a new "variant" field may be added in the config blob of the image. Personally, I've seen this happen only when building ARM images.

Here's a side by side diff of two config blobs. Left is Buildkit v0.19.0, and right is Buildkit v0.20.0:

diff -y config.b19 config.b20
{                                                               {
  "architecture": "arm64",                                        "architecture": "arm64",
  "config": {                                                     "config": {
    "Env": [                                                        "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin         "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
      "DEBIAN_FRONTEND=noninteractive"                                "DEBIAN_FRONTEND=noninteractive"
    ],                                                              ],
    "Cmd": [                                                        "Cmd": [
      "bash"                                                          "bash"
    ]                                                               ]
  },                                                              },
  "created": "2023-02-28T21:21:00Z",                              "created": "2023-02-28T21:21:00Z",
  "history": [                                                    "history": [
    {                                                               {
      "created": "2023-09-07T00:39:39.206776845Z",                    "created": "2023-09-07T00:39:39.206776845Z",
      "created_by": "/bin/sh -c #(nop) ADD file:fb5c8f411c4a1         "created_by": "/bin/sh -c #(nop) ADD file:fb5c8f411c4a1
    },                                                              },
    {                                                               {
      "created": "2023-09-07T00:39:39.650056081Z",                    "created": "2023-09-07T00:39:39.650056081Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",              "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
      "empty_layer": true                                             "empty_layer": true
    },                                                              },
    {                                                               {
      "created": "2023-02-28T21:21:00Z",                              "created": "2023-02-28T21:21:00Z",
      "created_by": "ENV DEBIAN_FRONTEND=noninteractive",             "created_by": "ENV DEBIAN_FRONTEND=noninteractive",
      "comment": "buildkit.dockerfile.v0",                            "comment": "buildkit.dockerfile.v0",
      "empty_layer": true                                             "empty_layer": true
    },                                                              },
    {                                                               {
      "created": "2023-02-28T21:21:00Z",                              "created": "2023-02-28T21:21:00Z",
      "created_by": "RUN /bin/sh -c repro-sources-list.sh &&          "created_by": "RUN /bin/sh -c repro-sources-list.sh && 
      "comment": "buildkit.dockerfile.v0"                             "comment": "buildkit.dockerfile.v0"
    }                                                               }
  ],                                                              ],
  "os": "linux",                                                  "os": "linux",
  "rootfs": {                                                     "rootfs": {
    "type": "layers",                                               "type": "layers",
    "diff_ids": [                                                   "diff_ids": [
      "sha256:341de903723838835820feffc6ff0ab04ea7ffddd3886cd         "sha256:341de903723838835820feffc6ff0ab04ea7ffddd3886cd
      "sha256:cdc84d7fdc4c77e4965366c040d1e0aae36b6adba12d7e2         "sha256:cdc84d7fdc4c77e4965366c040d1e0aae36b6adba12d7e2
    ]                                                               ]
  }                                                           |   },
                                                              >   "variant": "v8"
}                                                               }

Purpose of this bug report:

  • Let devs know about this change, in case it's not an intended one
  • Inform people that they can't reproduce bit-for-bit the same container image with the newest Buildkit version
    • I haven't tried Buildkit v0.18.0, so I don't know how stable image reproducibility is across Buildkit versions

Reproduction

Dockerfile: https://github.com/reproducible-containers/repro-sources-list.sh/blob/master/Dockerfile.debian-12

Command to build image with Buildkit 0.19.0:

$ docker buildx create \
    --name repro-build-0cdfa98a55df0f2f2a59e0f473a648f209a9d0fe83c04baa91a19931022a8d27 \
    --driver-opt image=docker.io/moby/buildkit:v0.19.0@sha256:14aa1b4dd92ea0a4cd03a54d0c6079046ea98cd0c0ae6176bdd7036ba370cbbe
$ docker buildx \
    --builder repro-build-0cdfa98a55df0f2f2a59e0f473a648f209a9d0fe83c04baa91a19931022a8d27 \
    build \
    --build-arg SOURCE_DATE_EPOCH=1677619260 \
    --provenance false \
    --output type=docker,dest=image.tar,rewrite-timestamp=true \
    -f Dockerfile.debian-12 \
    --platform linux/arm64
    .

Command to build image with Buildkit 0.20.0:

$ docker buildx create \
    --name repro-build-70d1bcdd371beedd0103a50e0d51a41627ea0ca45509fb6543675d8197a5b6bc \
    --driver-opt image=docker.io/moby/buildkit:v0.20.0
$ docker buildx \
    --builder repro-build-70d1bcdd371beedd0103a50e0d51a41627ea0ca45509fb6543675d8197a5b6bc \
    build \
    --build-arg SOURCE_DATE_EPOCH=1677619260 \
    --provenance false \
    --output type=docker,dest=image.tar,rewrite-timestamp=true \
    -f Dockerfile.debian-12 \
    --platform linux/arm64
    .

If you untar the image.tar tarballs, and check the config blog, you'll see that the "variant": "v8" field is present only in the tarball produced by Buildkit v0.20.0.

Version information

I have reproduced this both locally and in GitHub actions (docker/setup-buildx-action@v3). Here's the version information of the latter:

Docker info:

/usr/bin/docker version
  Client: Docker Engine - Community
   Version:           26.1.3
   API version:       1.45
   Go version:        go1.21.10
   Git commit:        b72abbb
   Built:             Thu May 16 08:40:13 2024
   OS/Arch:           linux/arm64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          26.1.3
    API version:      1.45 (minimum version 1.24)
    Go version:       go1.21.10
    Git commit:       8e96db1
    Built:            Thu May 16 08:40:13 2024
    OS/Arch:          linux/arm64
    Experimental:     false
   containerd:
    Version:          1.7.25
    GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
   runc:
    Version:          1.2.4
    GitCommit:        v1.2.4-0-g6c52b3f
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/bin/docker info
  Client: Docker Engine - Community
   Version:    26.1.3
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.20.0
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.27.1
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 0
   Server Version: 26.1.3
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: systemd
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
   runc version: v1.2.4-0-g6c52b3f
   init version: de40ad0
   Security Options:
    apparmor
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.8.0-1020-azure
   Operating System: Ubuntu 24.04.1 LTS
   OSType: linux
   Architecture: aarch64
   CPUs: 4
   Total Memory: 15.58GiB
   Name: fv-az2130-671
   ID: 91b18afc-55ea-45ee-9a5e-ab3c5c3e87d5
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false
@tonistiigi tonistiigi added this to the v0.20.1 milestone Feb 24, 2025
@tonistiigi tonistiigi linked a pull request Feb 25, 2025 that will close this issue
Open
@tonistiigi
Copy link
Member

Fix in #5776 . This is defined in Dockerfile frontend, so if you want the previous behavior, you can use #syntax=docker/dockerfile:1.13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.20.1
Development

Successfully merging a pull request may close this issue.

dockerfile: normalize platform in image config tonistiigi/buildkit
2 participants
@tonistiigi @apyrgio