buildah --creds only checks permissions at GET request then assumes it has POST too

[makhomed]

Description

I have my own registry, read only access (via GET and HEAD http request methods) allowed for all without password, but write access (via POST http request method) allowed only with login and password provided.

private registry write access protection created using nginx limit_except directive.

nginx configuration for registry

server {

    listen [::]:443 ssl;
    listen 443 ssl;

    server_name registry.ideil.dev;

    ssl_certificate     /etc/letsencrypt/live/ideil.dev/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ideil.dev/privkey.pem;

    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;

    client_max_body_size 0;
    chunked_transfer_encoding on;

    location / {

        # https://distribution.github.io/distribution/about/

        # https://distribution.github.io/distribution/spec/api/

        limit_except GET {
            auth_basic "Registry realm";
            auth_basic_user_file /etc/distribution/htpasswd;
        }

        add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

        proxy_pass http://127.0.0.1:5000;
        proxy_set_header  Host              $host;
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
    }
}

but buildah can't work correctly with such registry and buildah fail with 125 error code.

Steps to reproduce the issue:

echo
echo push new image to registry
registry=registry.ideil.dev
username=$(cat /root/.config/podman/$registry/username)
password=$(cat /root/.config/podman/$registry/password)
echo $password | buildah login --username $username --password-stdin https://$registry
new_image=$(buildah images --quiet --no-trunc localhost/php70-php-fpm)
buildah push --log-level=debug --rm $new_image docker://$registry/php70-php-fpm
echo buildah push return code: $?

Describe the results you received:

buildah push --log-level=debug

push new image to registry
Login Succeeded!
time="2025-01-27T22:15:32Z" level=debug msg="effective capabilities: [audit_control=true audit_read=true audit_write=true block_suspend=true bpf=true checkpoint_restore=true chown=true dac_override=true dac_read_search=true fowner=true fsetid=true ipc_lock=true ipc_owner=true kill=true lease=true linux_immutable=true mac_admin=true mac_override=true mknod=true net_admin=true net_bind_service=true net_broadcast=true net_raw=true perfmon=true setfcap=true setgid=true setpcap=true setuid=true sys_admin=true sys_boot=true sys_chroot=true sys_module=true sys_nice=true sys_pacct=true sys_ptrace=true sys_rawio=true sys_resource=true sys_time=true sys_tty_config=true syslog=true wake_alarm=true]"
time="2025-01-27T22:15:32Z" level=debug msg="[graphdriver] trying provided driver \"overlay\""
time="2025-01-27T22:15:32Z" level=debug msg="Cached value indicated that overlay is supported"
time="2025-01-27T22:15:32Z" level=debug msg="Cached value indicated that overlay is supported"
time="2025-01-27T22:15:32Z" level=debug msg="Cached value indicated that metacopy is being used"
time="2025-01-27T22:15:32Z" level=debug msg="Cached value indicated that native-diff is not being used"
time="2025-01-27T22:15:32Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
time="2025-01-27T22:15:32Z" level=debug msg="backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true"
time="2025-01-27T22:15:32Z" level=debug msg="Looking up image \"sha256:e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" in local containers storage"
time="2025-01-27T22:15:32Z" level=debug msg="Trying \"e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" ..."
time="2025-01-27T22:15:32Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\""
time="2025-01-27T22:15:32Z" level=debug msg="Found image \"sha256:e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" as \"e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" in local containers storage"
time="2025-01-27T22:15:32Z" level=debug msg="Found image \"sha256:e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" as \"e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0)"
time="2025-01-27T22:15:32Z" level=debug msg="Pushing image sha256:e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0 to docker://registry.ideil.dev/php70-php-fpm:latest"
time="2025-01-27T22:15:32Z" level=debug msg="Normalized platform linux/amd64 to {amd64 linux  [] }"
time="2025-01-27T22:15:32Z" level=debug msg="Copying source image [overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0 to destination image //registry.ideil.dev/php70-php-fpm:latest"
time="2025-01-27T22:15:32Z" level=debug msg="Using registries.d directory /etc/containers/registries.d"
time="2025-01-27T22:15:32Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf\""
time="2025-01-27T22:15:32Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/000-shortnames.conf\""
time="2025-01-27T22:15:32Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/001-rhel-shortnames.conf\""
time="2025-01-27T22:15:32Z" level=debug msg="Loading registries configuration \"/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf\""
time="2025-01-27T22:15:32Z" level=debug msg="Found credentials for registry.ideil.dev/php70-php-fpm in credential helper containers-auth.json in file /run/user/0/containers/auth.json"
time="2025-01-27T22:15:32Z" level=debug msg=" No signature storage configuration found for registry.ideil.dev/php70-php-fpm:latest, using built-in default file:///var/lib/containers/sigstore"
time="2025-01-27T22:15:32Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.ideil.dev"
time="2025-01-27T22:15:32Z" level=debug msg="Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite"
time="2025-01-27T22:15:32Z" level=debug msg="IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]@e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0"
time="2025-01-27T22:15:32Z" level=debug msg=" Using transport \"containers-storage\" policy section \"\""
time="2025-01-27T22:15:32Z" level=debug msg=" Requirement 0: allowed"
time="2025-01-27T22:15:32Z" level=debug msg="Overall: allowed"
Getting image source signatures
time="2025-01-27T22:15:32Z" level=debug msg="Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]"
time="2025-01-27T22:15:32Z" level=debug msg="... will first try using the original manifest unmodified"
Copying blob sha256:ccebfeac7bd72c41d5041092c371b90761dc402e75096ccd3c22dc46aee96957
time="2025-01-27T22:15:32Z" level=debug msg="Checking if we can reuse blob sha256:ccebfeac7bd72c41d5041092c371b90761dc402e75096ccd3c22dc46aee96957: general substitution = true, compression for MIME type \"application/vnd.oci.image.layer.v1.tar\" = true"
time="2025-01-27T22:15:32Z" level=debug msg="Checking /v2/php70-php-fpm/blobs/sha256:ccebfeac7bd72c41d5041092c371b90761dc402e75096ccd3c22dc46aee96957"
time="2025-01-27T22:15:32Z" level=debug msg="GET https://registry.ideil.dev/v2/"
time="2025-01-27T22:15:32Z" level=debug msg="Ping https://registry.ideil.dev/v2/ status 200"
time="2025-01-27T22:15:32Z" level=debug msg="HEAD https://registry.ideil.dev/v2/php70-php-fpm/blobs/sha256:ccebfeac7bd72c41d5041092c371b90761dc402e75096ccd3c22dc46aee96957"
time="2025-01-27T22:15:32Z" level=debug msg="... not present"
time="2025-01-27T22:15:32Z" level=debug msg="Trying to reuse blob with cached digest sha256:ccebfeac7bd72c41d5041092c371b90761dc402e75096ccd3c22dc46aee96957 in destination repo with no location match, checking current repo"
time="2025-01-27T22:15:32Z" level=debug msg="... Already tried the primary destination"
time="2025-01-27T22:15:32Z" level=debug msg="exporting filesystem layer \"ccebfeac7bd72c41d5041092c371b90761dc402e75096ccd3c22dc46aee96957\" without compression for blob \"sha256:ccebfeac7bd72c41d5041092c371b90761dc402e75096ccd3c22dc46aee96957\""
time="2025-01-27T22:15:35Z" level=debug msg="No compression detected"
time="2025-01-27T22:15:35Z" level=debug msg="Compressing blob on the fly"
time="2025-01-27T22:15:35Z" level=debug msg="Uploading /v2/php70-php-fpm/blobs/uploads/"
time="2025-01-27T22:15:35Z" level=debug msg="POST https://registry.ideil.dev/v2/php70-php-fpm/blobs/uploads/"
time="2025-01-27T22:15:35Z" level=debug msg="Error initiating layer upload, response http.Response{Status:\"401 Unauthorized\", StatusCode:401, Proto:\"HTTP/1.1\", ProtoMajor:1, ProtoMinor:1, Header:http.Header{\"Connection\":[]string{\"keep-alive\"}, \"Content-Length\":[]string{\"172\"}, \"Content-Type\":[]string{\"text/html\"}, \"Date\":[]string{\"Mon, 27 Jan 2025 22:15:35 GMT\"}, \"Docker-Distribution-Api-Version\":[]string{\"registry/2.0\"}, \"Server\":[]string{\"nginx\"}, \"Www-Authenticate\":[]string{\"Basic realm=\\\"Registry realm\\\"\"}}, Body:(*http.bodyEOFSignal)(0xc00052c0c0), ContentLength:172, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000766120), TLS:(*tls.ConnectionState)(0xc0002711e0)}"
time="2025-01-27T22:15:35Z" level=debug msg="Looking up image \"sha256:e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" in local containers storage"
time="2025-01-27T22:15:35Z" level=debug msg="Trying \"e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" ..."
time="2025-01-27T22:15:35Z" level=debug msg="parsed reference into \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\""
time="2025-01-27T22:15:35Z" level=debug msg="Found image \"sha256:e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" as \"e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0\" in local containers storage"
Error: pushing image "sha256:e870f8a997556836335f46167c868c85c38aaa5d52bf8d61a12ddaea20bf2ae0" to "docker://registry.ideil.dev/php70-php-fpm": writing blob: initiating layer upload to /v2/php70-php-fpm/blobs/uploads/ in registry.ideil.dev: authentication required
time="2025-01-27T22:15:35Z" level=debug msg="shutting down the store"
time="2025-01-27T22:15:35Z" level=info msg="failed to shutdown storage: \"a layer is mounted: layer is in use by a container\""
buildah push return code: 125

As you can see, Login Succeeded! returned for buildah login command, but buildah push failed with 125 error code.

Output of rpm -q buildah or apt list buildah:

# rpm -q buildah
buildah-1.37.5-1.el9_5.x86_64

Output of buildah version:

# buildah version
Version:         1.37.5
Go Version:      go1.22.7 (Red Hat 1.22.7-2.el9_5)
Image Spec:      1.1.0
Runtime Spec:    1.2.0
CNI Spec:        1.1.0
libcni Version:  v1.2.3
image Version:   5.32.2
Git Commit:
Built:           Tue Nov 12 12:10:34 2024
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Additional info:

the same bug was reported in the 2018 year: #1227

but this buildah bug still not fixed yet.

does any workarounds exists for this bug?

it it possible to fix this bug in the buildah?