podman image build does not hash bind mounts correctly

[elbehery95]

I recently noticed a behavior different than docker when podman builds an image.

Scenario

mkdir my_src && echo "hello" > my_src/test.txt
echo "
FROM alpine:latest
RUN --mount=type=bind,src=./my_src,dst=/my_src cat /my_src/test.txt
RUN echo last layer
" > Containerfile

podman image build .

# now try-rebuilding again, expecting all layers to be re-used from cache
podman image build .

In the above scenario it is expected that the second image build command would be re-using the previous build layer, and output would be like the following

STEP 1/3: FROM alpine:latest
STEP 2/3: RUN --mount=type=bind,src=./my_src,dst=/my_src cat /my_src/test.txt
--> Using cache 1b838c0f375b0c30928cb1f5ee0928605ac1f144919d2b3a2f324ab76c46caad
--> 1b838c0f375b
STEP 3/3: RUN echo last layer
--> Using cache c8d55ac6afeb0185e520c53254ad79542e68134881d32884bfadefe818445f36
--> c8d55ac6afeb
c8d55ac6afeb0185e520c53254ad79542e68134881d32884bfadefe818445f36

The strange thing happen if any of the content of the bind mount are changed for example

echo "world" >> my_src/test.txt
podman image build .

The above command should invalidate the second image layer, in podman it does not do this. Basically podman will re-use a stale layer in this case. And output will be as the following

STEP 1/3: FROM alpine:latest
STEP 2/3: RUN --mount=type=bind,src=./my_src,dst=/my_src cat /my_src/test.txt
--> Using cache 1b838c0f375b0c30928cb1f5ee0928605ac1f144919d2b3a2f324ab76c46caad
--> 1b838c0f375b
STEP 3/3: RUN echo last layer
--> Using cache c8d55ac6afeb0185e520c53254ad79542e68134881d32884bfadefe818445f36
--> c8d55ac6afeb
c8d55ac6afeb0185e520c53254ad79542e68134881d32884bfadefe818445f36

In docker this behavior works as expected and layer would be invalid in this case given that the content of one of the files inside the bind mount source are changed.

My question is, how does one use bind mounts with podman while at the same time ensures it handles layer caching as expected?

Thank you

Originally posted by @elbehery95 in #21859

[baude]

@nalind @TomSweeneyRedHat can either of you comment on this ?

[elbehery95]

Closing this as we switched to using docker instead of podman which in this case 2X faster.

I think podman is not a good solution when context size is big, as it forces user to use COPY instead of RUN --mount in order to avoid faulty cache for the bind mount layer.

Attaching hyperfine output for any future reference

  docker image build . -f bind.Dockerfile ran
    1.77 ± 0.18 times faster than docker image build . -f copy.Dockerfile
    2.08 ± 0.17 times faster than podman image build . -f copy.Dockerfile

[djhoese]

It is unfortunate that this issue got closed as I'm running into a similar situation, but the opposite and am having trouble tracking down existing bug reports on it. My issue is that no matter what I do the RUN with the bind mount doesn't seem to be cached and is always rerun. It looks like this:

RUN --mount=type=bind,source=my.tar.gz,target=/tmp/my.tar.gz,from=extra_context,z \
    tar --no-same-owner --exclude some/big/dir -xf /tmp/my.tar.gz && \
    chmod -R a+rX extracted_root/

No matter what I do this layer is not cached, but my understanding is that it should be if the tarball didn't change. I can create a separate issue if needed, but this particular issue seems related even if it is having the opposite result. Otherwise if someone can point me in the right direction, let me know.

[elbehery95]

@djhoese we had to switch to using buildkit. Consider switching to it (either docker buildx plugin or buildctl inside podman)

Basically this allowed us to obtain all the buildkit features when building images that we later use podman to run