-
Notifications
You must be signed in to change notification settings - Fork 512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multiarch/qemu-user-static not working on 1.23 #4215
multiarch/qemu-user-static not working on 1.23 #4215
Comments
I tried using a bootstrap container - it wasn't any different to executing it from a normal container. https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.1.107
I guess because this is executed in a container typically - with this new kernel once the container exits , the binfmt entries are unmounted and thus the arm64 commands I'm trying to run aren't recognized. I guess if it was executed in some sort of non containerized bootstrap script, the entries would persist. Need to figure out if that would work. |
Nice work tracking this down!
This seems like a pretty clear regression, and one that breaks a couple of the popular "fire and forget" solutions for deploying binfmt support: I'll see about reporting this to LKML.
I'll also try patching host-ctr to pass in the host's |
This doesn't work, since it runs afoul of the checkProcMount safety check:
|
hi @bcressey , thanks for looking at this. I don't suppose there is a way to add a "non containerized" init script to a bottlerocket node without building your own image? I had a look but couldn't find anything. It makes sense there isn't another way to do it. |
@gbucknel that's correct - containers are the only way to run custom code on Bottlerocket. While poking at this, I noticed that the |
@gbucknel I was able to get the following bootstrap container working. The two challenges involved were that the host didn't have its own
|
@bcressey !!! That's so cool, I'll try it out, thank you! |
Happy to help! If it's easier to integrate, I expect it'd be possible to make this work in a k8s pod also, with a spec like this:
|
@bcressey , thanks again, I've tested your container and it works well. Appreciate the pod yaml as well since I'm not sure if a bootstrap container is the right approach - I had quite a few nodes silently fail while playing with this today. Was just wondering, if the binfmt feature was enabled in systemd for bottlerocket, do you think this container wouldn't be required? Would that be a better solution? Perhaps not because it isn't as secure? I was testing with AL2023 yesterday just to try to repro the issue and was surprised to find that it worked fine on the newer kernel and the multiarch container. Maybe that's because systemd is handling the state of the proc filesystem ? |
@gbucknel I'm planning to enable that on the Bottlerocket side. Not sure whether this would make it work in your environment or not, but it's worth a try. It sounds like it might help if AL2023 is working. |
@bcressey oh that's awesome, let me know if I can test anything. |
hi @bcressey , I was wondering if the change to systemd was targeted for 1.25? It doesn't seem like there is a change to the file you linked to ? |
@gbucknel It won't make the release train for 1.25; I need to finish writing a test for the SELinux policy changes, and also verify they are actually required. Previously the host did not mount binfmt_misc at all, so writing to it required For your binfmt misc installer - can you confirm you are running that inside a privileged pod? The SELinux rule I've tested would effectively require that. |
No problem, just thought I'd check. At the moment, I run
So yes, I am in a privileged pod . Thanks! |
This is a pretty strange one, but I wanted to raise it in case someone else was hitting it.
In a CICD context, we build multiarch images on x64 by executing the
multiarch/qemu-user-static
image before trying to build arm64 and amd64 docker images.We use this command to do this :
As far as I know this registers alternative executable types in the kernel, so QEMU will know when to step in to emulate the commands. This has been working for quite a while, but when 1.23 was pushed out, this process broke.
Expected results :
Image to build correctly
Actual results:
Image build fails with exec format error
We were able to get everything working again by going back to 1.22.
Also , running the register script by hand in a sheltie session on the host seems to make arm64 builds work in 1.23 again, so trying to do the same thing in a bootstrap container is something I'd like to try.
Looking at the changelog, I'm a bit puzzled as to where the change could've been. Will continue to try to figure it out. If anyone else has seen this, it would be really interesting.
The text was updated successfully, but these errors were encountered: