[RFC] New field to do hidden/seamless transition between fuseoverlay and native overlay on first run.

[flouthoc]

There is a use case where a central config is shipped for rootless users where mount_program is hardcoded to fuse-overlayfs the reason is that it works for both users with kernel lower than 5.13 and above than 5.13. However there is no need for newer users to keep using fuse-overlayfs but hard-coding config to not use fuse-overlayfs break old users. This creates a problem in creating more kernel agnostic storage config for rootless users. Example issues seen in past: containers/buildah#4715, containers/buildah#4669, containers/buildah#4727

Since there is no way for buildah image to determine on which kernel the buildah will be executed, could we add field ForceUseNativeOverlay bool to c/storage config. When configured true c/storage will on runtime decide if a newer kernel is present and in such case it will ignore configured fuse-overlayfs as mount_Program and switch to native overlay.

Addition of following field will ensure easier way to ship config for rootless users in pre-built podman and buildah images.

[flouthoc]

@containers/storage-maintainers @rhatdan @giuseppe @nalind WDYT ?

[vrothberg]

Some mechanism to default to using native overlayfs when possible sounds good to me 👍

[giuseppe]

why should the config file hardcode that value? That seems incorrect.

For users that were already using fuse-overlayfsd we already store that in the .has-mount-program. Once that file is created, the user will keep using fuse-overlayfs no matter what kernel it is currently running on. The file is deleted only on a complete storage wipeout, on a fresh state we try again to use overlay if possible

[flouthoc]

No its not for the users who are already using fuse-overlayfs but for the users who are trying podman or buildah for the first time and their config contains mount_program = fuse_overlayfs but they are okay to discard it and to use native overlay instead.

The reason they came up with this config is because buildah image was shipped with this config so it works for both new and old users. But since config contains fuse-overlayfs so new users are stuck on this. https://github.com/containers/buildah/blob/main/contrib/buildahimage/Containerfile

[giuseppe]

it seems wrong to have fuse-overlayfs in the configuration if they don't want to use it.

IMO we don't need a new option to say ignore the previous option set when they can just unset it.

[flouthoc]

Buildah once forced native overlay for rootless users in default image and it broke many users who are using older kernel in CI, I think this hidden option can solve the issue which was seen here: containers/buildah#4715 keeping both new/old kernel users happy. As workaround solution for above problem config was reverted to use fuse-overlayfs for rootless setups.

There were more user who got affected because of it some reports are scattered in different issues: containers/buildah#4727

[giuseppe]

Another option to "force native" and do not attempt any fallback to fuse-overlayfs is to uninstall fuse-overlayfs

Before we add a new flag let's make sure we understand why it failed. What is wrong with the mechanism where we write down if a user is using fuse-overlayfs and then keep using it once it is set but attempt using native overlay where possible?

The new flag will still require a change in the configuration, so couldn't it be changed just to unset mount_program?

[flouthoc]

Another option to "force native" and do not attempt any fallback to fuse-overlayfs is to uninstall fuse-overlayfs

@giuseppe Do you mean user should manually uninstall fuse-overlayfs on newer kernels, by default quay.io/buildah/stable:latest ships a config and fuse-overlayfs package so it can work for rootless users on older kernels. I mean yes users who want to use native overlay can create a new base by uninstalling fuse-overlay and then using it but it needs manual step.

Before we add a new flag let's make sure we understand why it failed. What is wrong with the mechanism where we write down if a user is using fuse-overlayfs and then keep using it once it is set but attempt using native overlay where possible?

I think this can be done but it prohibits the use-case of using fuse-overlayfs, for example if I am on a newer kernel and for some reason I don't want to use native overlay and use fuse-overlay this will prohibit the use-case ( sounds a weird case), but otherwise this SGTM.

The new flag will still require a change in the configuration, so couldn't it be changed just to unset mount_program?

I mean if we have this new flag and we modify quay.io/buildah/stable:latest to use it then c/storage can on runtime use fuse-overlayfs if its on older kernel and native overlay on newer kernel , keeping both users happy. At-least that is what I am thinking right now, I could be wrong though ?

[flouthoc]

Before we add a new flag let's make sure we understand why it failed. What is wrong with the mechanism where we write down if a user is using fuse-overlayfs and then keep using it once it is set but attempt using native overlay where possible?

Edit: I misunderstood it before one change and one bug fix is needed in c/storage I think it might work without a new field.

[lacinos]

Hi, do we still need to use the --device=/dev/fuse in rootless buildah on kernels 5.13 and higher?

[giuseppe]

it depends, if you use a volume for the containers storage, then you can use native overlay in the nested container too.

Otherwise, if you are running on overlay, you need fuse-overlayfs on top of it.

In any case, I t think we can close this issue. @flouthoc are you fine with it?

[flouthoc]

Yes I think we should close this issue, this is not needed anymore. Thanks.