update_agent: support rebase to OCI pullspec #1241

jbtrystram · 2024-12-04T14:04:10Z

Add a configuration knob in update to optionnaly use OCI pullspec instead of ostree checksums.

This will default to false.
Requires coreos/fedora-coreos-cincinnati#99 and coreos/rpm-ostree#5120

travier

Very quick look but looks good 👍🏻

src/rpm_ostree/cli_deploy.rs

tests/fixtures/00-config-sample.toml

jlebon

Let's cross-link to coreos/fedora-coreos-tracker#1823 in the commit message for the larger context.

tests/fixtures/00-config-sample.toml

src/cincinnati/mod.rs

src/rpm_ostree/cli_deploy.rs

jbtrystram

I updated this to use ostree-remote-image so we validate the ostree signatures, as suggested in coreos/fedora-coreos-tracker#1823 (comment)

I think this is ready for review now. I will clean up and sqash commits once the design is agreed

src/rpm_ostree/cli_status.rs

src/rpm_ostree/cli_deploy.rs

jbtrystram · 2025-01-13T13:47:10Z

src/rpm_ostree/cli_status.rs

    pub fn base_revision(&self) -> String {
-        self.base_checksum
+        self.container_image_reference
            .clone()
+            .map(|pullspec| {
+                pullspec
+                    .strip_prefix("ostree-remote-image:fedora:docker://")
+                    .map(|s| s.to_string())
+            })
+            .flatten()
+            .or(self.base_checksum.clone())


Maybe it would make more sense to add an Option<String> to the Deployement struct to be able to have both the pullspec and the base commit ?
However the cincinnati graph does not contains checksum in the OCI case so we can't compare ostree commits when doing rebase later anyway.

haven't had the need for it in the Zincati context, except maybe if we want Zincati to do OCI -> OSTree ? (in that case, after calling rpm-ostree deploy zincati will fail as the checksum won't match

jbtrystram · 2025-01-13T14:36:54Z

OK, had some more discussion with Jlebon and travier today and there are some more things to flesh out

src/rpm_ostree/cli_status.rs

src/rpm_ostree/mod.rs

jbtrystram · 2025-01-15T17:07:39Z

Allright, I think this is ready for review now. I highlighted a couple of things I am unsure of.
I will squash the commits after addressing review comments

Switch boot images to use OCI for updates. This is a step towards bootable containers support and bootc rebase. See https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates Requires coreos/zincati#1241 See coreos/fedora-coreos-tracker#1823

src/cincinnati/client.rs

src/cincinnati/mod.rs

src/rpm_ostree/cli_deploy.rs

src/rpm_ostree/cli_status.rs

src/cincinnati/mod.rs

src/rpm_ostree/actor.rs

tests/fixtures/rpm-ostree-oci-status.json

jlebon · 2025-01-28T21:56:47Z

src/rpm_ostree/cli_finalize.rs

+    // get the latest commit but that would be racy, so let's finalize the latest
+    // commit.
+    if release.is_oci {
+        cmd.arg("--allow-missing-checksum")


I think probably the cleaner thing is to have rpm-ostree accept a digested pullspec the way it accepts commit checksums and it can verify that it matches the digested pullspec that was staged. Then we can drop this.

Yes, definitely, but that requires work in rpm-ostree, let's not block on it ?

src/rpm_ostree/mod.rs

src/update_agent/actor.rs

jbtrystram · 2025-02-05T15:52:36Z

Allright, I reworked this PR a bunch following the review comments, that ended up being pretty substantial changes.

Overview :

A local deployment is now unserialized with an enum containing either a Checksum or a Pullspec. Both are simply strings, but I use that to match my way around, rather than a is_oci boolean.
custom-origin fields are now purely cosmetic : Zincati will cue from the container-image-reference.
Now properly parses the OCI manifest output from rpm-ostree --json and get the fedora-coreos.stream key from that. This is working around rpm-ostree status --json does not properly serialize ostree.manifest for OCI deployments rpm-ostree#5196
imported a couple of new crates : oci-spec for the above issue and ostree-rs to properly parse ostree image references and containers pull specs. I removed the hacky split(':') and split(":sha256")` in favor of that.

Testing

Here are my notes to test this out it case someone wants to experiment:

Start a VM with Zincati disabled : cosa run --qemu-image fedora-coreos-41.20241215.2.0-qemu.x86_64.qcow2 --add-ignition noautoupdate

Rebase to an OCI image in graph (recent enough to get rpm-ostree-2024.9) :

    sudo rpm-ostree rebase ostree-remote-image:fedora:registry:quay.io/fedora/fedora-coreos@sha256:74448159fae255797012a12eaf9089ea3c1018ffb0b3e76e03483cba59b50bb2 \
     --custom-origin-url=quay.io/fedora/fedora-coreos@sha256:74448159fae255797012a12eaf9089ea3c1018ffb0b3e76e03483cba59b50bb2 \
     --custom-origin-description="Fedora CoreOS testing stream"
     sudo reboot

Now import Zincati build with this PR then:

    sudo rm /etc/zincati/config.d/90-disable-auto-updates.toml
    sudo systemctl restart zincati

Zincati should now pick up an OCI update through the OCI graph

Getting zincati custom build in the COSA VM

While we can make a custom FCOS build with an override to get our Zincati build inside, it will be erased by the content of the OCI image with a rebase, so here is how I did, after building zincati and copy the binary into $COSA_DIR/tmp

rpm-ostree usroverlay
cp /mnt/workdir-tmp/zincati /usr/libexec/zincati
cp /mnt/workdir-tmp/zincati.rules /usr/share/polkit-1/rules.d/zincati.rules

# restart polkit
systemctl restart polkit.service

# re enable updates
rm /etc/zincati/config.d/90-disable-auto-updates.toml

# restart zincati
systemctl restart zincati.service

# Show the update
journalctl --no-pager -o cat -u zincati --follow

In the meantime, I'll squash this and try to make commits that make sense :)

When going through the update graph, Zincati searches for a node that matches the current deployement, as the starting point for the update. If the graph does not contains a matching node, zincati errors silently. While this should almost never happen, it's useful to print a warning to explain the situation if we hit that case.

Add support for local OCI deployements, no longer erroring out. Use the `ostree-ext` crate to parse the ostree image reference properly. This will allow zincati to query the OCI graph and rebase to OCI images for updates. Also change the base-commit-meta object to support the OCI case where this is actually an escaped serialized OCI manifest. Use the `oci-spec` crate to deserialize it and extract the core os stream info from the base OCI annotations. This works around coreos/rpm-ostree#5196

Local deployements are now unserialized with a enum being either `Checksum` or `Pullspec`. This will cue the cincinnati client to request the OCI graph in the pullspec case. [1] The custom-origin-description, if set, is passed along the rebase call, and the custom-url is updated to match the new pullspec. [2] Finally, update the polkit policy to allow Zincati to do a rebase operation. [1] Requires coreos/fedora-coreos-cincinnati#99 and coreos/rpm-ostree#5120

Switch boot images to use OCI for updates. This is a step towards bootable containers support and bootc rebase. See https://fedoraproject.org/wiki/Changes/CoreOSOstree2OCIUpdates Requires coreos/zincati#1241 See coreos/fedora-coreos-tracker#1823

jbtrystram force-pushed the oci_scheme branch 2 times, most recently from f0713e1 to 1ca8a87 Compare December 4, 2024 14:18

jbtrystram changed the title ~~update_agent: support rebaseing to OCI pullspec~~ update_agent: support rebase to OCI pullspec Dec 4, 2024

travier reviewed Dec 4, 2024

View reviewed changes

cgwalters reviewed Dec 4, 2024

View reviewed changes

src/rpm_ostree/cli_deploy.rs Outdated Show resolved Hide resolved

dustymabe reviewed Dec 9, 2024

View reviewed changes

tests/fixtures/00-config-sample.toml Outdated Show resolved Hide resolved

jlebon reviewed Dec 10, 2024

View reviewed changes

tests/fixtures/00-config-sample.toml Outdated Show resolved Hide resolved

src/cincinnati/mod.rs Outdated Show resolved Hide resolved

src/cincinnati/mod.rs Outdated Show resolved Hide resolved

src/rpm_ostree/cli_deploy.rs Outdated Show resolved Hide resolved

jbtrystram force-pushed the oci_scheme branch 4 times, most recently from 11aa880 to 7000352 Compare December 16, 2024 13:51

jbtrystram commented Jan 13, 2025

View reviewed changes

jbtrystram marked this pull request as draft January 13, 2025 14:36

jbtrystram commented Jan 15, 2025

View reviewed changes

src/rpm_ostree/cli_status.rs Outdated Show resolved Hide resolved

jbtrystram commented Jan 15, 2025

View reviewed changes

src/rpm_ostree/mod.rs Outdated Show resolved Hide resolved

jbtrystram requested review from jlebon and cgwalters January 15, 2025 17:07

jbtrystram marked this pull request as ready for review January 15, 2025 17:20

jbtrystram mentioned this pull request Jan 16, 2025

Move from OSTree to OCI for updates coreos/fedora-coreos-tracker#1823

Open

jbtrystram force-pushed the oci_scheme branch from 90c5b25 to 450550b Compare January 23, 2025 09:59

jbtrystram mentioned this pull request Jan 23, 2025

[testing] deploy container images by default coreos/fedora-coreos-config#3333

Open

cgwalters mentioned this pull request Jan 23, 2025

parse OCI deployment status into ImageManifest coreos/rpm-ostree#5208

Closed

jbtrystram force-pushed the oci_scheme branch 2 times, most recently from add9bf7 to c5e0459 Compare January 24, 2025 10:59

jlebon reviewed Jan 28, 2025

View reviewed changes

jbtrystram mentioned this pull request Jan 30, 2025

rust/rpmostree-client: parse OCI deployment manifest coreos/rpm-ostree#5266

Open

jbtrystram force-pushed the oci_scheme branch from d3a9096 to 6043d0b Compare February 6, 2025 16:19

jbtrystram closed this Feb 7, 2025

jbtrystram force-pushed the oci_scheme branch from 6043d0b to a221fbc Compare February 7, 2025 08:34

jbtrystram reopened this Feb 7, 2025

jbtrystram force-pushed the oci_scheme branch from a99b5bf to 5b17cae Compare February 7, 2025 10:13

jbtrystram requested a review from jlebon February 7, 2025 12:28

jbtrystram added 3 commits February 7, 2025 13:36

apply clippy suggestions

eb84d1e

jbtrystram force-pushed the oci_scheme branch 4 times, most recently from 71ad09b to 6f2313f Compare February 7, 2025 14:21

jbtrystram added 2 commits February 7, 2025 17:37

use cosa containeri for CI

eb6f05e

jbtrystram force-pushed the oci_scheme branch from 6f2313f to eb6f05e Compare February 7, 2025 16:37

jbtrystram mentioned this pull request Feb 7, 2025

overlay.d/15fcos: add a migration script to move to OCI images coreos/fedora-coreos-config#3355

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update_agent: support rebase to OCI pullspec #1241

update_agent: support rebase to OCI pullspec #1241

jbtrystram commented Dec 4, 2024

travier left a comment

jlebon left a comment

jbtrystram left a comment

jbtrystram Jan 13, 2025

jbtrystram Feb 5, 2025 •

edited

Loading

jbtrystram commented Jan 13, 2025

jbtrystram commented Jan 15, 2025 •

edited

Loading

jlebon Jan 28, 2025

jbtrystram Feb 7, 2025

jbtrystram commented Feb 5, 2025

update_agent: support rebase to OCI pullspec #1241

Are you sure you want to change the base?

update_agent: support rebase to OCI pullspec #1241

Conversation

jbtrystram commented Dec 4, 2024

travier left a comment

Choose a reason for hiding this comment

jlebon left a comment

Choose a reason for hiding this comment

jbtrystram left a comment

Choose a reason for hiding this comment

jbtrystram Jan 13, 2025

Choose a reason for hiding this comment

jbtrystram Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

jbtrystram commented Jan 13, 2025

jbtrystram commented Jan 15, 2025 • edited Loading

jlebon Jan 28, 2025

Choose a reason for hiding this comment

jbtrystram Feb 7, 2025

Choose a reason for hiding this comment

jbtrystram commented Feb 5, 2025

Overview :

Testing

Getting zincati custom build in the COSA VM

jbtrystram Feb 5, 2025 •

edited

Loading

jbtrystram commented Jan 15, 2025 •

edited

Loading