add script to compare git to rpm repo #9230

evgeni · 2023-03-22T11:08:07Z

No description provided.

evgeni · 2023-03-22T11:09:13Z

compare_rpm_repo.py

+from pathlib import Path
+
+from packaging.version import Version
+from list_updatable_packages import Spec


this implies a git mv list_updatable_packages list_updatable_packages.py but I didn't include that in this PR as it also would require changes to

foreman-packaging/.github/workflows/bump_packages.yml

Line 26 in e4e1ab5

run: ./list_updatable_packages

and I wonder if there is a better way

I thought about merging it into a single script and make it something like ./util --list-updatable vs ./util --compare-git-and-repo, but given the size of this script and the other one I think it could become too big.

evgeni · 2023-03-22T11:12:32Z

% python compare_rpm_repo.py                                     
foreman
In repo but not in git:
[]
In git but not in repo:
[]
katello
In repo but not in git:
[]
In git but not in repo:
[('rubygem-qpid_proton', '0.37.0'), ('rhel8-kickstart-setup', '0.0.2'), ('jsoncpp', '1.8.4')]
plugins
In repo but not in git:
[('yggdrasil', '0.2.0')]
In git but not in repo:
[('yggdrasil-worker-forwarder', '0.0.1')]
client
In repo but not in git:
[('qpid-proton', '0.32.0')]
In git but not in repo:
[]

% python compare_rpm_repo.py --staging
foreman
In repo but not in git:
[]
In git but not in repo:
[]
katello
In repo but not in git:
[]
In git but not in repo:
[('rubygem-qpid_proton', '0.37.0'), ('jsoncpp', '1.8.4'), ('rhel8-kickstart-setup', '0.0.2')]
plugins
In repo but not in git:
[('yggdrasil', '0.2.0')]
In git but not in repo:
[('yggdrasil-worker-forwarder', '0.0.1')]
client
In repo but not in git:
[('qpid-proton', '0.32.0')]
In git but not in repo:
[]

rhel8-kickstart-setup fix is in fix rhel8-kickstart-setup shebang to use python3 #9231
yggdrasil-worker-forwarder fixed in fix yggdrasil-worker-forwarder entry in comps #9236
yggdrasil is special (it's packaged for client but used in plugins and client)
qpid-proton is special (it's packaged for katello but used for katello and client)
jsoncpp is build-time only dep (see Add jsoncpp #6229)
rubygem-qpid_proton fix is in drop rubygem-qpid_proton package #9255

ehelms · 2023-03-22T11:49:44Z

A couple of questions:

Does this catch things that obal verify-koji-tag does not?
This creates a(nother) split in where you go and what you use to do one thing or another.
The name of the script is not as obvious to me what repositories are being compared to whom. If I ran this on the rpm/3.6 branch I would expect it to compare 3.6 repositories?

evgeni · 2023-03-22T11:53:17Z

obal verify-koji-tag

damn I forgot about that one!

The name of the script is not as obvious to me what repositories are being compared to whom. If I ran this on the rpm/3.6 branch I would expect it to compare 3.6 repositories?

and it will!

evgeni · 2023-03-22T12:11:39Z

so obal verify-koji-tag compares the list of packages in git with the one that is allowed (koji add-pkg) in the tag (using koji list-pkgs --tag ...)

the script here compares the list of packages and their versions between git and whatever is on our repos, the reasoning is:

just because a package is allowed (pkg-add) doesn't mean there is a build
just because a package is built doesn't mean it shows up in the repo

The first one could be solved by using koji list-tagged --latest instead of koji list-pkgs.
The second one can't be fixed by koji itself, but it might be not that much of a problem overall.

I don't mind too much where the script lives (even tho I like that it doesn't need a koji cert and can be run via a github action daily), but I want to be able to compare packages with versions :)

ehelms · 2023-03-23T17:34:01Z

compare_rpm_repo.py

+from packaging.version import Version
+from list_updatable_packages import Spec
+
+DISTS = ['el8']


To Do to use this?

ekohl · 2023-03-23T19:26:14Z

compare_rpm_repo.py

+    r = requests.get(f'{repo_url}/repodata/repomd.xml')
+    repomd = ET.fromstring(r.content)
+
+    sqlite_location = repomd.find('.//*[@type="primary_db"]/{http://linux.duke.edu/metadata/repo}location').get('href')


FYI: https://fedoraproject.org/wiki/Changes/createrepo_c_1.0.0 has:

Update createrepo_c to 1.0.0, new release will include change of default compression to zstd, no longer generating metadata in sqlite database format by default and simplified comps xml type in repodata.

So that would invalidate this approach.

Whyyyyyy. SQLite is so nice for that!

Technically we can make it explicit and use --database to generate it, which is probably a good thing to do today. It mentions Fedora already does this.

It would be good to also look at zstd decompression.

You could also start to depend on https://pypi.org/project/repomd but that only has gzip support and also relies on defusedxml as an additional dependency.

At that point I could just use dnf ;)

Doesn't zstd break EL7 which we still serve?

Yes, but at this point it's just the defaults that they're proposing to change. It'll be a while before that affects us, but I just saw it pop up on fedora-devel and it looked relevant :)

evgeni · 2023-03-24T09:34:27Z

@ehelms side-quest(ion): we have qpid-proton in the katello folder as that's a dependency for the gem and some other things we need. however, it is also a dependency (via python3-qpid-proton subpackage) of gofer that is in the client folder. because of that, it used to be tagged into both katello and client, but seems not anymore. should this be fixed, and if, how can we ensure this doesn't happen again?

ehelms · 2023-03-24T12:42:14Z

@ehelms side-quest(ion): we have qpid-proton in the katello folder as that's a dependency for the gem and some other things we need. however, it is also a dependency (via python3-qpid-proton subpackage) of gofer that is in the client folder. because of that, it used to be tagged into both katello and client, but seems not anymore. should this be fixed, and if, how can we ensure this doesn't happen again?

Ahh this happened when we switched away from tito, and now qpid-proton is defined for katello packages (https://github.com/theforeman/foreman-packaging/blob/rpm/develop/package_manifest.yaml#L507) but not getting the client tags associated to it. We could move it to it's own special little group and then make that a child of katello and client?

evgeni · 2023-03-24T12:47:47Z

@ehelms side-quest(ion): we have qpid-proton in the katello folder as that's a dependency for the gem and some other things we need. however, it is also a dependency (via python3-qpid-proton subpackage) of gofer that is in the client folder. because of that, it used to be tagged into both katello and client, but seems not anymore. should this be fixed, and if, how can we ensure this doesn't happen again?

Ahh this happened when we switched away from tito, and now qpid-proton is defined for katello packages (https://github.com/theforeman/foreman-packaging/blob/rpm/develop/package_manifest.yaml#L507) but not getting the client tags associated to it. We could move it to it's own special little group and then make that a child of katello and client?

Not sure how that would work as obal can add missing tags, but when you build the initial tag is done by koji, so effectively you'd have to obal build twice to get both tags? But if that'd work, I'm down.

Same applies for yggdrasil that goes to plugins and client btw :)

ehelms · 2023-03-28T00:26:19Z

Not sure how that would work as obal can add missing tags, but when you build the initial tag is done by koji, so effectively you'd have to obal build twice to get both tags? But if that'd work, I'm down.

Hrmm, you are right, it's not built to handle same build different tags in the list of tags. However, I think because we run the tags in a loop, Ansible will run each chunk serially? So if it finds the build it will tag it?

evgeni · 2023-03-28T10:04:35Z

Mhh, you're right. Reading the code, this is what should happen. I agree. I'll try to add an explicit test for that behaviour

Edit: theforeman/obal#341

evgeni · 2023-03-29T07:08:06Z

TIL: there is also obal check 🤯

ehelms · 2023-03-29T15:59:39Z

TIL: there is also obal check exploding_head

I mean, I tried to close the gaps and add all the things to it 😄

theforeman-bot added the Not yet reviewed label Mar 22, 2023

evgeni commented Mar 22, 2023

View reviewed changes

evgeni force-pushed the rpm-compare branch from 86a2032 to 5a29f7c Compare March 22, 2023 11:12

ehelms reviewed Mar 23, 2023

View reviewed changes

compare_rpm_repo.py

from packaging.version import Version

from list_updatable_packages import Spec

DISTS = ['el8']

Copy link

Member

ehelms Mar 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To Do to use this?

theforeman-bot added the RPM label Mar 23, 2023

ekohl reviewed Mar 23, 2023

View reviewed changes

evgeni mentioned this pull request Mar 29, 2023

Update tags instructions theforeman/tool_belt#495

Merged

evgeni force-pushed the rpm-compare branch from 5a29f7c to c32c495 Compare March 29, 2023 12:13

evgeni added 4 commits December 21, 2023 08:42

add script to compare git to rpm repo

1a00422

fixup compare script to use xml not sql

2148f17

skip satellite folder, we do not publish that

748dc87

map katello versions correctly

dfff150

evgeni force-pushed the rpm-compare branch from c32c495 to dfff150 Compare December 21, 2023 07:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add script to compare git to rpm repo #9230

add script to compare git to rpm repo #9230

evgeni commented Mar 22, 2023

evgeni Mar 22, 2023

ekohl Mar 23, 2023

evgeni commented Mar 22, 2023 •

edited

Loading

ehelms commented Mar 22, 2023

evgeni commented Mar 22, 2023 •

edited

Loading

evgeni commented Mar 22, 2023

ehelms Mar 23, 2023

ekohl Mar 23, 2023

evgeni Mar 23, 2023

ekohl Mar 23, 2023

evgeni Mar 23, 2023

ekohl Mar 23, 2023

evgeni commented Mar 24, 2023

ehelms commented Mar 24, 2023

evgeni commented Mar 24, 2023

ehelms commented Mar 28, 2023

evgeni commented Mar 28, 2023 •

edited

Loading

evgeni commented Mar 29, 2023

ehelms commented Mar 29, 2023

add script to compare git to rpm repo #9230

Are you sure you want to change the base?

add script to compare git to rpm repo #9230

Conversation

evgeni commented Mar 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evgeni commented Mar 22, 2023 • edited Loading

ehelms commented Mar 22, 2023

evgeni commented Mar 22, 2023 • edited Loading

evgeni commented Mar 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evgeni commented Mar 24, 2023

ehelms commented Mar 24, 2023

evgeni commented Mar 24, 2023

ehelms commented Mar 28, 2023

evgeni commented Mar 28, 2023 • edited Loading

evgeni commented Mar 29, 2023

ehelms commented Mar 29, 2023

evgeni commented Mar 22, 2023 •

edited

Loading

evgeni commented Mar 22, 2023 •

edited

Loading

evgeni commented Mar 28, 2023 •

edited

Loading