Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow bumping puppet-agent-* deps #9410

Open
wants to merge 1 commit into
base: rpm/develop
Choose a base branch
from
Open

allow bumping puppet-agent-* deps #9410

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions list_updatable_packages
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,13 @@ def find_gemfile_lock_specs(ci_job: str, subdir: str) -> Generator[Tuple[Spec, s
base_path = Path('packages') / subdir

for gem in parse_gemfile_lock(fetch_gemfile_lock(url)):
package_name = f'rubygem-{gem["name"]}'
path = base_path / package_name / f'{package_name}.spec'
if path.is_file():
spec = Spec(path)
if spec.is_updateable:
yield spec, gem['version']
for package_prefix in ['rubygem', 'puppet-agent']:
package_name = f'{package_prefix}-{gem["name"]}'
path = base_path / package_name / f'{package_name}.spec'
if path.is_file():
spec = Spec(path)
if spec.is_updateable:
yield spec, gem['version']


def find_packaged_specs() -> Generator[Tuple[Spec, None], None, None]:
Expand Down Expand Up @@ -191,6 +192,7 @@ def main() -> None:
specs = chain(
find_packaged_specs(),
find_gemfile_lock_specs('foreman-develop-source-release', 'foreman'),
find_gemfile_lock_specs('foreman-installer-develop-source-release', 'foreman'),
Copy link
Member Author

@evgeni evgeni May 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ekohl I recall you had said in the past (in #9075) that this is not worth it, but I think with the puppet-agent gems, it actually is?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that we have many gems that are actually irrelevant for packaging. Then some things end up pulling in older gems, for example because of the old rubocop version we have (which we should update).

You can also see that we don't even pin puppet-strings in our Gemfile (https://github.com/theforeman/foreman-installer/blob/c6422b45f1bb6e1af8e9792857411b077fe306ac/Gemfile#L12) so if that's the only thing then I wonder if there's a better way.

What we can also consider is a more advanced solution where we can inherit things.

Breaking it down we have:

  • Pure runtime: needed to run the installer.
    • puppet-agent (can be AIO)
    • Kafo (+ deps)
  • Build time:
    • puppet-strings (to build the parser cache)
    • librarian-puppet (to download the bundled modules)
  • Development / testing (irrelevant, so not specified)

Build time is actually split into two. librarian-puppet is needed to build the source tarball while puppet-strings is needed to build the cache. The reason is that it only caches items for classes included in the answer file(s). Technically we could generate one huge cache file for all the classes, but that makes the cache bigger and will slow down runtime.

So that makes me think about building some common file + Gemfile with just the ones we use in packaging (kafo, puppet-strings) and generate a lockfile for that. Then also include the common bits in the generic Gemfile.

Any thoughts on that?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, the script finds:

packages/foreman/puppet-agent-oauth 1.1.0
packages/foreman/puppet-agent-yard 0.9.34
packages/foreman/puppet-agent-puppet-strings 4.0.0
packages/foreman/puppet-agent-rgen 0.9.1

The alternative would be to add something like template: puppet to the specs and allow the script to always bump that to "latest", but I am not sure I like that.

I think having a Gemfile.lock with --without test:development would be the right thing (and would also apply to the other projects, actually, it's just that there its less of an issue).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure --without really ignores those dependencies. For example, if you have a git repo listed in your development group bundler will still clone it to determine the dependencies. Even if you specified --without development. That's why I suggested separate files. But there may be different ways (like looking at ENV vars).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I shouldn't have said --without but "a lockfile built in a way that it would not list the unwanted deps" (= what you suggested, without implying an implementation)

find_gemfile_lock_specs('smart-proxy-develop-source-release', 'foreman'),
find_gemfile_lock_specs('katello-master-source-release', 'katello'),
find_gemfile_lock_specs('hammer-cli-master-source-release', 'foreman'),
Expand Down