Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloud: add new Uploader interface and implement for AWS #1185

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

mvo5
Copy link
Contributor

@mvo5 mvo5 commented Jan 30, 2025

This commit adds a new cloud.Uploader interface that combines the upload and register into a single operation. The rational is that with that we avoid leaking resource if e.g. the upload works but the registration fails
(c.f. #1145). This would replace most of the existing bib uplodaer code so that it is shareable.

This would looks like this:

After that I would like to look into the following:

  • I would like remake the cloud interface for images around something like this
  • can we remove (most?) of the existing awscloud in images. because osbuild-compser impleents it all too but has now moved to aws-v2 and it seems we have a lot of unused code in images right now(?)
  • how important is cmd/boot-aws ? it uses a lot of the existing API but the composer does not have it

@mvo5 mvo5 requested a review from achilleas-k January 30, 2025 19:34
mvo5 added a commit to mvo5/bootc-image-builder that referenced this pull request Jan 31, 2025
This commit switches bib to the new uploader interface in images.

This requires osbuild/images#1185
mvo5 added a commit to mvo5/image-builder-cli that referenced this pull request Jan 31, 2025
This commit adds a new `upload` command that can be used to
upload a raw image to the cloud. Currently only AWS is
supported but as images adds more clouds to the uploader interfac
we can trivially expand more.

Note that this needs osbuild/images#1185
mvo5 added a commit to mvo5/image-builder-cli that referenced this pull request Jan 31, 2025
This commit adds a new `upload` command that can be used to
upload a raw image to the cloud. Currently only AWS is
supported but as images adds more clouds to the uploader interfac
we can trivially expand more.

Note that this needs osbuild/images#1185
@mvo5 mvo5 changed the title [RFC] cloud: add new Uploader interface and implement for AWS cloud: add new Uploader interface and implement for AWS Jan 31, 2025
This commit adds a new `cloud.Uploader` interface that combines
the upload and register into a single operation. The rational
is that with that we avoid leaking resource if e.g. the upload
works but the registration fails.
@mvo5 mvo5 marked this pull request as ready for review January 31, 2025 16:32
@mvo5 mvo5 requested a review from a team as a code owner January 31, 2025 16:32
@mvo5 mvo5 requested review from schuellerf and supakeen January 31, 2025 16:32
@achilleas-k
Copy link
Member

  • how important is cmd/boot-aws ? it uses a lot of the existing API but the composer does not have it

It's used in our testing and was meant to be the first of a handful of similar commands for different clouds. We should still do that.

@achilleas-k
Copy link
Member

  • can we remove (most?) of the existing awscloud in images. because osbuild-compser impleents it all too but has now moved to aws-v2 and it seems we have a lot of unused code in images right now(?)

I would prefer the opposite. Having cloud uploaders as part of image definitions might seem strange, but we discussed it a few times in the past and came to the agreement that it makes sense because integrating image definitions with a client for their target platform's API makes for a more complete story. One argument that often comes up goes "an AMI isn't an AMI until it's uploaded and registered on AWS EC2".

Copy link
Member

@achilleas-k achilleas-k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the end goal here and what does this PR do?

The answer I'm assuming is that it defines an interface for uploading stuff to cloud platforms and begins with one implementation based on the existing AWS code.
But I think it doesn't go far enough and isn't providing much value right now (though, the testing is nice). I'd like to see this common interface happen, but I think it makes more sense to look at the other cloud providers and consider what it might look like when we add those to the mix. UploadAndRegister() sounds generic enough but I don't know if it makes sense for Azure, or GCP, or Oracle (it probably does). It's not set in stone, we can always change later, so maybe we should go with this and start iterating with the other platforms, but if we're going to start working on cloud clients seriously, I think we should review what we already have and potentially tear it down and build it back up in a way that works for what we're trying to do.

The boot-aws command was written specifically for our testing, so the run, setup, and teardown subcommands were written to serve development and testing and those requirements in turn informed some of the choices in the client library (though parts of it were already there from the split from osbuild-composer). For example, I never considered adding a Check() function like we have now because it was never targeted at any user that might need that function; the keys were set in the CI setup and never looked at again and when I use it myself for troubleshooting, I have my own set that works (and have probably way more permissions than they need but let's not think about that right now). I'm all in favour of making a cloud.Uploader interface for all our uploaders, but I also want to do it right, especially if it's going to serve osbuild-composer, ib-cli, and bib.

Now all that said, I'm fine to merge this as is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants