Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report pulled image size #1208

Merged
merged 1 commit into from
Mar 14, 2023
Merged

Conversation

nievesmontero
Copy link
Collaborator

#752

Signed-off-by: Nieves Montero [email protected]

@softwarefactory-project-zuul
Copy link

Build failed.

unit-test FAILURE in 23m 12s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 2m 57s
✔️ system-test-fedora-rawhide SUCCESS in 33m 28s
✔️ system-test-fedora-36 SUCCESS in 12m 35s
✔️ system-test-fedora-35 SUCCESS in 13m 50s

@softwarefactory-project-zuul
Copy link

Build failed.

unit-test FAILURE in 27m 39s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 28s
✔️ system-test-fedora-rawhide SUCCESS in 42m 19s
✔️ system-test-fedora-36 SUCCESS in 13m 12s
✔️ system-test-fedora-35 SUCCESS in 14m 23s

@softwarefactory-project-zuul
Copy link

Build failed.

✔️ unit-test SUCCESS in 9m 53s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 35s
✔️ system-test-fedora-rawhide SUCCESS in 20m 09s
system-test-fedora-36 TIMED_OUT in 20m 31s
system-test-fedora-35 RETRY_LIMIT in 46s

@debarshiray
Copy link
Member

recheck

@softwarefactory-project-zuul
Copy link

Build failed.

✔️ unit-test SUCCESS in 10m 04s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 38s
✔️ system-test-fedora-rawhide SUCCESS in 21m 19s
system-test-fedora-36 TIMED_OUT in 20m 36s
✔️ system-test-fedora-35 SUCCESS in 13m 41s

@debarshiray
Copy link
Member

The previous RETRY_LIMIT on Fedora 35 seems to have been a transient error:

TASK [Install RPM packages]
fedora-35 | ERROR
fedora-35 | {
fedora-35 |   "msg": "Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 (IP: 38.145.60.21)",
fedora-35 |   "rc": 1,
fedora-35 |   "results": []
fedora-35 | }

Copy link
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, Nieves! Some comments below:

src/cmd/create.go Outdated Show resolved Hide resolved
src/cmd/create.go Outdated Show resolved Hide resolved
src/cmd/create.go Outdated Show resolved Hide resolved
src/pkg/podman/podman.go Outdated Show resolved Hide resolved
src/pkg/podman/podman.go Outdated Show resolved Hide resolved
src/cmd/create.go Outdated Show resolved Hide resolved
src/cmd/create.go Outdated Show resolved Hide resolved
debarshiray added a commit to debarshiray/toolbox that referenced this pull request Jan 18, 2023
Currently, the CI has been timing out on Fedora 36 nodes [1].  It's
possible that this is due to the recent expansion of the test suite,
which already required increasing the timeout for Fedora Rawhide [2].

[1] containers#1191
    containers#1208

[2] Commit e8f4e9c
    containers#1195
debarshiray added a commit to debarshiray/toolbox that referenced this pull request Jan 18, 2023
Currently, the CI has been timing out on Fedora 36 nodes [1].  It's
possible that this is due to the recent expansion of the test suite,
which already required increasing the timeout for Fedora Rawhide [2].

[1] containers#1191
    containers#1208

[2] Commit e8f4e9c
    containers#1195

containers#1212
@debarshiray
Copy link
Member

Due to the recent expansion of the test suite, we might have to increase the timeout for stable Fedoras: #1212

We recently had to do that for Fedora Rawhide.

@softwarefactory-project-zuul
Copy link

Build succeeded.

✔️ unit-test SUCCESS in 10m 27s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 32s
✔️ system-test-fedora-rawhide SUCCESS in 21m 18s
✔️ system-test-fedora-36 SUCCESS in 22m 17s
✔️ system-test-fedora-35 SUCCESS in 13m 52s

Copy link
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for rebasing this on top of #1212

We still need to address the review comments about the code, though.

src/pkg/podman/podman.go Outdated Show resolved Hide resolved
src/pkg/podman/podman.go Outdated Show resolved Hide resolved
@nievesmontero nievesmontero force-pushed the image-size branch 2 times, most recently from 005f794 to a1d9fa0 Compare January 19, 2023 12:17
@softwarefactory-project-zuul
Copy link

Build failed.

✔️ unit-test SUCCESS in 10m 50s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 22s
✔️ system-test-fedora-rawhide SUCCESS in 14m 07s
✔️ system-test-fedora-36 SUCCESS in 28m 57s
system-test-fedora-35 RETRY_LIMIT in 40m 50s

@debarshiray
Copy link
Member

recheck

@softwarefactory-project-zuul
Copy link

Build succeeded.

✔️ unit-test SUCCESS in 10m 48s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 12s
✔️ system-test-fedora-rawhide SUCCESS in 21m 51s
✔️ system-test-fedora-36 SUCCESS in 12m 57s
✔️ system-test-fedora-35 SUCCESS in 36m 18s

Copy link
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates, @nievesmontero ! Could you please squash the commits into one? You can use git rebase -i for that.

Further comments below:

src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
src/cmd/create.go Outdated Show resolved Hide resolved
@softwarefactory-project-zuul
Copy link

Build succeeded.

✔️ unit-test SUCCESS in 10m 48s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 4m 07s
✔️ system-test-fedora-rawhide SUCCESS in 21m 18s
✔️ system-test-fedora-36 SUCCESS in 13m 23s
✔️ system-test-fedora-35 SUCCESS in 32m 23s

Copy link
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates, @nievesmontero ! It looks like there are some remnants from a git rebase or git merge or something similar that had conflicts and wasn't fully cleaned up.

src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
Copy link
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of the comments from the past reviews still need to be addressed.

src/cmd/create.go Outdated Show resolved Hide resolved
src/cmd/create.go Show resolved Hide resolved
src/pkg/podman/podman.go Outdated Show resolved Hide resolved
@debarshiray
Copy link
Member

The main things that are left to do here are:

  • Avoid a hard dependency on Skopeo by ensuring that the errors from skopeo inspect ... are not hard failures.

  • Use a custom Image type that implements the json.Unmarshaler interface for parsing JSON.

  • Move the docker:// prefix into skopeo.Inspect.

See above for details on each of these points.

@nievesmontero nievesmontero force-pushed the image-size branch 2 times, most recently from e7d832a to 2d86d83 Compare February 22, 2023 10:46
src/cmd/create.go Outdated Show resolved Hide resolved
@nievesmontero nievesmontero force-pushed the image-size branch 2 times, most recently from 328ea8a to 3033d29 Compare February 22, 2023 10:50
@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/786223ef1bfc4c89a9657c0b5d4cfa25

✔️ unit-test SUCCESS in 8m 20s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 2m 55s
✔️ system-test-fedora-rawhide SUCCESS in 14m 32s
✔️ system-test-fedora-37 SUCCESS in 12m 14s
✔️ system-test-fedora-36 SUCCESS in 12m 18s

@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/c7daff2ba8084feaa4608fe06e77c430

✔️ unit-test SUCCESS in 9m 04s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 2m 42s
✔️ system-test-fedora-rawhide SUCCESS in 14m 56s
✔️ system-test-fedora-37 SUCCESS in 12m 41s
✔️ system-test-fedora-36 SUCCESS in 12m 44s

@softwarefactory-project-zuul
Copy link

Build failed.
https://softwarefactory-project.io/zuul/t/local/buildset/0ceef1fe98aa416aa0f6bcf68b555964

✔️ unit-test SUCCESS in 9m 24s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 2m 56s
✔️ system-test-fedora-rawhide SUCCESS in 15m 29s
system-test-fedora-37 NODE_FAILURE Node request 200-0005737446 failed in 0s
✔️ system-test-fedora-36 SUCCESS in 12m 59s

@nievesmontero
Copy link
Collaborator Author

recheck

@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/a2d383d8821a462e9bc997b8f6653335

✔️ unit-test SUCCESS in 9m 15s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 05s
✔️ system-test-fedora-rawhide SUCCESS in 15m 27s
✔️ system-test-fedora-37 SUCCESS in 13m 12s
✔️ system-test-fedora-36 SUCCESS in 12m 59s

Copy link
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates, @nievesmontero ! This is looking a lot better, well done. Some minor details:

src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
src/cmd/create.go Outdated Show resolved Hide resolved
src/cmd/create.go Outdated Show resolved Hide resolved
Copy link
Member

@debarshiray debarshiray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One quick note about the commit message ...

Unless the change is massive, it's not so important to write things like:

The following package and file have been created src/pkg/skopeo/skopeo.go

... because it's obvious from looking at the commit (with git show) and is of no interest to downstream distributors. What's more important is that it adds an optional dependency on Skopeo, because that's going to be important for downstream distributors, and to clarify why it's optional. Those details aren't so obvious until someone carefully looks at the commit, but they are still important.

@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/f0ac35f400974c0dae6e7517d1357eb7

✔️ unit-test SUCCESS in 9m 03s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 08s
✔️ unit-test-restricted SUCCESS in 8m 28s
✔️ system-test-fedora-rawhide SUCCESS in 15m 00s
✔️ system-test-fedora-37 SUCCESS in 12m 31s
✔️ system-test-fedora-36 SUCCESS in 12m 36s

src/cmd/create.go Outdated Show resolved Hide resolved
src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/ca1cebdba5ed4076880fc001f8f9c2ad

✔️ unit-test SUCCESS in 8m 42s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 01s
✔️ unit-test-restricted SUCCESS in 8m 29s
✔️ system-test-fedora-rawhide SUCCESS in 14m 39s
✔️ system-test-fedora-37 SUCCESS in 12m 19s
✔️ system-test-fedora-36 SUCCESS in 12m 13s

@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/df2a31a2de054691a9d16fb9a135c186

✔️ unit-test SUCCESS in 8m 52s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 2m 56s
✔️ unit-test-restricted SUCCESS in 8m 23s
✔️ system-test-fedora-rawhide SUCCESS in 15m 00s
✔️ system-test-fedora-37 SUCCESS in 12m 43s
✔️ system-test-fedora-36 SUCCESS in 12m 41s

src/pkg/skopeo/skopeo.go Outdated Show resolved Hide resolved
src/pkg/skopeo/skopeo.go Show resolved Hide resolved
@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/b9f2f9f6df934391b855c861bc6f927a

✔️ unit-test SUCCESS in 8m 30s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 2m 56s
✔️ unit-test-restricted SUCCESS in 7m 46s
✔️ system-test-fedora-rawhide SUCCESS in 14m 28s
✔️ system-test-fedora-37 SUCCESS in 11m 35s
✔️ system-test-fedora-36 SUCCESS in 11m 37s

This uses 'skopeo inspect' to get the size of the image on the registry,
which is usually less than the size of the image in a local
containers/storage image store after download (eg., 'podman images'),
because they are kept compressed on the registry.  Skopeo >= 1.10.0 is
needed to retrieve the sizes [1].

However, this doesn't add a hard dependency on Skopeo to accommodate
size-constrained operating systems like Fedora CoreOS.  If skopeo(1) is
missing or too old, then the size of the image won't be shown, but
everything else would continue to work as before.

Some changes by Debarshi Ray.

[1] Skopeo commit d9dfc44888ff71a6
    containers/skopeo@d9dfc44888ff71a6
    containers/skopeo#641

containers#752

Signed-off-by: Nieves Montero <[email protected]>
@softwarefactory-project-zuul
Copy link

Build succeeded.
https://softwarefactory-project.io/zuul/t/local/buildset/1bd61c59e3d042e9878b227cb3c786a3

✔️ unit-test SUCCESS in 9m 14s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 11s
✔️ unit-test-restricted SUCCESS in 9m 33s
✔️ system-test-fedora-rawhide SUCCESS in 15m 54s
✔️ system-test-fedora-37 SUCCESS in 13m 33s
✔️ system-test-fedora-36 SUCCESS in 12m 36s

@debarshiray debarshiray merged commit a1c3095 into containers:main Mar 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants