Add support for Ubuntu comand-not-found (CNF) indices

[pulpbot]

Author: wilful (wilful)

Redmine Issue: 9239, https://pulp.plan.io/issues/9239

---

Hi! I trying create mirror for official ubuntu repository: http://archive.ubuntu.com/ubuntu/
After creation on my test Ubuntu server i see error:

/pulp/content/archive.ubuntu.com/dists/focal/main/cnf/Commands-amd64: 404  Not Found [IP: 172.24.160.15 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.

It content (CNF) have in source but not in my repository
I use verbatim publisher, on_demand policy

      "pulp_href": "/pulp/api/v3/remotes/deb/apt/25c10483-64e2-492a-ac30-6efb5bbe3683/",
      "pulp_created": "2021-08-13T13:58:19.364006Z",
      "name": "archive.ubuntu.com",
      "url": "http://archive.ubuntu.com/ubuntu/",
      "ca_cert": null,
      "client_cert": null,
      "tls_validation": true,
      "proxy_url": null,
      "pulp_labels": {},
      "pulp_last_updated": "2021-08-13T13:58:19.364027Z",
      "download_concurrency": null,
      "max_retries": null,
      "policy": "on_demand",
      "total_timeout": null,
      "connect_timeout": null,
      "sock_connect_timeout": null,
      "sock_read_timeout": null,
      "headers": null,
      "rate_limit": null,
      "distributions": "trusty focal",
      "components": null,
      "architectures": "amd64",
      "sync_sources": false,
      "sync_udebs": false,
      "sync_installer": false,
      "gpgkey": null,
      "ignore_missing_package_indices": false
    },

Can i add it is cnf folder?

[pulpbot]

From: @quba42 (quba42)
Date: 2021-08-17T11:16:56Z

---

It looks like Commands-amd64 is a new type of metadata file used by Ubuntu repos, that pulp_deb does not currently support. (This is the first time I have heard of this type of file.)

As such, this ticket is essentially a feature request, and I have reformulated it as such!

Note: I have found the following statement regarding this feature: "Starting with Ubuntu Focal (20.04), the APT client expects APT sources to provide command-not-found (cnf) metadata files."

[pulpbot]

From: ajsween (ajsween)
Date: 2021-08-18T01:55:39Z

---

There are three different meta data types not being sync’d as of Ubuntu 20.04:

• cnf - command-not-found (apt purge command-not-found will remove this)
• i11n - translation files (echo ‘Acquire::Languages "none";’ > /etc/apt.conf.d/99translations)
• appstream - (apt purge appstream will remove but it disables installs from GUI app store)

I included how to work around if you are okay with forgoing apt install recommendations with cnf, translation files with i11n, and App Store installs with appstream.

[pulpbot]

From: @quba42 (quba42)
Date: 2021-08-18T06:47:00Z

---

There is a open issue for translation files: https://pulp.plan.io/issues/8734

I will have a look at the other type you mentioned.

[pulpbot]

From: ajsween (ajsween)
Date: 2021-08-18T11:48:02Z

---

appstream is the “DEP-11” folder in main for Ubuntu Focal Fossa.

[pulpbot]

From: mgoddard (mgoddard)
Date: 2021-10-29T19:51:02Z

---

Also hitting this issue. Is there a timeframe for implementation?

[pulpbot]

From: @quba42 (quba42)
Date: 2021-11-02T08:40:51Z

---

mgoddard wrote:

Also hitting this issue. Is there a timeframe for implementation?

No timeframe as of yet I am afraid. I will throw this ticket into our internal triage and see what happens.

As I understand this is just a Warning, so the workaround is to ignore the issue and carry on without CNF indices.

[itbane]

The issue with language files (i18n) is usually a warning, but is blocking when doing a fresh installation, as the debian/ubuntu installer fails when there are issues with apt update.

A workaround for this is:

1. not use pkgsel in the preseed.cfg
2. add workaround from this post in /target/etc/apt/apt.conf.d/ in late_command
3. run in-target apt update, in-target apt install -y <packages> in late_command

[stale]

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!

[jgrammen-agilitypr]

this is still an issue:

Ign:20 https://hostname/pulp/content/ubuntu_mirror_focal focal/main amd64 Packages
Ign:22 https://hostname/pulp/content/ubuntu_mirror_focal focal/main Translation-en
Ign:23 https://hostname/pulp/content/ubuntu_mirror_focal focal/main amd64 c-n-f Metadata
Get:20 https://hostname/pulp/content/ubuntu_mirror_focal focal/main amd64 Packages [970 kB]
Ign:22 https://hostname/pulp/content/ubuntu_mirror_focal focal/main Translation-en
Ign:23 https://hostname/pulp/content/ubuntu_mirror_focal focal/main amd64 c-n-f Metadata
Ign:22 https://hostname/pulp/content/ubuntu_mirror_focal focal/main Translation-en
Ign:23 https://hostname/pulp/content/ubuntu_mirror_focal focal/main amd64 c-n-f Metadata
Ign:22 https://hostname/pulp/content/ubuntu_mirror_focal focal/main Translation-en
Err:23 https://hostname/pulp/content/ubuntu_mirror_focal focal/main amd64 c-n-f Metadata
  404  Distribution is not pointing to a publication. [IP: 10.2.10.67 443]
Ign:22 https://hostname/pulp/content/ubuntu_mirror_focal focal/main Translation-en
Fetched 3,466 kB in 3s (1,261 kB/s)
Reading package lists... Done
E: Failed to fetch https://hostname/pulp/content/ubuntu_mirror_focal/dists/focal/main/cnf/Commands-amd64  404  Distribution is not pointing to a publication. [IP: 10.2.10.67 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.
d.

This makes the mirror completely un useable since the pulp repo is flagged as failed.
Is there a way to turn off the cnf for a specific repository (just pulp)?
"apt purge command-not-found will remove this", this work around is a little drastic since it disable cnf for all repo's.

pulp os: Ubuntu 20.04
vm with repo : Ubuntu 20.04

Version
Pulpcore version:
3.19
Pulp plugins installed and their versions:
pulp-deb 2.18.0
pulp-file 1.10.2

[stale]

This issue is no longer marked for closure.

[jgrammen-agilitypr]

any chance of some traction on this issue? it would be nice if pulp-deb fully supported the newer versions of ubuntu (>= 20.04).

[quba42]

@jgrammen-agilitypr Not really. This is just not a use case my employer is currently pursuing, and we have a gazillion other issues to work on, so unless there is a community contribution I don't see this feature happening any time soon.