debian packages: new versions of apt report "Malformed OpenPGP message"

[Infinoid]

Description

New versions of apt in debian testing (trixie) and debian unstable (sid) are unable to read package lists from a gitea debian package repo.

I uploaded an arm64 package to the demo site. I followed the instructions to add the repo to an arm64 machine running debian trixie (testing), but it reports an error:

$ sudo apt update
Get:1 http://http.us.debian.org/debian trixie InRelease [175 kB]
Get:2 http://http.us.debian.org/debian trixie/main Sources [10.3 MB]                    
Get:3 https://demo.gitea.com/api/packages/infinoid/debian kernel InRelease [2,010 B]    
Err:3 https://demo.gitea.com/api/packages/infinoid/debian kernel InRelease              
  Sub-process /usr/bin/sqv returned an error code (1), error message is: Error: Malformed Message: Malformed OpenPGP message
Get:4 http://http.us.debian.org/debian trixie/main arm64 Packages [9,340 kB]
Get:5 http://http.us.debian.org/debian trixie/main Translation-en [6,308 kB]
Warning: GPG error: https://demo.gitea.com/api/packages/infinoid/debian kernel InRelease: Sub-process /usr/bin/sqv returned an error code (1), error message is: Error: Malformed Message: Malformed OpenPGP message
Error: The repository 'https://demo.gitea.com/api/packages/infinoid/debian kernel InRelease' is not signed.
Notice: Updating from such a repository can't be done securely, and is therefore disabled by default.
Notice: See apt-secure(8) manpage for repository creation and user configuration details.

This is not arm64-specific, it happens on x86-64 too. That just happens to be where I first saw the issue.

Apt recently switched to a new implementation of PGP, called Sequoia, for signature verification. It's written in rust and has stricter parsing. This is in the apt v2.9.19 changelog:

• Replace GnuPG with Sequoia on supported Debian platforms
  • methods: Add new sqv method
  • debian: Add default policy to allow SHA-1 self-signatures until 2026
  • debian: Plug sqv into the package build

Debian has packaged several of Sequoia's command line tools, including sqv and sq. sq looks useful for getting more info from the parsing process. Here's what it thinks of gitea's generated signatures, and a main Debian mirror:

$ curl -s https://demo.gitea.com/api/packages/infinoid/debian/dists/kernel/InRelease | sq packet dump
Unknown or Unsupported Packet, new CTB, 284 bytes
    Tag: Signature Packet
    Error: Malformed MPI: leading bit is not set: expected bit 8 to be set in  1011001 (59)
  
$ curl -s http://http.us.debian.org/debian/dists/trixie/InRelease | sq packet dump
Signature Packet, old CTB, 563 bytes
    Version: 4
    Type: Text
    Pk algo: RSA
    Hash algo: SHA256
    Hashed area:
      Issuer Fingerprint: A7236886F3CCCAAD148A27F80E98404D386FA1D9
      Signature creation time: 2025-01-25 20:24:46 UTC
    Unhashed area:
      Issuer: 0E98404D386FA1D9
    Digest prefix: 2771
    Level: 0 (signature over data)
  
Signature Packet, old CTB, 563 bytes
    Version: 4
    Type: Text
    Pk algo: RSA
    Hash algo: SHA256
    Hashed area:
      Issuer Fingerprint: 4CB50190207B4758A3F73A796ED0E7B82643E131
      Signature creation time: 2025-01-25 20:24:48 UTC
    Unhashed area:
      Issuer: 6ED0E7B82643E131
    Digest prefix: A44F
    Level: 0 (signature over data)
  
$ 

(For reference, the files retrieved by curl are in this gist.)

Debian testing (trixie) and unstable (sid) both include this change, and are unable to retrieve package lists from gitea. Stable (bookworm) is not affected.

Gitea Version

1.22.6, 1.23.1, 1.24.0+dev-217-g06ff9b6256

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

linux

How are you running Gitea?

docker

Database

PostgreSQL

[Infinoid]

I tried to make a standalone test program to reproduce this signature problem, but the results are confusing.

It looks like my test program is sensitive to the go-crypto dependency:

• keybase/go-crypto (used in gitea v1.22 and v1.23) fails
• ProtonMail/go-crypto (used in main branch and the demo site) works

But the demo site also failed, so this doesn't quite match the problem I saw above.

Here's what I have (gist). It compares signature verification results for gpg and sequoia. It was run on debian trixie with the sopv-gpgv, sq and sqopv packages installed.

The ProtonMail build is able to be verified by both gpg and sequoia, and sq packet dump dumps the signature contents correctly.
The keybase build is able to be verified by gpg, but sequoia's parser chokes on it.

If that was the only problem, the demo site should have worked, because the demo site was built using the ProtonMail version. So maybe there's more to it?

[wxiaoguang]

I think the problem is that keybase/go-crypto is also used in main for the "debian package".

Maybe you could try to switch the debian package implementation to ProtonMail/go-crypto in main branch to try?

[Infinoid]

Oh, you're right, the main branch is using both versions. I saw the new one in go.mod and assumed that the old one was completely replaced.

That makes more sense, then. The keybase fork of go-crypto is no longer maintained (repo is archived) and it generates malformed signatures. I suggest switching it.

I don't know how well I can test this locally, but I'll poke at it.

[wxiaoguang]

I managed to propose a PR: Use ProtonMail/go-crypto to replace keybase/go-crypto #33402

[nwalfield]

See also https://github.com/orgs/community/discussions/27607 for some additional discussion of this issue.

[Infinoid]

I noticed (and the discussion linked above also notes) that this issue is intermittent. So I grabbed a random .deb package from the cache folder and ran this test loop:

for SEQ in `seq -w 1 1000`; do
    # delete package
    curl -s --user test:testtest -X DELETE http://localhost:3000/api/v1/packages/test/debian/zip/3.0-14 || true
    # upload package
    curl -s --user test:testtest --upload-file zip_3.0-14_amd64.deb http://localhost:3000/api/packages/test/debian/pool/test/test/upload
    # test signature
    curl -s http://localhost:3000/api/packages/test/debian/dists/test/InRelease | sq packet dump
done 2>&1 | grep Malformed | wc -l

It failed 542 times out of 1000.

I patched services/packages/debian/repository.go in the same way your PR does (I haven't tried the rest of your PR), and ran the test loop again.

It failed 0 times out of 1000.

[wxiaoguang]

So I think we can mark #33402 as "fix this issue" now