Drop XMLRPC API to use PyPI Index API

[pombredanne]

The XMLRPC is now obsolete and deprecated and even the parts that may
not be deprecated no longer work. Instead, let's use the simple Index
API with JSON to collect all packages and their changes.

Reference: #1897
Signed-off-by: Philippe Ombredanne [email protected]

[codecov]

Codecov Report

Attention: Patch coverage is 88.23529% with 4 lines in your changes missing coverage. Please review.

Project coverage is 86.43%. Comparing base (4d020e8) to head (262a137).
Report is 244 commits behind head on main.

Files with missing lines	Patch %	Lines
src/bandersnatch/master.py	80.95%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1898      +/-   ##
==========================================
+ Coverage   79.69%   86.43%   +6.74%     
==========================================
  Files          31       35       +4     
  Lines        4324     4438     +114     
  Branches      780      463     -317     
==========================================
+ Hits         3446     3836     +390     
+ Misses        721      425     -296     
- Partials      157      177      +20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cooperlees]

Many thanks! This is great and much cleaner / simpler to implement that I thought.

• My only worry is to see how long PyPI caches these pages ... cc: @ewdurbin - Where do we find this setting?
  • We should document that as XMLRPC had 0 caching ...
  • But maybe this is low as we've not had many problems fetching packages supplied via the XMLRPC to mirror/sync ...

I'm also sad I forgot this was a stretch goal that we put into a PEP I co-authored. This was all clever additions by @dstufft <3.

Can we just fix typing, add info on the Simple API caching (if there is any) and that test name unless 1__1 means something I don't understand, but we don't have it anywhere else I think ...

[cooperlees]

O nice catch. I need to go through and see if we can slim down our requirements file.

Or maybe we move to uv to auto manage this stuff. Wonder if PyPA has any policy on using uv yet ...

[pombredanne]

Actually, I just brutally looked for "rpc" and this looked unused. Does uv check for imports?

[cooperlees]

The speed increase is amazing too based on the integration tests - >10s saved :

• XMLRPC - https://github.com/pypa/bandersnatch/actions/runs/13679190278/job/38246889516
  • 27s
• SimpleJSON - https://github.com/pypa/bandersnatch/actions/runs/13686864483/job/38272166947?pr=1898
  • 10s

[ewdurbin]

• My only worry is to see how long PyPI caches these pages ... cc: @ewdurbin - Where do we find this setting?

Caching for endpoints is configured in their view configurations, as "orgin_cache": For the /simple/ route that is here: https://github.com/pypi/warehouse/blob/ef9978cdf279bc8cf953eec7092cc2846de6b982/warehouse/api/simple.py#L69-L73

That endpoint is one of the only ones we don't purge in response to changes, due to its poor performance on cold-cache. Most relevant endpoints are purged in more-or-less real-time on upload, such as /simple/bandersnatch/ and /pypi/bandersnatch/json.