Expand section on packaging compiled extensions :)

[sneakers-the-rat]

Creating an issue to check interest/feasibility/etc. in expanding the section on packaging with compiled extensions!

There are a bunch of moving pieces here, and I don't think we can quite arrive at a single "best practices" because of the amount of variability inherent in the problem, but I think we can get a nice starting point and compile resources for further reading.

Sketching a few of the pieces that would need to be covered:

• CD: building for multiple architectures/versions, releasing from CI/CD
• Config: setting up pyproject.toml and accessory scripts for building
• Tooling: how a sample of package managers handle extension modules
• Frameworks: e.g. Cython, pyo3, etc.
• Integration with other build systems like cmake, meson, etc.
• Typing: type stub files and ux when exposing compiled modules
• Docs: documenting build process, the code itself, and making contribution accessible

[Midnighter]

Definitely an interesting topic. I have a candidate package that can use good guidance, too.

[sneakers-the-rat]

My nefarious plan to slowly convince someone to help me write "rust for python ppl" with pyo3 as a gateway.....

[Midnighter]

I assume you would want to switch to pdm for managing a package with extensions?

[lwasser]

I would love for someone to work on this. @ucodery created some examples iwth hatch of compiled C (i think) in last year's sprint at pycon and it worked. People have asked for this addition if anyone is interested in taking it on! (maybe it could be a team effort?)

[ucodery]

I think some of this is necessary and valuable to the packaging guide. However, I do want to caution how much we attempt to cover. Do we have a guid on the CAPI? General C code? Do we adopt best practices for compiled code layout, and linting, etc for all languages used in conjunction with Python?

I like our mantra of making opinionated decisions about Python packaging. But does that mean we choose build tools that work with all compiled languages, so we just pick one, or do we have a bit of TMO as there is often a different best choice depending on the extension? (e.g. meson will work with Rust code and PyO3 (I think), but for projects that are just py+rs maturin will probably be a better choice).

Hatch can definitely make package extensions. Its harder using Hatchling, and I didn't--I used hatch+meson.

[lwasser]

Good point! You know more about the complexity than I do. Could the guide possibly provide an overview as you begin to lay out above in terms of what tools might be better for which applications? And we could link to resources. Then have examples for more common (if that exists) applications?

[ucodery]

It seems inevitable that PyOS eventually gets at least a C-extension guide. But I wonder if this is another topic that is best to direct readers to Scientific Python? IIRC we link to them for guides to non-hatch projects and a big part of compiling extension modules comes down to switching your build-backend and following that tool's best practices. They already cover multiple compiled build-backends, including all we are discussing here, in addition to guides on several pure-python backends.