Proposal: New arguments to control pre-releases

[notatallshaw]

Summary

This proposal introduces two new flags to pip:

• --all-releases <format_control>: Allows installing all versions of a package, including pre-releases.
• --only-final <format_control>: Restricts installation to final and post-releases only.

These flags provide fine-grained control over release selection, following similar syntax and semantics of --no-binary and --only-binary.

Motivation

Currently, pip's handling of pre-releases is controlled globally via --pre, but there's no package-specific control and no way to specify disabling pre-releases.

Further, pip has historically not selected prereleases in places where the spec says it should due to bugs in packaging (pypa/packaging#872 and pypa/packaging#794) so users haven't had as much need to "turn off" prereleases as they might in the future.

Specification

--all-releases <format_control>

Allows installation of all versions of a package, including pre-releases. Can be supplied multiple times, and each time adds to the existing value.

• Accepts :all: to allow pre-releases for all packages
• Accepts :none: to empty the set
• Accepts a package name to enable pre-releases for specific packages

Example Usage:

pip install --all-releases foo
pip install --all-releases foo --all-releases bar
pip install --all-releases :all:
pip install --all-releases :none:

--only-final <format_control>

Restricts installation to final and post-releases only. Can be supplied multiple times, and each time adds to the existing value.

• Accepts :all: to enforce only final releases for all packages
• Accepts :none: to empty the set
• Accepts a package name to enforce only final releases for specific packages

Example Usage:

pip install --only-final foo
pip install --only-final foo --only-final bar
pip install --only-final :all:
pip install --only-final :none:

Design Notes

• --pre will be the equivalent of --all-releases :all:, and could eventually be deprecated
• Like --no-binary and --only-binary more specific things override less specific things, e,g, --all-releases :all: --only-final foo does not allow pre-releases for foo.
• I have not included the :none: syntax from --no-binary and --only-binary as I don't really understand the use case, but if there is a strong use case for it I can add it
• I have not included comma separated package names in the CLI as this seems like an unneeded complexity, but I haven't fully thought through how this would work with environmental variables, so I might come back to including this

Backward Compatibility

This change is fully backward-compatible, as it introduces new optional flags without altering existing behavior.

Implementation

If there's no significant objection I'm volunteering to implement this.

However this is not being funded by anyone so work will go at the pace that I have free time, if someone else is willing to fund or work on this please let me know and we can organize something.

Bikeshedding

I'm not strongly attached to the names here, I'm happy to accept other names, I'm more interested if there are objections to the design.

[notatallshaw]

An existing workaround

Granular enabling of pre-releases is technically already possible by creating a constraints file and adding:

foo>=0.0dev

However, as far as I'm aware, it is not an intended use case for constraints, and requires creating a file (can't fully be specified by either CLI or Env Vars).

[pfmoore]

In general, I'm in favour of this, as I think that giving users finer control in the (exceptional, hopefully!) cases where they need it is a good thing. But I'm a little concerned that pip's UI is already complex, and this adds more complexity. Unfortunately, I don't know enough about UI design to have a better idea of how to expose this functionality (I thought of having a "package selection mini-language" covering this and the various --{prefer,only,no}-binary options, but that's probably more complicated than a bunch of options).

I don't want the perfect to be the enemy of the good, though, so I'm fine with this proposal as it stands.

[potiuk]

I like the proposal and we would find it very useful in Airflow.

Small suggestion maybe - to do a bit more fine-grained behaviour and consistency with https://packaging.python.org/en/latest/specifications/version-specifiers/#version-specifiers:

• --developmental-releases
• --no-developmental-releases
• --pre-releases
• --no-pre-releases
• --only-final-releases

That's a suggestion only and it would also cover the developmental case which I think currently has no separate flag. Currently --pre covers both pre-releases and developmental releases of course, but maybe with this change we could add a bit more fine-grained behaviour.

Also maybe we could update the docs of pip while implementing it - some of the docs says stable releases where version specification uses final. Though stable is also used in one case in the specs :).

[notatallshaw]

Small suggestion maybe - to do a bit more fine-grained behaviour and consistency with https://packaging.python.org/en/latest/specifications/version-specifiers/#version-specifiers

Before considering distinguishing dev-releases and pre-releases it needs to be understood, does the spec consider dev-releases to be a type of pre-release, I pose this question from reading the spec yesterday over on packaging: pypa/packaging#875 (comment).

That's a suggestion only and it would also cover the developmental case which I think currently has no separate flag. Currently --pre covers both pre-releases and developmental releases of course, but maybe with this change we could add a bit more fine-grained behaviour.

If we are distinguishing pre-releases from dev-releases this opens a bunch of questions I don't think the spec answers, to implement this I would be specifically working off "Handling of pre-releases" which does not distinguish pre-releases from dev-releases:

Pre-releases of any kind, including developmental releases

And furthermore I would be leveraging Packaging's SpecifierSet to correctly implement the spec, it accepts the pre-release value in it's __init__ that would be used for this. It does not distinguish dev releases and it also doesn't even currently follow the spec for pre-release, this proposal is partly motivated by assuming the fix will land at some point.

And these kind of bugs would make me very wary of implementing the logic myself, if we distinguish dev-releases from pre-releases (and we don't distinguish final releases from post releases) there are 4 types of releases:

1. Final releases
2. Pre-releases
3. Dev-releases
4. Dev-Pre-releases

So when you say --pre-releases what combination of these 4 do you get? What combination do you get for --developmental-releases?

I think there are multiple answers that could be given, and I'm not personally motivated to solve that problem in this proposal, and I think it needs to be clear on the packaging side first, as in the original linked issue is_prerelease returns true for a dev-release in packaging.

In general I do like the idea of giving users more control here, but I don't want to try implementing this on pip side when there are open questions about what do these terms even mean. I'm only now confident I have a good grasp on the "Handling of pre-releases" section of the spec.

Also maybe we could update the docs of pip while implementing it - some of the docs says stable releases where version specification uses final. Though stable is also used in one case in the specs :).

I see no reason to wait on fixing that. PRs welcome from anyone for docs change, otherwise I will try to take a look myself at some point.

[pfmoore]

IMO the current spec is clear that dev releases are pre-releases, as per the comment @notatallshaw quoted:

Pre-releases of any kind, including developmental releases

Treating dev releases differently than pre-releases would require a spec change, and therefore a PEP. I am strongly against pip implementing something like this without the backing of the spec, as (a) we use packaging, and they follow the spec, so we'd have to implement our own custom logic on top of packaging, and (b) the various edge cases and interactions would be very complex, and I'm confident that different people would come up with different answers - having a spec to arbitrate would avoid putting that burden on the pip maintainers.

Frankly we've had enough trouble just pinning down the semantics of what a pre-release means. I have no appetite for adding more complexity (although if someone decides to write a PEP, I'll inevitably get sucked into the debate 🙁).

[potiuk]

So when you say --pre-releases what combination of these 4 do you get? What combination do you get for --developmental-releases?

Some time ago I've been caught by suprise that pre-release and dev-releae are very different things:

Public version identifiers are separated into up to five segments:

• Epoch segment: N!
• Release segment: N(.N)*
• Pre-release segment: {a|b|rc}N
• Post-release segment: .postN
• Development release segment: .devN

Which means that 1.0.1.rc1.dev0 is a thing - and it is a developmental version of rc1 pre-release :). Similarly 1.0.1.dev0 is a developmental version of .... final relaase. And it's not a pre-release IMHO.

So I'd say reasonable thing would be

• --pre-relase - includes pre-release versions (but not necessarily developmental release)
• --developmental-release - includes developmental releases (but not necessarily pre-release)

So:

• --pre-release x would match x.1.0.1.rc1, but not x.1.0.1.dev0 nor x.1.0.1.rc1.dev0
• --developmental-release x would match x.1.0.1.dev0 but not x.1.0.1.rc1.dev0
• --pre-release x --developmental-release x would match both x.1.0.1.dev0 and x.1.0.1.rc1.dev0

And while a little chatty, it mkes perfect sense and is consistent.

[potiuk]

But yeah. there is ambiguity in the specs, so you can understand it differently, I agree. I would be fine with either choice you make.

[pfmoore]

I would be fine with either choice you make.

My point is that we shouldn't be making a choice. Nor, for that matter, should the packaging library. If the spec is ambiguous, we should fix the spec. If no-one has the appetite to do that, then IMO, pip's position is that "we use packaging, so we follow their interpretation of the spec".

And more fundamentally, I'm a strong -1 on extending this issue beyond the options in the original post. And this discussion has changed my view a little on that - I now think that even what's in the original post is risky, as the specification only allows user control in the form of the user explicitly requesting pre-releases to be included, so --only-final could easily have undesirable effects, in terms of either giving non-compliant behaviour, or not doing what the user expects (depending on how we decide to implement it).

So I'm now of the view that --all-releases <format_control> is a reasonable option to add. I'm not comfortable with --only-final <format_control> unless its effect can be clearly described in a spec-compliant manner¹. And I'm against any options beyond this.

All of this subject to change if a change to the standard gets proposed and approved, of course.

Footnotes

1. One spec-compliant way of defining --only-final would be that it removes all versions that contain .aN, .bN, .rcN or .devN from the list of available versions whenever pip would see them. So they are completely invisible to pip. That could still have unexpected consequences, for example if there's a non-final version is already installed (as not seeing installed versions is weird), but it would nevertheless be spec-compliant. ↩

[notatallshaw]

as the specification only allows user control in the form of the user explicitly requesting pre-releases to be included

Actually the spec says that tools SHOULD allow users to exclude pre-releases:

Dependency resolution tools SHOULD also allow users to request the following alternative behaviours:

• accepting pre-releases for all version specifiers
• excluding pre-releases for all version specifiers (reporting an error or warning if a pre-release is already installed locally, or if a pre-release is the only way to satisfy a particular specifier)

The --only-final part of this proposal implements the second bullet point that the spec says tools SHOULD implement and pip currently doesn't offer anything for.

Further I do think that tools have flexibility here to filter versions based on things that aren't defined in the spec, my main issue with tackling dev-releases in this proposal is that I do think there is ambiguity in the spec (though fortunatly not in the section "Handling of pre-releases") because if you just read:

1. https://packaging.python.org/en/latest/specifications/version-specifiers/#pre-releases
2. https://packaging.python.org/en/latest/specifications/version-specifiers/#developmental-releases

You would definitely come away thinking that dev-releases are not pre-releases, but then other parts of the spec state or imply that dev-releases are pre-releases. I do think the spec needs clarity here as I say in pypa/packaging#875 (comment), but I'm not motivated right now to seek that clarity.

For pip though, I think it should defer to packaging, and rely on packaging to follow the spec, file issues and PRs where we don't think it is, and leverage whatever packaging has to offer.

[pfmoore]

The --only-final part of this proposal implements the second bullet point that the spec says tools SHOULD implement and pip currently doesn't offer anything for.

Apologies. I missed that part of the spec.

Further I do think that tools have flexibility here to filter versions based on things that aren't defined in the spec

I agree, although I think that because of the way the spec makes statements about including (or not) pre-releases, it's hard to reconcile such flexibility with the things the spec requires. IMO it would have been better if the spec had confined itself to defining ordering, and avoided all of the complexity around pre-releases. Unfortunately that would have been a significant break with user expectations (from what I recall of the discussions at the time) and is now a backward incompatible change with non-trivial implications (because version specifiers are included in dependency metadata which can therefore change meaning if pre-release handling changes).

I do think the spec needs clarity here as I say in pypa/packaging#875 (comment), but I'm not motivated right now to seek that clarity.

Again, I agree, but my most recent attempt to rationalise things didn't work, because it simplified the pre-release handling too much, and so wasn't compatible with the current spec. And I have zero motivation for trying to make an incompatible change to the spec - I have no good answers to all the "but what if...?" questions that would inevitably come up.

IMO, the biggest issue with the spec is the "Handling of pre-releases" section. I think¹ that the rest of the spec is (fairly) self-consistent and clear. But too much of that one section just muddies the water. Ideally, we'd just remove it. But someone would have to confirm that doing so didn't cause significant problems (and I have a depressing feeling that it would).

Footnotes

1. I don't have the time or inclination right now to re-read the spec confirming this, but I'm pretty sure the last time I did, that's the conclusion I came to. ↩

[notatallshaw]

Again, I agree, but my most recent attempt to rationalise things didn't work, because it simplified the pre-release handling too much, and so wasn't compatible with the current spec. And I have zero motivation for trying to make an incompatible change to the spec - I have no good answers to all the "but what if...?" questions that would inevitably come up.

Noted, I will have a crack at this after my attempt to formalize the semantics of Requires Python.

I am reading some literature, and doing some experiments, on solvability and resolution algorithms (e.g. PubGrub), and spending some time to see if Python's current version semantics can actually fit within their constraints (rather than making simplifying assumptions like uv does).

IMO, the biggest issue with the spec is the "Handling of pre-releases" section. I think that the rest of the spec is (fairly) self-consistent and clear. But too much of that one section just muddies the water. Ideally, we'd just remove it. But someone would have to confirm that doing so didn't cause significant problems (and I have a depressing feeling that it would).

I do think this would cause significant disruption. It might be possible to rewrite this section without breaking the existing spec, by carefully defining a couple of operators that do quite closely fit standard mathematical operators, and then carefully defining the remaining operators in terms of the well defined operators.

But I don't plan to have something ready soon, and like you, I think this propsal is best to carefully follow the langauge and suggestions of the current spec.

[notatallshaw]

On the side question of "Are Developmental releases a type of pre-release?": https://discuss.python.org/t/are-developmental-releases-a-type-of-pre-release/81666