Add workflow to test against bug fix branch

[pllim]

Sometimes astropy bugfix release somehow includes breaking changes, making downstream maintainers cry. Would be nice if we run integration testing against backport branch or RC before pushing out a bugfix release. cc @astropy/astropy-project-release-team

Current outcome: Some crying when picking up new bugfix version of astropy.

Desired outcome: No crying.

xref astropy/astropy#12889 (comment)

[astrofrog]

We could also always potentially have a workflow in astropy core that only gets run on releases branches, not main.

[Cadair]

I think having this as a workflow on core is going to be the only sane way to integrate it with any bugfix automation.

I think the downstream workflows from asdf and gwcs are good examples of how to do this.

[astrofrog]

We could even make it so that it would get run if the extra ci label is added, and in weekly cron

[pllim]

astropy has too many downstreams and I monitor its CI. I am afraid putting this workflow in core is going to give me a lot of churns.

[Cadair]

I don't see how we can automate running downstream tests on automated bugfixes while not having that workflow in core.

I am in favour of having the downstream run be almost explicitly opt-in on core apart from on release branches. For most cases we should be relying on downstreams doing dev testing not the other way around.

[astrofrog]

@pllim - to be clear I personally think if we added a workflow to astropy core we should restrict ourselves to downstreams that we maintain in the astropy org, not all downstreams! (not even all affiliated packages) And yes, that includes not sunpy 😅 Only things we have control over in astropy org.

[pllim]

For most cases we should be relying on downstreams doing dev testing not the other way around.

Won't that be nice.

Is it possible to have bugfix dev wheel in the same place as nightly dev wheel? I can see how the version matching could be confusing... But if we can pull it off, I can batch issue to ask downstream packages to use it in their CI.

There might be a way to have automated bug fix release check an external workflow (have to research the details again, my memory is vague now). I was thinking this workflow only runs once in a while or right before release.

[Cadair]

Won't that be nice.

To be clear, I mean this is true for testing main.

I agree with @astrofrog, we should be checking things we maintain against releases before we release.