-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix: update release workflow to follow pypa and pypi guidelines #73
base: main
Are you sure you want to change the base?
Changes from all commits
81ed956
f7ae35d
878094c
cf61545
aa4c259
ddc5597
48f3cc2
375b29e
471236b
00884d7
ad839c5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# pyOpenSci packaging template Changelog | ||
|
||
## [Unreleased] | ||
|
||
* Update release workflow to follow PyPA / PyPI recommended practices (@lwasser, #48) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# Contributing to the pyOpenSci Python package template | ||
|
||
To work on the template locally, you can call the copier template directly. Note that by default, `copier` uses the latest tag in your commit history. To ensure it uses the latest commit on your current active branch use: | ||
|
||
`copier copy -r HEAD /path/to/your/template destination-dir` | ||
|
||
If you want to test it against the latest tag in your local commit history, you can use: | ||
|
||
`copier copy /path/to/your/template destination-dir` | ||
|
||
## Run the tests | ||
|
||
You can use Hatch to run all of the tests for the template: | ||
|
||
`hatch run test:run` |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -77,7 +77,7 @@ documentation: | |
type: str | ||
help: "Do you want to include documentation for your project and which framework do you want to use?" | ||
choices: | ||
"Sphinx (https://www.pyopensci.org/pyos-sphinx-theme)": sphinx | ||
"Sphinx (https://pydata-sphinx-theme.readthedocs.io/en/stable/index.html)": sphinx | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is just a tiny change - pyos-sphinx-theme is specific to our guidebooks. so let's go with the community lead pydata-sphinx-theme instead 🚀 :) |
||
"mkdocs-material (https://squidfunk.github.io/mkdocs-material)": mkdocs | ||
No: "" | ||
default: "{% if use_default != 'minimal' %}sphinx{% else %}{% endif %}" | ||
|
@@ -111,7 +111,7 @@ license: | |
type: str | ||
help: | | ||
Which license do you want to use? Includes a LICENSE file in the repository root. | ||
For more information, see: | ||
For more information, see: | ||
- https://www.pyopensci.org/python-package-guide/documentation/repository-files/license-files.html | ||
- https://opensource.org/licenses | ||
choices: | ||
|
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -1,53 +1,77 @@ | ||||||||||||||||||||||||||||||||||
name: CD | ||||||||||||||||||||||||||||||||||
name: Publish to PyPI | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
on: | ||||||||||||||||||||||||||||||||||
push: | ||||||||||||||||||||||||||||||||||
tags: | ||||||||||||||||||||||||||||||||||
- 'v?[0-9]+.[0-9]+.[0-9]+' | ||||||||||||||||||||||||||||||||||
- 'v?[0-9]+.[0-9]+.[0-9]+(a|b|rc|post|dev)[0-9]+' | ||||||||||||||||||||||||||||||||||
release: | ||||||||||||||||||||||||||||||||||
types: [published] | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
jobs: | ||||||||||||||||||||||||||||||||||
prerequisites: | ||||||||||||||||||||||||||||||||||
uses: ./.github/workflows/test.yml | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
release: | ||||||||||||||||||||||||||||||||||
# Setup build separate from publish for added security | ||||||||||||||||||||||||||||||||||
# See https://github.com/pypa/gh-action-pypi-publish/issues/217#issuecomment-1965727093 | ||||||||||||||||||||||||||||||||||
build: | ||||||||||||||||||||||||||||||||||
needs: [prerequisites] | ||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @Midnighter i added the test run back -- here before we build the package. |
||||||||||||||||||||||||||||||||||
strategy: | ||||||||||||||||||||||||||||||||||
matrix: | ||||||||||||||||||||||||||||||||||
os: [ubuntu-latest] | ||||||||||||||||||||||||||||||||||
python-version: ["3.12"] | ||||||||||||||||||||||||||||||||||
runs-on: ${{ matrix.os }} | ||||||||||||||||||||||||||||||||||
permissions: | ||||||||||||||||||||||||||||||||||
# Write permissions are needed to create OIDC tokens. | ||||||||||||||||||||||||||||||||||
id-token: write | ||||||||||||||||||||||||||||||||||
# Write permissions are needed to make GitHub releases. | ||||||||||||||||||||||||||||||||||
contents: write | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
runs-on: ubuntu-latest | ||||||||||||||||||||||||||||||||||
# Environment is encouraged for increased security | ||||||||||||||||||||||||||||||||||
environment: build | ||||||||||||||||||||||||||||||||||
Comment on lines
+15
to
+16
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a new one for me, so I'm curious. Could you please share how it works? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh! @agriyakhetarpal environments allow you to setup a trusted connection with pypi. This is the build environment, so we don't need to worry too much about security here in terms of PYPI. so anyone can run the build step in theory. but by having a publish environment, we can control who can trigger that step of the build. (if that is what you're asking) . here is a bit more about this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, I'm aware of trusted publishing and environments, thanks! My question was more around the need for this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh gosh my apologies. i suspected I was telling you something you already knew. 🙈 This is a great point. I don't know the answer to your question!! But i suspect this is ME introducing complexity. i just checked the tutorials and there is a build step but not an additional environment. Shall we remove this environment as you are correct it's not required. we need the job but not an environment AFAIK . There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
No worries at all! 😄
Thank you for dropping it! |
||||||||||||||||||||||||||||||||||
steps: | ||||||||||||||||||||||||||||||||||
- uses: actions/checkout@v4 | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
- name: Set up Python ${{ matrix.python-version }} | ||||||||||||||||||||||||||||||||||
uses: actions/setup-python@v5 | ||||||||||||||||||||||||||||||||||
with: | ||||||||||||||||||||||||||||||||||
python-version: ${{ matrix.python-version }} | ||||||||||||||||||||||||||||||||||
- name: Checkout | ||||||||||||||||||||||||||||||||||
uses: actions/checkout@v4 | ||||||||||||||||||||||||||||||||||
with: | ||||||||||||||||||||||||||||||||||
# This fetch element is only important if you are use SCM based | ||||||||||||||||||||||||||||||||||
# versioning (that looks at git tags to gather the version) | ||||||||||||||||||||||||||||||||||
fetch-depth: 100 | ||||||||||||||||||||||||||||||||||
# Need the tags so that setuptools-scm can form a valid version number | ||||||||||||||||||||||||||||||||||
- name: Fetch git tags | ||||||||||||||||||||||||||||||||||
run: git fetch origin 'refs/tags/*:refs/tags/*' | ||||||||||||||||||||||||||||||||||
Comment on lines
+18
to
+26
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This can be simplified with the
Suggested change
I have to note that this isn't working with There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ahhhh ok perhaps we hold off if this is bleeding edge and not vetted across platforms. As we do want our users to have a happy experience creating their first package! It's more important that this template is easy to use and understand! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, it's not an entirely new feature (it calls a variant of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. cool! Thanks, @agriyakhetarpal. What do you think about us moving this to an issue and applying when the bugs have been ironed out on Linux/Ubuntu? All of your comments are awesome BTW - THANK YOU for taking the time to review. I'm really excited to teach with this template in the spring! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sounds good to me! I'll remember to open an issue that links back to this review thread, once this PR is merged. The pleasure is all mine! |
||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
- name: Install hatch | ||||||||||||||||||||||||||||||||||
uses: pypa/hatch@install | ||||||||||||||||||||||||||||||||||
- name: Setup Python | ||||||||||||||||||||||||||||||||||
uses: actions/setup-python@v5 | ||||||||||||||||||||||||||||||||||
with: | ||||||||||||||||||||||||||||||||||
# You can modify what version of Python you want to use for your release | ||||||||||||||||||||||||||||||||||
python-version: "3.11" | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
- name: Build package | ||||||||||||||||||||||||||||||||||
run: hatch build | ||||||||||||||||||||||||||||||||||
# Security recommends we should pin deps. Should we pin the workflow version? | ||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, please. I'll recommend that all GitHub Actions—whether by GitHub as an actor or external ones available on the marketplace—should be pinned to their commit hashes, at least for a workflow concerning a release. There is a case to be made about how it makes things a bit more challenging for new contributors/packagers, but I think supply-chain security shouldn't be at the end of a compromise. There are tools such as There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. great. So, looking at the action here Ofek suggests that too - we'd use this - this hash is almost the most recent but the most recent is just a readme update.
We should do that as part of this repo's maintenance, but also, we'll want to tell users to update this periodically, right? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Updating it weekly for our maintenance could be a good cadence; what do you think? It would depend on the amount of activity we have on this template. If we were to ask our users to enable Dependabot in the GitHub settings and inherit the same settings, they should also get the same updates. However, I'm not sure whether it's too much to ask for beginners (😕). As long as it is pinned for a start, we can proceed with a small guide on how to keep this workflow maintained, where we can describe this more (or link to an external guide if that's alright and vetted by us?). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @agriyakhetarpal this is awesome. We, I think, have Depdendabot set up in the org, I think, but I didn't configure it. This is a basic question but would it help update things like this version? IE could we use it in this repo? My gut tells me we won't have the bandwidth to perform weekly updates here unless someone wants to take that one! But perhaps monthly or quarterly would work. I am not sure what is best, however and would love suggestions. also if you were interested in helping us maintain here, that would be awesome (only if you have bandwidth!) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Oh, I notice that here are a few moving parts, here. As this file is not a workflow of its own but a template that will be generated and moved into That means that we have two possible scenarios:
The first one is surely not ideal, so we should explore if Dependabot will choose to be the star here or not 😉
I agree with you. Monthly updates would be nicer – and users can always bump it to daily, or weekly, and so on, based on their bandwidth.
I'd love to! I've also helped out with the development and maintenance of especially other |
||||||||||||||||||||||||||||||||||
- name: Install hatch | ||||||||||||||||||||||||||||||||||
uses: pypa/hatch@a3c83ab3d481fbc2dc91dd0088628817488dd1d5 | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
# We rely on a trusted publisher configuration being present on PyPI, | ||||||||||||||||||||||||||||||||||
# see https://docs.pypi.org/trusted-publishers/. | ||||||||||||||||||||||||||||||||||
- name: Publish to PyPI | ||||||||||||||||||||||||||||||||||
uses: pypa/gh-action-pypi-publish@release/v1 | ||||||||||||||||||||||||||||||||||
- name: Build package using Hatch | ||||||||||||||||||||||||||||||||||
run: | | ||||||||||||||||||||||||||||||||||
hatch build | ||||||||||||||||||||||||||||||||||
echo "" | ||||||||||||||||||||||||||||||||||
echo "Generated files:" | ||||||||||||||||||||||||||||||||||
ls -lh dist/ | ||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||
- name: GH release | ||||||||||||||||||||||||||||||||||
uses: softprops/action-gh-release@v2 | ||||||||||||||||||||||||||||||||||
with: | ||||||||||||||||||||||||||||||||||
body: > | ||||||||||||||||||||||||||||||||||
Please see | ||||||||||||||||||||||||||||||||||
https://github.com/${{ github.repository }}/blob/${{ github.ref_name }}/CHANGELOG.md | ||||||||||||||||||||||||||||||||||
for the full release notes. | ||||||||||||||||||||||||||||||||||
draft: false | ||||||||||||||||||||||||||||||||||
prerelease: false | ||||||||||||||||||||||||||||||||||
# Store an artifact of the build to use in the publish step below | ||||||||||||||||||||||||||||||||||
- name: Store the distribution packages | ||||||||||||||||||||||||||||||||||
uses: actions/upload-artifact@v4 | ||||||||||||||||||||||||||||||||||
with: | ||||||||||||||||||||||||||||||||||
name: python-package-distributions | ||||||||||||||||||||||||||||||||||
path: dist/ | ||||||||||||||||||||||||||||||||||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||||||||||||||||||||
if-no-files-found: error | ||||||||||||||||||||||||||||||||||
publish: | ||||||||||||||||||||||||||||||||||
name: >- | ||||||||||||||||||||||||||||||||||
Publish Python 🐍 distribution 📦 to PyPI | ||||||||||||||||||||||||||||||||||
# Modify the repo name below to be your project's repo name. | ||||||||||||||||||||||||||||||||||
if: github.repository_owner == "{{ username }}" | ||||||||||||||||||||||||||||||||||
needs: | ||||||||||||||||||||||||||||||||||
- build | ||||||||||||||||||||||||||||||||||
runs-on: ubuntu-latest | ||||||||||||||||||||||||||||||||||
# Environment required here for trusted publisher | ||||||||||||||||||||||||||||||||||
environment: | ||||||||||||||||||||||||||||||||||
name: pypi | ||||||||||||||||||||||||||||||||||
# Modify the url to be the name of your package | ||||||||||||||||||||||||||||||||||
url: https://pypi.org/p/${{ package_name }} | ||||||||||||||||||||||||||||||||||
permissions: | ||||||||||||||||||||||||||||||||||
id-token: write # this permission is mandatory for PyPI publishing | ||||||||||||||||||||||||||||||||||
steps: | ||||||||||||||||||||||||||||||||||
- name: Download dists | ||||||||||||||||||||||||||||||||||
uses: actions/download-artifact@v4 | ||||||||||||||||||||||||||||||||||
with: | ||||||||||||||||||||||||||||||||||
name: python-package-distributions | ||||||||||||||||||||||||||||||||||
path: dist/ | ||||||||||||||||||||||||||||||||||
lwasser marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||||||||||||||||||||
merge-multiple: true | ||||||||||||||||||||||||||||||||||
- name: Publish package to PyPI | ||||||||||||||||||||||||||||||||||
# Only publish to real PyPI on release | ||||||||||||||||||||||||||||||||||
if: github.event_name == 'release' | ||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this line really needed since the whole workflow only triggers on release? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would suggest including it, as (in my opinion) CD/release-related workflows like this are also supposed to run on a periodic basis using a CRON job (with a If at all, I would suggest making it more stringent, like this:
Suggested change
in the hope of making a release-oriented developer more careful (similar to my point above) to check things twice. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. so this is a triple check. i'm going to call in @webknjaz - Sviatoslav this is a template for pyOpenSci that people can use to quickly create a package structure. I Updated the GitHub action part to reflect what PyPA suggests in terms of security. Here we are having a small friendly debate around how many event checks should go into the build. I'm curious what your take on this is? More can't hurt, I would presume. it just adds a bit of complexity to the build to explain to people. So i want to make sure we keep things simple yet safe! thank you!! |
||||||||||||||||||||||||||||||||||
uses: pypa/gh-action-pypi-publish@release/v1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By the way, I noticed that this file has a few trailing whitespace in a few lines, you might consider fixing them with a local pre-commit hook: https://github.com/pre-commit/pre-commit-hooks#trailing-whitespace. For example, you fixed one such case yourself in line 114 of
copier.yml
🙈 (not with this PR, of course!)