Skip to content

Latest commit

 

History

History
83 lines (75 loc) · 16.6 KB

devlog.md

File metadata and controls

83 lines (75 loc) · 16.6 KB

Devlog

The python implementation, being the first implementation worked on in this repo, ended up not being as documented in this devlog as many of the others were; the length and assortment of examples increasing for each subsequent devlog makes this one look pretty sparse and empty in comparison. So I'll come back and retroactively rewrite it. Before begining this, a python package named collatz already existed on pypi, owned by NGeorgescu. Looping back to this after completing the julia implementation, I'd like to thank NGeorgescu for being generous with passing over the project name, so that this can live under the collatz name.

This is the Python implementation. Python (CPython source), known for its zen, uses pip to install packages, usually from those hosted on pypi. Virtual environments can be managed with venv. We can follow the Packaging Python Projects for the recommended steps to start setting up (which utilises some build tools you'll need to install). Or have a look at the developer guide. Or checkout python's list of recommended IDE's per feature (we're using Python extension for Visual Studio Code here). Plenty of other handy docs can be found in Python Packaging User Guide

Our packaging process here is to use build (PEP 517) to wrap setuptools, to create the wheel that can be published with twine. There are a number of different ways to get to the same end result of a python package that can be uploaded to pypi, and ours is;

  1. python -m build
  2. Build uses ./.pyproject.toml
  3. ./.pyproject.toml's [build-system] specifies a build-backend
  4. Ours is build-backend = "setuptools.build_meta" (see setuptools' build_meta)
  5. Setuptools, using ./.pyproject.toml, runs a ./setup.py if present, which also uses ./setup.cfg
    1. Although ./setup.py is now recommended against, in favour of just ./setup.cfg, it's the easiest dynamic entry, i.e. to use a __version__.py file to specify the version. (Although it looks like something similar can be done in ./setup.cfg with something like version = attr: my_package.VERSION)
  6. This creates the source distribution and wheel (PEP 427).
  7. We then can use twine to publish the source.
    1. We actually use a GitHub action to upload the build from a workflow; pypa/gh-action-pypi-publish. Twine recipes that do the same thing are in the ./Makefile, but this is a pypa maintained action to make it easier, rather than proctoring the existence of the necessary ./.pypirc file on the runner.

./.pypirc example

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
  username = __token__
  password = <pypi-token>

[testpypi]
  username = __token__
  password = <test-pypi-token>

"Local" version vs primary version used on GH runner.

My python --version is 3.8.10 locally, and picking a version you know you're using as the one to run things in in the pipelines can be helpful in minimising the potential for unforeseeable issues, but this is a list of python versions the installer action can accept. 3.8.10 doesn't have an ubuntu 22.04 release. The closest version available to the runner is 3.8.12.

Linting

We can lint with pylama, which wraps multiple other linters. Re can generate pylint rc file with pylint --generate-rcfile > .pylintrc. For more details on configuring pylint see this SO answer (pylint docs, codes and pylintrc). Configuring pyflakes is annoying. It reports;

src/collatz/__init__.py:2:1 W0611 '.parameterised._KNOWN_CYCLES' imported but unused [pyflakes]
src/collatz/__init__.py:3:1 W0611 '.parameterised.__VERIFIED_MAXIMUM' imported but unused [pyflakes]
src/collatz/__init__.py:4:1 W0611 '.parameterised.__VERIFIED_MINIMUM' imported but unused [pyflakes]
src/collatz/__init__.py:5:1 W0611 '.parameterised._ErrMsg' imported but unused [pyflakes]
src/collatz/__init__.py:6:1 W0611 '.parameterised._CC' imported but unused [pyflakes]

but neither configuring it in pylama.ini with;

[pylama:src/collatz/__init__.py]
ignore = W0401,W0611

Or in a pyflakes.ini with;

[pyflakes]
per-file-ignores =
    # imported but unused
    src/collatz/__init__.py: W0611

is dismissing these errors, and any googling for how to ignore specific erros in pyflakes suggest to either swap to flake8, a tool not listed on pylama's supported wraps, or to implement your own patch to ignore. We could " # NOWQA" all the offending lines on the file but the src/collatz/__init__.py: W0611 is the only complain the pyflakes is having, so we may as well just not run pyflakes, even thought pyflakes seemed to be the only linter to complain about unused imports, it seems a shame to have to get rid of it. We also seem to not be able to configure pycodestyle to ignore an error code. It seems like only pylint is getting configuration passed to it from pylama. Maybe it would be easiest sto swap to us pylint and flake8 together rather than pylama.

Documentation

Two tools for python documentation (see PEP 257 on docstrings) can be pydoc (wiki), included as part of python's core lib, and sphinx (readthedocs, wiki, pypi). We can run sphinx-quickstart in a new docs folder to create the basic structure. Part of this includes a conf.py that can be configured according to this. We chose "Y" when asked;

You have two options for placing the build directory for Sphinx output.
Either, you use a directory "_build" within the root path, or you separate
"source" and "build" directories within the root path.
> Separate source and build directories (y/n) [n]: y

So we need to generate our docs and put then in docs/source, adjacent to the index.rst, so we can run sphinx-apidoc -o docs/source src/. We should then be able to run sphinx-build docs/source docs/build or make html -C docs to build the html, but we get ERROR: Unknown directive type "automodule". We might need to add some extensions. Well, we need to sys.path.insert(0, os.path.abspath('../../src')) from the conf.py to locate the module, and use the sphinx.ext.autodoc extension, and set the -W --keep-going -n flags on sphinx-build (meaning "convert warnings into errors, keep going past the first one, and be nitpicky"), and we finally have the workflow yielding an error that lists all the errors in the docstrings in the source.

Now we can look at why there are a bunch of errors, for what I thought was the more common docstring format? PEP 257 – Docstring Conventions, this SO question, and this older blog reveal a few things. Apparently the assumptive default for pythonic docstrings is reStructuredText, and I've been relying on Google's python styleguide for docstrings (or the sphinx example). We can use the napoleon extension to allow us to use google style docstrings. The fix is to use the extension, "sphinx.ext.napoleon", and replace all Kwargs: in docstrings with the section it expects, Keyword Args:. We can also .gitkeep the docs/source/_static and docs/source/_templates to preserve the existence of these folders when running in the workflow, as their lack of existence for not having any file in them causes a warning in the workflow to error even though nothing is actually wrong.

Now we can pick a sphinx theme. I like both the Read the Docs Theme and the Guzzle Theme. "Read the docs" has the edge of ubiquity, and guzzle includes a warning it might not play nicely with "readthedocs" (I imagine it's refering to deploying a build using guzzle on the read the docs hosting site). It appears that sphinx-rtd-theme's requires a sphinx version less than 7. The issue on sphinx-rtd-theme to support sphinx version 7 has a comment that mentions another theme that wasn't listed on the sphinx themese site, furo. It might not be "read the docs", but to the credit of the quote in its Used By section, if it's got traction on pip, it must be pretty solid. I guess we'll try furo! (Turns out there's more themes here).

The initial appearance of the sphinx docs are pretty bare. If we look at Furo's recommendations, one of them is MyST, which is also recommended by sphinx. But in the process of trying to add anything else or edit it in anyway, sphinx is quickly becoming one of the more frustrating to work with auto-documentation tools out of this whole project. I don't want to have to learn a whole entire new file format, as debugging even simple issues that crop up requires an understanding of what's going on in the reStructuredText that the *.rst files are written in, and besides the effort of already having to locate some tool to extend sphinx with that advertises itself as making it possible to include markdown files, it not only doesn't fully explain how to actually use it but there aren't any easy or consistant answers to find on SO for "how do I get sphinx to actually do X", which ends up meaning, for the simple task of rendering the markdown in the <repo-root>/python/README.md, I've gone through about 5 or 6 different posts with several answers each suggesting that they managed to get sphinx to finally behave how they expected it to with some magic solution, but none of them have worked here. I've managed to get the raw text of the file to be included, which was itself already an hour of digging around online, and every time it runs the build, myst appears to pop in a little message that doesn't really give any information about why it isn't trying to render the page. So far most of the answers have revolved around using an rst file next to the index rst file to .. include:: ../../README.md. At this stage, if I wasn't committed to getting it done with sphinx to have a demonstrable experience of sphinx assisted masochism, and I was actually trying to create documentation with an "easy to use" tool, I'd abandon sphinx and go use a tool that was actually easy to use. That's not to say sphinx can't be powerful, or that there aren't great sites that are apparently written in sphinx.. so why is the barrier to entry high enough that trying to include a markdown file is an Achillies heel, for someone who just wants to auto gen some docs? It appears that copying the entire file and having a duplicate of it exist in here lets it properly render it, but that's a shitty expectation, to have a duplicate? Why can it not see the relative path outside of the source folder. Symlinking, ln -s ../../README.md README.md, appears to let it work the same way. I finally found a way to make it work, and it took locating this SO answer, althought frustraingly, the answer was apparently in some FAQ on MyST's site all along, tucked out of site. This SO answer gets an honourable mention, but now that we have an answer that allows us to inject the whole contents, parsed of course, into the index, and it's one that can accept paths to files outside of the docs source folder, we can just point to the regular readme and not worry about going via the symlink.

It also turns out that, after believing I had sphinx in a place where it was working and I wouldn't need to worry about it anymore, after merging and deploying the sphinx built docs to gh-pages, the site was mangled garbage. It turns out this is because the jekyll deplyoment gh-pages do ignores folders / files that start with underscores, and doesn't include them in the generated site. Sphinx's solution to this was apparently the add an extension that would yield a .nojekyll file. But that file is required to be at the root of the gh-pages deployment location, i.e. it must be in the folder from which the site would have been generated. It does nothing if it's in a subdirectory, which is our pattern here. So, with a bunch of files that sphinx has put in subdirectories it decided needed to start with underscores, the only option to get around with was to set up gh-pages as a proper jekyll deployment and add all the files in sphinx's deployment to the list of specific inclusions.

Empty Orphan

We're now ready to add once again create an empty orphan branch;

  1. git checkout --orphan gh-pages-python
  2. rm .git/index ; git clean -fdx
  3. git commit -m "Initial empty orphan" --allow-empty
  4. git push --set-upstream origin gh-pages-python