[Feat] Integrate grass-addons into mkdocs doc

[wenzeslaus]

What is needed to make it on par with the current HTML doc:

• Build addons into the doc (example in CI: Add workflow to build mkdocs in GHA #5152)
• Subdirectory for the tools as in addons/name.html cronjobs: Add Markdown and generalize addon build grass-addons#1330
• Keyword/tag index contains keywords from addon tools (e.g., columns of v.vect.stats.multi). build: Allow separate build of Markdown and HTML keywords #5177 (or docs: Add support for addon keywords to Markdown #5175 plus custom fixes to run the build)
• Keyword/tag links in addon tools go to the right directory. docs: Add support for addon keywords to Markdown #5175
• Links to core tools and links to other addon tools go to their respective directories. (This would be actually a bonus, I didn't find good examples of how or that this works now.) The status of this can be checked just with the mkdocs output in the Documentation workflow in the CI.
• Links to source code and history work. Latest change is visible and correct. This can be checked with the artifact from CI. (Bottom section/block)
• Build the Addons index page in CI as part of the MkDocs site.
• description and keywords (together with build output files) are extracted to grass.osgeo.org/addons/grass$MAJOR/modules.xml by utils/cronjobs_osgeo_lxd/build-xml.py in utils/cronjobs_osgeo_lxd/cron_grass_preview_build_binaries.sh

Optional tasks:

• Edit on GitHub works also for addons.

How to get it

CI

Grab mkdocs site artifact from main, related PR, or the Action tab / Documentation.

Locally

Assumes you have core repo in grass and addons repo in grass-addons.

Assumes you are in environment with mkdocs.

Compile and install:

cd grass
.github/workflows/build_ubuntu-22.04.sh "$HOME/install"

Get the get version numbers:

eval $(./utils/update_version.py status --bash)
cd ..

Set path to compiled binary:

export PATH="$HOME/install/bin":$PATH

Compile addons:

./grass-addons/utils/cronjobs_osgeo_lxd/compile_addons_git.sh "$MAJOR" "$MINOR" $(pwd)/grass-addons/src $(grass --config path) $(pwd)/addons-build-dir grass

Path to where the addons doc should be:

MKDOCS_DIR="$(grass --config path)/docs/mkdocs"

Copy the relevant part of compiled doc:

mkdir -p "$MKDOCS_DIR/source/addons"
mv addons-build-dir/docs/md/source/* "$MKDOCS_DIR/source/addons"

Rebuild keywords:

export ARCH="linux"
export ARCH_DISTDIR="$(grass --config path)"
export VERSION_NUMBER="$VERSION"
grass --tmp-project XY --exec python grass/man/build_keywords.py "$MKDOCS_DIR/source" "$MKDOCS_DIR/source/addons"

export SITE_NAME="GRASS GIS $VERSION Reference Manual"
export COPYRIGHT="© 2003-$YEAR GRASS Development Team, GRASS GIS $VERSION Reference Manual"
cd "$MKDOCS_DIR"
mkdocs build

The workflow is similar to the CI code from #5152.

[wenzeslaus]

What is not clear to me here is how keywords should handled. I have to do local changes to the keywords script to make it work, i.e., have keywords from addons in the keyword index. Then, I still need to fix links in the addons dir to point to the parent directory. I can't find that code anywhere.

[neteler]

Probably @tmszi knows more about the keywords magic?

[wenzeslaus]

I think I found it. The missing piece of puzzle for me was HTML_PAGE_FOOTER_PAGES_PATH.

[wenzeslaus]

Current state on main

Keywords

Keywords generally work in both directions.

There are some anchor naming issues (mkdocs drops special characters from anchor while our custom tooling for HTML preserved them):

INFO    -  Doc file 'addons/g.proj.identify.md' contains a link '../keywords.md#.prj', but the doc 'keywords.md' does not contain an anchor '#.prj'.
INFO    -  Doc file 'addons/r.mcda.input.md' contains a link '../keywords.md#multi-criteria-decision-analysis-(mcda)', but the doc 'keywords.md' does not contain an anchor '#multi-criteria-decision-analysis-(mcda)'.
INFO    -  Doc file 'addons/r.mcda.output.md' contains a link '../keywords.md#multi-criteria-decision-analysis-(mcda)', but the doc 'keywords.md' does not contain an anchor '#multi-criteria-decision-analysis-(mcda)'.
INFO    -  Doc file 'addons/r.stream.stats.md' contains a link '../keywords.md#horton's-statistics', but the doc 'keywords.md' does not contain an anchor '#horton's-statistics'.

Additionally, there is a number of broken links reported by mkdocs which are mostly also in the HTML doc (passing silently) related to non-existent topics, addon tools, missing files, core tools. However, there is couple of issues which definitely look new, for example:

WARNING -  Doc file 'addons/r.futures.parallelpga.md' contains a link 'r.futures.devpressure.md', but the target 'addons/r.futures.devpressure.md' is not found among documentation files.
WARNING -  Doc file 'addons/r.futures.parallelpga.md' contains a link 'r.futures.demand.md', but the target 'addons/r.futures.demand.md' is not found among documentation files.
WARNING -  Doc file 'addons/r.futures.parallelpga.md' contains a link 'r.futures.calib.md', but the target 'addons/r.futures.calib.md' is not found among documentation files.

Source code links

Source code and history links work.

The commit does not, Here it is a link which always goes to grass repo, never to grass-addons, and it is always the latest repo commit, not the specific dir commit, even for the core tools (so issue not necessarily related to addons). This might be more the CI setup issue, than the Markdown vs HTML issue (or both).

Edit button

Works for core tools, but not for addons.

Footer

Links in footer of addon tools go to addons dir, not to to ../.

[echoix]

I have a general opinion on addons docs built in the main repo.

I'm not sure if it won't be causing us more problems by coupling our high activity repo, with the low activity grass-addons.

Repeatedly building addons in the main repo seems like a waste to me.

In the best of worlds, a solution where each of the repos could be independent, and each could build the docs and upload them and still being usable on the web (not too separated, still integrated together) would be ideal.

[neteler]

In the best of worlds, a solution where each of the repos could be independent, and each could build the docs and upload them and still being usable on the web (not too separated, still integrated together) would be ideal.

In this case, the integration of the addons into the keyword index, full index and other overview pages would no longer be possible. That was actually a great achievement, done esp. by @tmszi. This said: it used to happen on the server, once a day. Doing everything in GHA changes the situation. So we need to find the best of the two worlds...

[echoix]

I understand things might not be possible, I didn't evaluate the feasibility, that's why I didn't object myself really hard in that effort. It's only a dream.

[wenzeslaus]

I'm open to suggestions. I just don't have a better idea at this point.

Not completely wasted though, it is good to check that addons compile after change on main, but that's of course not a deep test. The documentation also goes online for both and is influenced by combination of both. Most changes won't matter, but that's the case always.

What I don't like is actually the possibility to break main with addons change.