04_dependencies_ci.qmd

---
title: "Dependencies and Continuous Integration"
format: 
    html: default
    revealjs:
      output-file: 04_dependencies_ci_slides.html
      slide-number: true
footer: Python package development
logo: academy_logo.png
---


##

:::: {.columns}

::: {.column}

**Application**

*A program that is run by a user*

* command line tool
* script
* web application

Pin versions to ensure reproducibility, e.g. `numpy==1.11.0`
:::

::: {.column}

**Library**

*A program that is used by another program*

* Python package
* Low level library (C, Fortran, Rust, ...)

Make the requirements as loose as possible, e.g. `numpy>=1.11.0`
:::

::::

::: {.notes}
Make the requirements loose, to avoid conflicts with other packages.

:::

## Dependency management

Example of pinning versions:

```{.txt filename="requirements.txt"}
numpy==1.11.0
scipy==0.17.0
matplotlib==1.5.1
```
. . .

Or using a range of versions:

```{.txt filename="requirements.txt"}
numpy>=1.11.0
scipy>=0.17.0
matplotlib>=1.5.1,<=2.0.0
```
. . .

Install dependencies:

```{.bash}
$ pip install -r requirements.txt
```

::: {.notes}

A common way to declare dependencies is to use a `requirements.txt` file.

:::

## Creating an installable package


![](images/project.png)


## `setup.py` vs `pyproject.toml` {.smaller}

`setup.py`

* Traditional approach to defining package configuration and dependencies.
* Defines metadata, dependencies, and entry points in a Python script.
* Uses `setuptools` to generate packages and install the package.

. . . 

`pyproject.toml`

* Modern approach to defining package configuration and dependencies.
* Defines metadata, dependencies, build tools, and packaging config in a TOML file.
* Uses `poetry` or `hatchling` to generate packages and install the package.


## Install with optional dependencies

```{.toml filename="pyproject.toml"}
[project.optional-dependencies]
dev = ["pytest",
       "ruff",
       "sphinx",
       "sphinx-rtd-theme",
       "myst-parser",
       ]

test = ["pytest", "pytest-cov"]
```

&nbsp;

```{.bash}
$ pip install mini[test]
```




## Creating an installable package

Install package in editable mode:
```{.bash}
$ pip install -e .
```

. . .

Start a Python session:
```{.python}
>>> import mini
>>> mini.foo()
42
```

. . .

Run tests:
```{.bash}
$ pytest
...

tests/test_foo.py .                       [100%]

=============== 1 passed in 0.01s ===============
```




## Virtual environments

::: {.incremental}

* Creates a clean environment for each project
* Allows different versions of a package to coexist on your machine
* Can be used to create a reproducible environment for a project
* To achieve complete isolation, use Docker containers (not covered in this course)

:::

## Virtual environments

```{.bash code-line-numbers="|3-4|5-6|7"}
$ which python
/usr/bin/python
$ python -m venv venv
$ source venv/bin/activate # for 🐧 or venv\Scripts\activate.bat 🪟
(venv)$ which python
/home/user/src/myproj/venv/bin/python
(venv)$ pip install -r requirements.txt
```

::: {.notes}
* Back in the days, when disk space was limited, it was a good idea to have a separate environment for each project.
* Today, disk space is cheap, and it is a good idea to have a separate environment for each project.
:::

## Conda/mamba environments

Conda/mamba is a package manager that can be used to create virtual environments.

```{.console}
$ where python
C:\Users\JAN\AppData\Local\miniforge3\python.exe
$ conda create -n myproj -f requirements.txt
$ conda activate myproj
(myproj)$ where python
C:\Users\JAN\AppData\Local\miniforge3\envs\myproj\python.exe
```


## Continuous Integration

Running tests on every commit in a well defined environment ensures that the code is working as expected.

It solves the "it works on my machine" problem.

Executing code on a remote server is a good way to ensure that the code is working as expected.

There are many CI services available, e.g.:

* GitHub Actions
* Azure Pipelines
* Travis CI
* Circle CI


::: {.notes}

GitHub Actions was forked from Azure Pipelines and runs on the same type of infrastructure, thus are very similar technologies.

:::

## GitHub Actions {.smaller}

:::: {.columns}

::: {.column}

* Workflow are stored in the `.github/workflows` folder.
* Workflow is described in a YAML file.
* YAML is whitespace sensitive (like Python).
* YAML can contain lists, dictionaries and strings, and can be nested.

:::

::: {.column}

```bash
$ tree mikeio/.github/
mikeio/.github/
└── workflows
    ├── docs.yml
    ├── downstream_test.yml
    ├── full_test.yml
    ├── notebooks_test.yml
    ├── perf_test.yml
    ├── python-publish.yml
    └── quick_test.yml
```

:::
::::

---

```yaml
name: Quick test

on: # when to run the workflow
  push:
    branches: [ main]
  pull_request:
    branches: [ main ]

jobs: # what to run
  build:
    runs-on: ubuntu-latest # on what operating system

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: 3.9
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip

    - name: Install mikeio
      run: |
        pip install .[test]
    - name: Test with pytest
      run: |
        pytest
```

---

🙂🚀

![](images/github_action_result.png)

---

☹️

![](images/github_action_result_fail.png)

## Benefits of CI

::: {.incremental}

* Run tests on every commit
* Test on different operating systems
* Test on different Python versions
* Create API documentation (next week)
* Publish package to PyPI or similar package repository (two weeks from now)

:::

## Triggers

* `push` and `pull_request` are the most common triggers
* `schedule` can be used to run the workflow on a schedule
* `workflow_dispatch` can be used to trigger the workflow manually

```{.yaml code-line-numbers="2-3|4-5|6-7|8"}
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  schedule:
    - cron: '0 0 * * 0'
  workflow_dispatch:
```

## Jobs

* Operating system
* Python version
* ...

```{.yaml}
...
jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]
        python-version: [3.8, 3.9, "3.10","3.11"]
...
```

## GitHub Releases {.smaller}

:::: {.columns}

::: {.column}

* GitHub releases are a way to publish software releases.

* You can upload files, write release notes and tag the release.

* As a minimum, the release will contain the source code at the time of the release.

* Creating a release can trigger other workflows, e.g. publishing a package to PyPI.

:::

::: {.column}

![](images/github_release.png)

:::
::::


<https://github.com/pydata/xarray/releases/tag/v2022.12.0>




## Summary

::: {.incremental}

* Application vs library
* Prefer `pyproject.toml` over `setup.py`
* Use a separate virtual environment for each project
* Use GitHub Actions to run tests on every commit
* Use GitHub Releases to publish software releases

:::