Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a deb package #25

Merged
merged 48 commits into from
May 12, 2021
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
813165b
Make a deb
velovix May 4, 2021
c260e54
Add build_deb to now workflow
velovix May 4, 2021
8cee561
Install poetry, but for real
velovix May 4, 2021
7d2465c
Fix a dash
velovix May 4, 2021
f0b57b3
Do things
velovix May 4, 2021
5a5dddf
More thangs
velovix May 4, 2021
b3e28c4
Oh yeah we playin it
velovix May 4, 2021
0dc6577
Add missing sudo
velovix May 4, 2021
9ba1e14
I really bit the big one this time
velovix May 4, 2021
6f2d732
Use single quotes
velovix May 4, 2021
f003d91
Try to store the deb as a build artifact
velovix May 4, 2021
4db2ea7
Fix naming
velovix May 4, 2021
c2c8f55
Rename to fix artifacts
velovix May 4, 2021
c43cdc3
Fix paths
velovix May 4, 2021
a484204
Bake system package-friendly defaults in
velovix May 4, 2021
48181be
Don't use CircleCI's Python image for deb builds
velovix May 4, 2021
a76163b
Load defaults from a file
velovix May 4, 2021
2c5ccdd
Fix dumb mistakes
velovix May 4, 2021
d2b098b
Convert default
velovix May 4, 2021
03e98c3
Add distutils, which Docker Compose seems to need
velovix May 4, 2021
afe9fab
Use distutils.util.strtobool
velovix May 4, 2021
72ee69a
Use cimg/python's included Poetry version
velovix May 4, 2021
5d41542
Add type annotations
velovix May 4, 2021
2114786
Formatting
velovix May 4, 2021
ded8237
Convince MyPy regarding nullability of a var
velovix May 4, 2021
980b878
Handle situation where config provides a bool
velovix May 4, 2021
0ff8b4c
Restore ability to auto-install curl and rsync
velovix May 5, 2021
1bd04ed
Add deb test to CI
velovix May 7, 2021
bc3e763
Add requirement
velovix May 7, 2021
3a77d4b
Move jobs around
velovix May 7, 2021
8c4bcae
Add requires
velovix May 7, 2021
26cb89a
Try building with 20.04
velovix May 7, 2021
f5dc014
Make deb building a matrix
velovix May 9, 2021
09bbdae
Use executors
velovix May 9, 2021
59f8520
Move up executors
velovix May 9, 2021
ef8c5a7
Fix matrix syntax
velovix May 9, 2021
b90ce3b
Fix deb naming
velovix May 9, 2021
8b700f5
Try with env vars
velovix May 9, 2021
65df3b3
Try out some of that clever stuff
velovix May 9, 2021
b362da8
Fix naming mistake
velovix May 9, 2021
9039d5d
Include defaults.yaml in Pip version
velovix May 9, 2021
666b8b2
Use the right name when setting data path env var
velovix May 9, 2021
97b29c8
Formatting
velovix May 9, 2021
a64a1ae
Use `.get` for default env var value
velovix May 10, 2021
c8a2d0e
Code review fixes
velovix May 11, 2021
6c5a804
Merge remote-tracking branch 'origin/feature/13-deb-install' into fea…
velovix May 11, 2021
180670f
Use setdefault to set datapath
velovix May 11, 2021
15af206
Formatting
velovix May 12, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 47 additions & 15 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
version: 2.1

parameters:
poetry_version:
poetry-version:
type: string
default: "1.1.4"

workflows:
lint:
jobs:
- lint
build:
jobs:
- build-deb
deploy:
jobs:
- upload_to_pypi:
- upload-to-pypi:
filters:
# Ignore any commit on any branch by default
branches:
Expand All @@ -21,43 +24,61 @@ workflows:
only: /^v.+\..+\..+/
tests:
jobs:
- test_installation
- test-installation


jobs:
lint:
docker:
- image: circleci/python:3.6
- image: cimg/python:3.6
BryceBeagle marked this conversation as resolved.
Show resolved Hide resolved
steps:
- checkout
- run: "curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
| python3 - --version << pipeline.parameters.poetry_version >>"
- install-poetry
- run: poetry install --no-root
- run: poetry run isort --check .
- run: poetry run black --check .
- run: poetry run mypy -p "brainframe.cli"
upload_to_pypi:

build-deb:
docker:
- image: cimg/base:stable-18.04
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a matrix for each version of ubuntu?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure. It doesn't seem like the resulting deb has any attachment to 18.04 specifically, and the deb installs fine on my local 20.04 installation. I imagine this will become more clear when we start trying to put this thing in a package repository.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out you were right. I'm not sure how it worked before, but building the deb on one version and installing it on another breaks some stuff. I've made the build-deb step a matrix build for 18.04 and 20.04.

steps:
- run: sudo apt-get update
- run: sudo apt-get install -y software-properties-common dpkg-dev devscripts equivs
- run: sudo add-apt-repository ppa:jyrki-pulliainen/dh-virtualenv
- run: sudo apt-get install -y dh-virtualenv
- install-poetry
BryceBeagle marked this conversation as resolved.
Show resolved Hide resolved
- checkout
- run: cp -r package/debian .
- run: cp package/setup.py .
- run: cp package/system_package_defaults.yaml brainframe/cli/defaults.yaml
- run: sudo mk-build-deps --install debian/control
- run: poetry export --output requirements.txt
- run: dpkg-buildpackage --unsigned-source --unsigned-changes --build=binary
- run: mv ../brainframe-cli*.deb brainframe-cli.deb
- store_artifacts:
path: brainframe-cli.deb
destination: brainframe-cli.deb

upload-to-pypi:
docker:
- image: circleci/python:3.6
- image: cimg/python:3.6
environment:
POETRY_HTTP_BASIC_PYPI_USERNAME: __token__

steps:
- checkout
- run: "curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
| python3 - --version << pipeline.parameters.poetry_version >>"
- install-poetry
- run: poetry build
- run: POETRY_HTTP_BASIC_PYPI_PASSWORD=${PYPI_PASSWORD} poetry publish
test_installation:

test-installation:
machine:
image: ubuntu-2004:202010-01
BryceBeagle marked this conversation as resolved.
Show resolved Hide resolved
steps:
- run: sudo apt-get update
- run: sudo apt-get install python3 python3-dev curl git -y
- checkout
- run: "curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
| python3 - --version << pipeline.parameters.poetry_version >>"
- run: echo "export PATH=$PATH:$HOME/.poetry/bin" >> $BASH_ENV
- install-poetry
- run: sudo $(which poetry) install
- run: sudo $(which poetry) run brainframe install --noninteractive
- run: sudo $(which poetry) run brainframe compose up -d
Expand All @@ -78,3 +99,14 @@ jobs:
echo "BrainFrame core service is up."
no_output_timeout: 1m
- run: sudo $(which poetry) run brainframe compose down


commands:
install-poetry:
steps:
- run: sudo apt-get update && sudo apt-get install -y python3-pip
- run: >
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
| python3 - --version << pipeline.parameters.poetry-version >>
# Add to the PATH
- run: echo 'export PATH=$HOME/.poetry/bin:$PATH' >> $BASH_ENV
14 changes: 4 additions & 10 deletions brainframe/cli/commands/backup.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@

import i18n
from brainframe.cli import (
config,
dependencies,
docker_compose,
env_vars,
os_utils,
print_utils,
)
Expand All @@ -19,8 +19,8 @@

@command("backup")
def backup():
install_path = env_vars.install_path.get()
data_path = env_vars.data_path.get()
install_path = config.install_path.value
data_path = config.data_path.value

args = _parse_args(data_path)

Expand All @@ -31,7 +31,7 @@ def backup():

docker_compose.assert_installed(install_path)

dependencies.rsync.ensure(args.noninteractive, args.install_rsync)
dependencies.rsync.ensure(args.noninteractive, False)
apockill marked this conversation as resolved.
Show resolved Hide resolved

if not args.noninteractive:
stop_brainframe = print_utils.ask_yes_no("backup.ask-stop-brainframe")
Expand Down Expand Up @@ -94,10 +94,4 @@ def _parse_args(data_path: Path):
help=i18n.t("general.noninteractive-help"),
)

parser.add_argument(
"--install-rsync",
action="store_true",
help=i18n.t("backup.install-rsync-help"),
)

return subcommand_parse_args(parser)
5 changes: 2 additions & 3 deletions brainframe/cli/commands/compose.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,12 @@
import sys

import i18n
from brainframe.cli import docker_compose, env_vars, os_utils, print_utils
from brainframe.cli import config, docker_compose

from .utils import command


@command("compose")
def compose():
install_path = env_vars.install_path.get()
install_path = config.install_path.value
docker_compose.assert_installed(install_path)
docker_compose.run(install_path, sys.argv[2:])
10 changes: 5 additions & 5 deletions brainframe/cli/commands/info.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from argparse import ArgumentParser

import i18n
from brainframe.cli import docker_compose, env_vars, print_utils
from brainframe.cli import config, docker_compose, print_utils

from .utils import command, subcommand_parse_args

Expand All @@ -10,15 +10,15 @@
def info():
args = _parse_args()

docker_compose.assert_installed(env_vars.install_path.get())
docker_compose.assert_installed(config.install_path.value)

server_version = docker_compose.check_existing_version(
env_vars.install_path.get()
config.install_path.value
)

fields = {
"install_path": env_vars.install_path.get(),
"data_path": env_vars.data_path.get(),
"install_path": config.install_path.value,
"data_path": config.data_path.value,
"server_version": server_version,
}

Expand Down
42 changes: 14 additions & 28 deletions brainframe/cli/commands/install.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@

import i18n
from brainframe.cli import (
config,
dependencies,
docker_compose,
env_vars,
os_utils,
print_utils,
)
Expand All @@ -30,11 +30,8 @@ def install():
print()

# Check all dependencies
dependencies.curl.ensure(args.noninteractive, args.install_curl)
dependencies.curl.ensure(args.noninteractive, False)
dependencies.docker.ensure(args.noninteractive, args.install_docker)
dependencies.docker_compose.ensure(
args.noninteractive, args.install_docker_compose
)

_, _, download_version = docker_compose.check_download_version()
print_utils.translate("install.install-version", version=download_version)
Expand All @@ -55,8 +52,8 @@ def install():
# Ask the user if they want to specify special paths for installation
print_utils.translate(
"install.default-paths",
default_install_path=env_vars.install_path.default,
default_data_path=env_vars.data_path.default,
default_install_path=config.install_path.default,
default_data_path=config.data_path.default,
)
use_default_paths = print_utils.ask_yes_no(
"install.ask-use-default-paths",
Expand All @@ -66,22 +63,21 @@ def install():
if args.noninteractive:
install_path = args.install_path
elif use_default_paths:
install_path = env_vars.install_path.default
install_path = config.install_path.default
else:
install_path = print_utils.ask_path(
"install.ask-brainframe-install-path",
env_vars.install_path.default,
"install.ask-brainframe-install-path", config.install_path.default,
)
install_path.mkdir(exist_ok=True, parents=True)

# Set up the data path
if args.noninteractive:
data_path = args.data_path
elif use_default_paths:
data_path = env_vars.data_path.default
data_path = config.data_path.default
else:
data_path = print_utils.ask_path(
"install.ask-data-path", env_vars.data_path.default
"install.ask-data-path", config.data_path.default
)
data_path.mkdir(exist_ok=True, parents=True)

Expand Down Expand Up @@ -120,15 +116,15 @@ def install():
# Recommend to the user to add their custom paths to environment variables
# so that future invocations of the program will know where to look.
if (
install_path != env_vars.install_path.default
or data_path != env_vars.data_path.default
install_path != config.install_path.default
or data_path != config.data_path.default
):
print()
print_utils.translate("install.set-custom-directory-env-vars")
print(
f"\n"
f'export {env_vars.install_path.name}="{install_path}"\n'
f'export {env_vars.data_path.name}="{data_path}"\n'
f'export {config.install_path.name}="{install_path}"\n'
f'export {config.data_path.name}="{data_path}"\n'
)


Expand All @@ -146,30 +142,20 @@ def _parse_args():
parser.add_argument(
"--install-path",
type=Path,
default=env_vars.install_path.default,
default=config.install_path.default,
help=i18n.t("install.install-path-help"),
)
parser.add_argument(
"--data-path",
type=Path,
default=env_vars.data_path.default,
default=config.data_path.default,
help=i18n.t("install.data-path-help"),
)
parser.add_argument(
"--install-docker",
action="store_true",
help=i18n.t("install.install-docker-help"),
)
parser.add_argument(
"--install-docker-compose",
action="store_true",
help=i18n.t("install.install-docker-compose-help"),
)
parser.add_argument(
"--install-curl",
action="store_true",
help=i18n.t("install.install-curl-help"),
)
parser.add_argument(
"--add-to-group",
action="store_true",
Expand Down
4 changes: 2 additions & 2 deletions brainframe/cli/commands/update.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from argparse import ArgumentParser

import i18n
from brainframe.cli import docker_compose, env_vars, print_utils
from brainframe.cli import config, docker_compose, print_utils
from packaging import version

from .utils import command, subcommand_parse_args
Expand All @@ -11,7 +11,7 @@
def update():
args = _parse_args()

install_path = env_vars.install_path.get()
install_path = config.install_path.value

docker_compose.assert_installed(install_path)

Expand Down
Loading