Docs simplify (#1011)

* Docs/3786 (#1004) * rm katacoda Signed-off-by: Niels Bantilan <[email protected]> * update docs requirement Signed-off-by: Niels Bantilan <[email protected]> * add hello world examples Signed-off-by: Niels Bantilan <[email protected]> * implement auto_examples custom extension Signed-off-by: Niels Bantilan <[email protected]> * move all basics files to new folder Signed-off-by: Niels Bantilan <[email protected]> * exclude new examples dir from packages Signed-off-by: Niels Bantilan <[email protected]> * debug serialize example github action Signed-off-by: Niels Bantilan <[email protected]> * debugging github action Signed-off-by: Niels Bantilan <[email protected]> * rename src to basics Signed-off-by: Niels Bantilan <[email protected]> * comment push to ghcr Signed-off-by: Niels Bantilan <[email protected]> * debuggin workflow Signed-off-by: Niels Bantilan <[email protected]> * add top-level readme to flyte examples Signed-off-by: Niels Bantilan <[email protected]> * add task cache serialize Signed-off-by: Niels Bantilan <[email protected]> * update aws sagemaker requirements Signed-off-by: Niels Bantilan <[email protected]> * update kfpytorch requirements Signed-off-by: Niels Bantilan <[email protected]> * remove requirements-common from new examples Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * move all examples to new structure (#1006) * move control flow Signed-off-by: Niels Bantilan <[email protected]> * move type system Signed-off-by: Niels Bantilan <[email protected]> * update type system requirements Signed-off-by: Niels Bantilan <[email protected]> * update deps Signed-off-by: Niels Bantilan <[email protected]> * add testing Signed-off-by: Niels Bantilan <[email protected]> * move containerization Signed-off-by: Niels Bantilan <[email protected]> * move all user guide examples Signed-off-by: Niels Bantilan <[email protected]> * remove dockerfiles for non-docker examples Signed-off-by: Niels Bantilan <[email protected]> * update serialization Signed-off-by: Niels Bantilan <[email protected]> * update Signed-off-by: Niels Bantilan <[email protected]> * update image-spec deps Signed-off-by: Niels Bantilan <[email protected]> * add example tempalte Signed-off-by: Niels Bantilan <[email protected]> * fix example template Signed-off-by: Niels Bantilan <[email protected]> * move ml training tutorials Signed-off-by: Niels Bantilan <[email protected]> * move feature engineering Signed-off-by: Niels Bantilan <[email protected]> * move ml tutorials Signed-off-by: Niels Bantilan <[email protected]> * moving integrations Signed-off-by: Niels Bantilan <[email protected]> * wip integrations Signed-off-by: Niels Bantilan <[email protected]> * complete moving integrations Signed-off-by: Niels Bantilan <[email protected]> * rename plugins with _plugin suffix Signed-off-by: Niels Bantilan <[email protected]> * fix names Signed-off-by: Niels Bantilan <[email protected]> * debug serialize example Signed-off-by: Niels Bantilan <[email protected]> * debugging Signed-off-by: Niels Bantilan <[email protected]> * update dockerfiles Signed-off-by: Niels Bantilan <[email protected]> * update dockerfiles Signed-off-by: Niels Bantilan <[email protected]> * fix duckdb docker, update tutorial content Signed-off-by: Niels Bantilan <[email protected]> * update userguide toc Signed-off-by: Niels Bantilan <[email protected]> * update integrations docs Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * clean up cruft (#1010) * clean up cruft Signed-off-by: Niels Bantilan <[email protected]> * move everything to top-level, rm cookbook Signed-off-by: Niels Bantilan <[email protected]> * fix maxdepth Signed-off-by: Niels Bantilan <[email protected]> * add notebooks to docs build Signed-off-by: Niels Bantilan <[email protected]> * delete prquantifier.yaml Signed-off-by: Niels Bantilan <[email protected]> * delete stale tests, update contributing and env setup guides Signed-off-by: Niels Bantilan <[email protected]> * fix environment setup ref Signed-off-by: Niels Bantilan <[email protected]> * write sphinx to myst py script conversion Signed-off-by: Niels Bantilan <[email protected]> * convert sphinx gallery python scripts to myst markdown Signed-off-by: Niels Bantilan <[email protected]> * clean up markdown Signed-off-by: Niels Bantilan <[email protected]> * delete conversion script Signed-off-by: Niels Bantilan <[email protected]> * formatting updates Signed-off-by: Niels Bantilan <[email protected]> * fix feast notebook, formatting, update contributin Signed-off-by: Niels Bantilan <[email protected]> * lint all files Signed-off-by: Niels Bantilan <[email protected]> * clean up links, add contribution CTA Signed-off-by: Niels Bantilan <[email protected]> * fix sagemaker, papermill examples Signed-off-by: Niels Bantilan <[email protected]> * fix sagemaker example Signed-off-by: Niels Bantilan <[email protected]> * fix make setup Signed-off-by: Niels Bantilan <[email protected]> * add lint to github action Signed-off-by: Niels Bantilan <[email protected]> * clean up requirements files Signed-off-by: Niels Bantilan <[email protected]> * update dev requirement Signed-off-by: Niels Bantilan <[email protected]> * lint Signed-off-by: Niels Bantilan <[email protected]> * sandbox setup Signed-off-by: Niels Bantilan <[email protected]> * update flytectl config init Signed-off-by: Niels Bantilan <[email protected]> * debug registration action Signed-off-by: Niels Bantilan <[email protected]> * use demo sandbox config Signed-off-by: Niels Bantilan <[email protected]> * debugging Signed-off-by: Niels Bantilan <[email protected]> * add back all examples Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> * fix lint, formatting Signed-off-by: Niels Bantilan <[email protected]> * fix tar.gz package names, fix formatting Signed-off-by: Niels Bantilan <[email protected]> * trigger version bump Signed-off-by: Niels Bantilan <[email protected]> * unpack archive, update basics deps Signed-off-by: Niels Bantilan <[email protected]> * package flyte tgz to root Signed-off-by: Niels Bantilan <[email protected]> * only upload pb files Signed-off-by: Niels Bantilan <[email protected]> * fix issue with register Signed-off-by: Niels Bantilan <[email protected]> * Switch to flytectl demo (#1012) Signed-off-by: Jeev B <[email protected]> * revert comment Signed-off-by: Niels Bantilan <[email protected]> * revert to flytectl sandbox Signed-off-by: Niels Bantilan <[email protected]> * add back tags Signed-off-by: Niels Bantilan <[email protected]> --------- Signed-off-by: Niels Bantilan <[email protected]> Signed-off-by: Jeev B <[email protected]> Co-authored-by: Jeev B <[email protected]>
flyteorg · Jun 30, 2023 · 0cc5f52 · 0cc5f52
1 parent c8864e3
commit 0cc5f52
Show file tree

Hide file tree

Showing 620 changed files with 16,300 additions and 14,634 deletions.
diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml
@@ -5,70 +5,93 @@ on:
     branches:
       - master
     paths-ignore:
-      - 'cookbook/docs/**'
+      - 'docs/**'
   pull_request:
     paths-ignore:
-      - 'cookbook/docs/**'
+      - 'docs/**'
 jobs:
 
-  # Lists all directories that have Dockerfile in them...
-  list:
+  lint:
     runs-on: ubuntu-latest
-    name: "Create a list of packages"
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Cache pip
+      uses: actions/cache@v2
+      with:
+        # This path is specific to Ubuntu
+        path: ~/.cache/pip
+        # Look to see if there is a cache hit for the corresponding requirements files
+        key: ${{ format('{0}-pip-{1}', runner.os, hashFiles('dev-requirements.txt')) }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install -r dev-requirements.txt
+    - name: Pip info
+      run: python -m pip list
+    - name: Lint
+      run: pre-commit run --all --show-diff-on-failure
+
+  # This is the build system for the new example directory structure
+  list_examples:
+    runs-on: ubuntu-latest
+    name: "Create a list of example packages"
     steps:
       - uses: actions/checkout@v2
       - name: "Provide the list"
-        id: create-list
-        run: echo "::set-output name=packages::$(find cookbook -mindepth 1 -maxdepth 5 -type f -name Dockerfile -exec dirname '{}' \; | sort | jq --raw-input . | jq --slurp . | jq -c .)"
+        id: create-example-list
+        run: echo "PACKAGES=$(find examples -mindepth 1 -maxdepth 2 -type f -name Dockerfile -exec dirname '{}' \; | sort | jq --raw-input . | jq --slurp . | jq -c .)" >> "$GITHUB_OUTPUT"
     outputs:
-      packages: "${{ steps.create-list.outputs.packages }}"
+      packages: "${{ steps.create-example-list.outputs.PACKAGES }}"
 
-  trigger_serialize_register_workflow:
+  trigger_serialize_register_examples:
     name: Serialize & Register Flytesnacks workflow
-    needs: [list]
-    uses: ./.github/workflows/serialize_workflow.yml
+    needs: [list_examples]
+    uses: ./.github/workflows/serialize_example.yml
     with:
-      packages: ${{ needs.list.outputs.packages }}
+      packages: ${{ needs.list_examples.outputs.PACKAGES }}
     secrets:
       FLYTE_BOT_PAT: ${{ secrets.FLYTE_BOT_PAT }}
 
-  push_github:
-    name: Build & Push to GHCR
+  push_example_image_to_github:
+    name: Build & Push Example Image to GHCR
     runs-on: ubuntu-latest
-    needs: [list]
+    needs: [list_examples]
     strategy:
       matrix:
-        directory: "${{ fromJson(needs.list.outputs.packages) }}"
+        directory: "${{ fromJson(needs.list_examples.outputs.packages) }}"
     steps:
       - uses: actions/checkout@v2
         with:
           fetch-depth: "0"
       - name: 'Set example name'
         id: example_id
-        # add a step output `steps.build_id.outputs.id` as the unique id
-        run: |
-          echo "::set-output name=id::$(basename -- ${{ matrix.directory }})"
-          temp=${{ matrix.directory }}
-          echo "::set-output name=path::$(echo "./${temp%/*}")"
+        run: echo "EXAMPLE_NAME=$(basename -- ${{ matrix.directory }})" >> "$GITHUB_OUTPUT"
       - name: Build & Push Docker Image to Github Registry
         uses: whoan/docker-build-with-cache-action@v5
         with:
           # https://docs.github.com/en/packages/learn-github-packages/publishing-a-package
           username: "${{ secrets.FLYTE_BOT_USERNAME }}"
           password: "${{ secrets.FLYTE_BOT_PAT }}"
           image_name: ${{ github.repository_owner }}/flytecookbook
-          image_tag: ${{ steps.example_id.outputs.id }}-latest,${{ steps.example_id.outputs.id }}-${{ github.sha }}
+          image_tag: ${{ steps.example_id.outputs.EXAMPLE_NAME }}-latest,${{ steps.example_id.outputs.EXAMPLE_NAME }}-${{ github.sha }}
           registry: ghcr.io
           push_git_tag: ${{ github.event_name != 'pull_request' }}
           push_image_and_stages: ${{ github.event_name != 'pull_request' }}
           build_extra_args: "--compress=true --build-arg=tag=ghcr.io/${{ github.repository_owner }}/flytecookbook:${{ steps.example_id.outputs.id }}-${{ github.sha }}"
-          context: ${{ steps.example_id.outputs.path }}
-          dockerfile: ${{ steps.example_id.outputs.id }}/Dockerfile
+          context: ${{ matrix.directory }}
+          dockerfile: Dockerfile
 
   bump_version:
     name: Bump Version
     if: ${{ github.event_name != 'pull_request' }}
-    needs: [ trigger_serialize_register_workflow, push_github ] # Only to ensure it can successfully build
+    needs: [ trigger_serialize_register_examples, push_example_image_to_github ] # Only to ensure it can successfully build
     uses: flyteorg/flytetools/.github/workflows/bump_version.yml@master
     secrets:
       FLYTE_BOT_PAT: ${{ secrets.FLYTE_BOT_PAT }}
@@ -113,11 +136,11 @@ jobs:
           path: ./download-artifact/
       - name: Package Examples
         run: |
-          mkdir -p cookbook/release-snacks
+          mkdir -p release-snacks
           cd download-artifact
-          for i in */; do tar -czvf "../cookbook/release-snacks/${i%/}.tar.gz" "$i" & done; wait
+          for i in */; do tar -czvf "../release-snacks/${i%/}.tar.gz" "$i" & done; wait
           cd .. && sudo rm -rf download-artifact/
-          cp cookbook/flyte_tests_manifest.json cookbook/release-snacks/flyte_tests_manifest.json
+          cp flyte_tests_manifest.json release-snacks/flyte_tests_manifest.json
       - name: Release test manifest
         uses: goreleaser/goreleaser-action@v2
         with:

diff --git a/.github/workflows/docs_build.yml b/.github/workflows/docs_build.yml
@@ -20,7 +20,6 @@ jobs:
         # NOTE: myst-nb build warnings are logged in docs/_build/reports/*.err.log
         run: |
           sudo apt-get install python3-sphinx
-          cd cookbook
           pip install -r docs-requirements.txt
           SPHINXOPTS="-W" cd docs && make html
           cat _build/reports/**/*.err.log || echo 'no error reports'
diff --git a/.github/workflows/serialize_workflow.yml → .github/workflows/serialize_example.yml b/.github/workflows/serialize_workflow.yml → .github/workflows/serialize_example.yml
@@ -1,4 +1,4 @@
-name: Serialize & Register Examples
+name: Serialize & Register Docs Examples
 
 on:
   workflow_call:
@@ -25,24 +25,24 @@ jobs:
     strategy:
       matrix:
         directory: "${{ fromJson(inputs.packages) }}"
+      fail-fast: false
     steps:
       - uses: actions/checkout@v2
         with:
           fetch-depth: "0"
       - name: 'Set example id'
         id: example_id
-        # add a step output `steps.build_id.outputs.id` as the unique id
-        run: echo "::set-output name=id::$(echo ${{ matrix.directory }} | sed 's_/_-_g')"
-      - name: Serialize Examples
-        env:
-          SANDBOX: "1"
+        run: echo "EXAMPLE_ID=$(echo ${{ matrix.directory }} | sed 's@/@-@g' | sed 's@_@-@g' )" >> "$GITHUB_OUTPUT"
+      - name: Build docker image and serialize examples
         run: |
-          REGISTRY=ghcr.io/flyteorg make -C ${{ matrix.directory }} serialize
+          ./scripts/serialize-example.sh ${{ matrix.directory }} ${{ github.sha }}
+          tar -xvf ${{ matrix.directory }}/flyte-package.tgz -C ${{ matrix.directory }}
       - name: Upload artifacts
         uses: actions/upload-artifact@v2
         with:
-          name: snacks-${{ steps.example_id.outputs.id }}
-          path: ${{ matrix.directory }}/_pb_output/*
+          name: snacks-${{ steps.example_id.outputs.EXAMPLE_ID }}
+          path: ${{ matrix.directory }}/**/*.pb
+
 
   # Download all artifacts generated from the previous job. Startup a sandbox cluster then register all of them.
   register:
@@ -63,7 +63,8 @@ jobs:
           path: ./download-artifact/
       - name: setup sandbox
         run: |
-          make setup
+          flytectl sandbox start
+          flytectl config init
       - name: Register examples
         uses: unionai/[email protected]
         with:

diff --git a/.gitignore b/.gitignore
@@ -7,7 +7,6 @@ __pycache__/
 *.dat
 .DS_Store
 gen_modules
-cookbook*.pb
 *.h5
 data/
 *.pt
@@ -23,13 +22,24 @@ _build/
 *-out.html
 *-out.ipynb
 .python-version
-cookbook/release-snacks
+release-snacks
 .kube/
 .docker/
 .cache/
 download-artifact/
 typescript
 .bash_history
 .venv/
-cookbook/docs/_tags/
-cookbook/docs/_flyte_decks/
+docs/_tags/
+docs/_flyte_decks/
+.ipython/
+auto_*/
+docs/auto/*
+docs/_rsts/
+.hypothesis
+.config/
+test.ipynb
+jupyter_execute
+_flyte_decks
+.keras
+**/flyte-package.tgz
diff --git a/.goreleaser.yml b/.goreleaser.yml
@@ -10,4 +10,4 @@ release:
   name_template: "{{.ProjectName}} v{{.Version}}"
   disable: false
   extra_files:
-    - glob: ./cookbook/release-snacks/*
+    - glob: ./release-snacks/*
diff --git a/.hypothesis/unicode_data/12.1.0/charmap.json.gz b/.hypothesis/unicode_data/12.1.0/charmap.json.gz
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -11,7 +11,6 @@ repos:
   rev:  5.12.0
   hooks:
     - id: isort
-      args: ["--profile", "black"]
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: v4.0.1
   hooks:

diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -10,17 +10,13 @@ build:
     python: "3.9"
   apt_packages:
     - pandoc
-  jobs:
-    pre_build:
-      - pip install pypandoc
-      - python ./cookbook/scripts/ipynb_to_gallery.py
 
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
-  configuration: cookbook/docs/conf.py
+  configuration: docs/conf.py
 
 # Optionally set the version of Python and requirements required to build your docs
 python:
   install:
-    - requirements: cookbook/dev-requirements.txt
-    - requirements: cookbook/docs-requirements.txt
+    - requirements: dev-requirements.txt
+    - requirements: docs-requirements.txt
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
@@ -1,3 +1,2 @@
 This project is governed by LF AI Foundation's [code of conduct](https://lfprojects.org/policies/code-of-conduct/).
 All contributors and participants agree to abide by its terms.
-
diff --git a/Makefile b/Makefile
@@ -1,102 +1,22 @@
-export REPOSITORY=flytesnacks
-
 .SILENT:
 
-# Flyte sandbox configuration variables
-KUBERNETES_API_PORT := 30086
-FLYTE_PROXY_PORT := 30081
-K8S_DASHBOARD_PROXY_PORT := 30082
-MINIO_PROXY_PORT := 30084
-FLYTE_SANDBOX_NAME := flyte-sandbox
-FLYTE_DIR := ~/.flyte
-
-
-# Module of cookbook examples to register
-EXAMPLES_MODULE ?= core
-
-define LOG
-echo "$(shell tput bold)$(shell tput setaf 2)$(1)$(shell tput sgr0)"
+define PIP_COMPILE
+pip-compile $(1) ${PIP_ARGS} --upgrade --verbose --resolver=backtracking
 endef
 
-define RUN_IN_SANDBOX
-docker exec -it \
-	-e DOCKER_BUILDKIT=1 \
-	-e MAKEFLAGS \
-	-e REGISTRY \
-	-e VERSION \
-        -w /root \
-	$(FLYTE_SANDBOX_NAME) \
-	$(1)
-endef
-
-.PHONY: update_boilerplate
-update_boilerplate:
-	@curl https://raw.githubusercontent.com/flyteorg/boilerplate/master/boilerplate/update.sh -o boilerplate/update.sh
-	@boilerplate/update.sh
-
-.PHONY: fmt
-fmt: ## Format code with black and isort
-	pre-commit run black --all-files || true
-	pre-commit run isort --all-files || true
-
-.PHONY: lint
-lint: ## Run linters
-	pre-commit run --all-files
-
-.PHONY: spellcheck
-spellcheck:  ## Runs a spellchecker over all code and documentation
-	codespell -L "te,raison,fo" --skip="./docs/build,./.git"
-
-.PHONY: help
-help: ## Show help message
-	@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m\033[0m\n"} /^[$$()% a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-15s\033[0m %s\n", $$1, $$2 } /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) } ' $(MAKEFILE_LIST)
-
-# Helper to determine if a sandbox is up and running
-.PHONY: _requires-sandbox-up
-_requires-sandbox-up:
-ifeq ($(shell docker ps -f name=$(FLYTE_SANDBOX_NAME) --format='{{.ID}}'),)
-	$(error Cluster has not been started! Use 'make start' to start a cluster)
-endif
-
-.PHONY: setup
-setup:
-	$(call LOG,Starting Flyte sandbox)
-	flytectl sandbox start --source=$(shell pwd)
-	flytectl config init
-
-.PHONY: start
-start: setup fast_register
-	echo "Flyte is ready! Flyte UI is available at http://localhost:$(FLYTE_PROXY_PORT)/console."
-
-.PHONY: teardown
-teardown: _requires-sandbox-up  ## Teardown Flyte sandbox
-	$(call LOG,Tearing down Flyte sandbox)
-	flytectl sandbox teardown
-
-.PHONY: status
-status: _requires-sandbox-up  ## Show status of Flyte deployment
-	kubectl get pods -n flyte
+install-piptools:
+	pip install pip-tools
 
-.PHONY: shell
-shell: _requires-sandbox-up  ## Drop into a development shell
-	$(call RUN_IN_SANDBOX,bash)
+dev-requirements.txt: export CUSTOM_COMPILE_COMMAND := $(MAKE) dev-requirements.txt
+dev-requirements.txt: dev-requirements.in install-piptools
+	$(call PIP_COMPILE,dev-requirements.in)
 
-.PHONY: register
-register: _requires-sandbox-up  ## Register Flyte cookbook workflows
-	$(call LOG,Registering example workflows in cookbook/$(EXAMPLES_MODULE))
-	$(call RUN_IN_SANDBOX,make -C cookbook/$(EXAMPLES_MODULE) serialize)
-	make -C cookbook/$(EXAMPLES_MODULE) register
+.PHONY: dev-requirements
+dev-requirements: dev-requirements.txt
 
-.PHONY: fast_register
-fast_register: _requires-sandbox-up  ## Fast register Flyte cookbook workflows
-	$(call LOG,Fast registering example workflows from latest release of flytesnacks)
-	$(call RUN_IN_SANDBOX,make -C cookbook/$(EXAMPLES_MODULE) fast_serialize)
-	make -C cookbook/$(EXAMPLES_MODULE) fast_register
+docs-requirements.txt: export CUSTOM_COMPILE_COMMAND := $(MAKE) docs-requirements.txt
+docs-requirements.txt: docs-requirements.in install-piptools
+	$(call PIP_COMPILE,docs-requirements.in)
 
-.PHONY: setup-kubectl
-kubectl-config:
-	# In shell/bash, run: `eval $(make kubectl-config)`
-	# Makefiles run recipes in sub-processes. A sub-process cannot modify the parent process's environment.
-	# The best I (@EngHabu) can think of at the moment is to output this for the user to eval in the
-	# parent process.
-	echo "export KUBECONFIG=$(KUBECONFIG):~/.kube/config:$(FLYTE_DIR)/k3s/k3s.yaml"
+.PHONY: docs-requirements
+docs-requirements: docs-requirements.txt
Original file line number	Diff line number	Diff line change
		@@ -1,3 +1,2 @@
		This project is governed by LF AI Foundation's [code of conduct](https://lfprojects.org/policies/code-of-conduct/).
		All contributors and participants agree to abide by its terms.