diff --git a/.github/ISSUE_TEMPLATE/blank_issue_template.yml b/.github/ISSUE_TEMPLATE/blank_issue_template.yml new file mode 100644 index 0000000..521467d --- /dev/null +++ b/.github/ISSUE_TEMPLATE/blank_issue_template.yml @@ -0,0 +1,29 @@ +name: "Open a new issue" +description: "Have a question or an issue? Let us know here." +title: "Enter your issue title here" +labels: [] +body: + - type: markdown + attributes: + value: "## Welcome!" + - type: markdown + attributes: + value: >- + We value your input and are here to help. Please use this space to describe your question, + issue, or suggestion with as much detail as possible. Good descriptions include + specific steps to reproduce the issue, documentation of the observed versus expected behavior, + and any relevant screenshots or error messages. Your detailed account helps us to address + your concerns promptly and improve the `jdat-notebooks` for everyone. + - type: input + id: notebook-name + attributes: + label: "Notebook Name:" + placeholder: "Enter the name of the notebook here..." + validations: + required: true + - type: textarea + id: issue-description + attributes: + label: Description + validations: + required: true diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..f8ae7da --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,4 @@ +contact_links: + - name: Question/Help/Support + url: https://jwsthelp.stsci.edu + about: "JWST Help Desk" diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..3e025c2 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,54 @@ +This notebook checklist has been made available to us by the [the Notebooks For All team](https://github.com/Iota-School/notebooks-for-all/blob/main/resources/event-hackathon/notebook-authoring-checklist.md). +Its purpose is to serve as a guide for both the notebook author and the technical reviewer highlighting critical aspects to consider when striving to develop an accessible and effective notebook. + +### The First Cell + +- [ ] The title of the notebook in a first-level heading (eg. `

` or `# in markdown`). +- [ ] A brief description of the notebook. +- [ ] A table of contents in an [ordered list](https://www.markdownguide.org/basic-syntax/#ordered-lists) (`1., 2.,` etc. in Markdown). +- [ ] The author(s) and affiliation(s) (if relevant). +- [ ] The date first published. +- [ ] The date last edited (if relevant). +- [ ] A link to the notebook's source(s) (if relevant). + +### The Rest of the Cells + +- [ ] There is only one H1 (`#` in Markdown) used in the notebook. +- [ ] The notebook uses other [heading tags](https://www.markdownguide.org/basic-syntax/#headings) in order (meaning it does not skip numbers). + +## Text + +- [ ] All link text is descriptive. It tells users where they will be taken if they open the link. +- [ ] All acronyms are defined at least the first time they are used. +- [ ] Field-specific/specialized terms are used when needed, but not excessively. + +## Code + +- [ ] Code sections are introduced and explained before they appear in the notebook. This can be fulfilled with a heading in a prior Markdown cell, a sentence preceding it, or a code comment in the code section. +- [ ] Code has explanatory comments (if relevant). This is most important for long sections of code. +- [ ] If the author has control over the syntax highlighting theme in the notebook, that theme has enough color contrast to be legible. +- [ ] Code and code explanations focus on one task at a time. Unless comparison is the point of the notebook, only one method for completing the task is described at a time. + +### Images + +- [ ] All images (jpg, png, svgs) have an [image description](https://www.w3.org/WAI/tutorials/images/decision-tree/). This could be + - [ ] [Alt text](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/alt) (an `alt` property) + - [ ] [Empty alt text](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/alt#decorative_images) for decorative images/images meant to be skipped (an `alt` attribute with no value) + - [ ] Captions + - [ ] If no other options will work, the image is decribed in surrounding paragraphs. + +- [ ] Any [text present in images](https://www.w3.org/WAI/WCAG21/Understanding/images-of-text.html) exists in a text form outside of the image (this can be alt text, captions, or surrounding text.) + +### Visualizations + +- [ ] All visualizations have an image description. Review the previous section, Images, for more information on how to add it. +- [ ] [Visualization descriptions](http://diagramcenter.org/specific-guidelines-e.html) include + - [ ] The type of visualization (like bar chart, scatter plot, etc.) + - [ ] Title + - [ ] Axis labels and range + - [ ] Key or legend + - [ ] An explanation of the visualization's significance to the notebook (like the trend, an outlier in the data, what the author learned from it, etc.) + +- [ ] All visualizations and their parts have [enough color contrast](https://www.w3.org/WAI/WCAG21/Understanding/contrast-minimum.html) ([color contrast checker](https://webaim.org/resources/contrastchecker/)) to be legible. Remember that transparent colors have lower contrast than their opaque versions. +- [ ] All visualizations [convey information with more visual cues than color coding](https://www.w3.org/WAI/WCAG21/Understanding/use-of-color.html). Use text labels, patterns, or icons alongside color to achieve this. +- [ ] All visualizations have an additional way for notebook readers to access the information. Linking to the original data, including a table of the data in the same notebook, or sonifying the plot are all options. \ No newline at end of file diff --git a/.github/helpers/nb_flake8_magic.json b/.github/helpers/nb_flake8_magic.json new file mode 100644 index 0000000..f789b17 --- /dev/null +++ b/.github/helpers/nb_flake8_magic.json @@ -0,0 +1,43 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "

Reviewer note: Begin PEP8 check cells (delete below when finished)

" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# disable all imported packages' loggers\n", + "import logging\n", + "logging.root.manager.loggerDict = {}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# enable PEP8 checker for this notebook\n", + "%load_ext pycodestyle_magic\n", + "%flake8_on --ignore E261,E501,W291,W293\n", + "\n", + "# only allow the checker to throw warnings when there's a violation\n", + "logging.getLogger('flake8').setLevel('ERROR')\n", + "logging.getLogger('stpipe').setLevel('ERROR')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "

Reviewer note: Begin PEP8 check cells (delete above when finished)

" + ] + }] +} diff --git a/.github/helpers/pep8_nb_checker.py b/.github/helpers/pep8_nb_checker.py new file mode 100644 index 0000000..ca4b898 --- /dev/null +++ b/.github/helpers/pep8_nb_checker.py @@ -0,0 +1,182 @@ +import argparse +import copy +import json +import numpy as np +import pathlib +import pytz +import re +import subprocess +import sys + +from collections import defaultdict +from datetime import datetime as dt + +parser = argparse.ArgumentParser() +parser.add_argument('-f', '--file', type=str, + help='The notebook file to be checked') +args = parser.parse_args() + +nb_ext = '.ipynb' + +try: + nb_file = pathlib.Path(args.file) + if not nb_file.suffix == nb_ext: + raise ValueError(f"file extension must be {nb_ext}") + +except Exception as err: + parser.print_help() + raise err + +# create a separating line for the script file with unique text, like: +# #################################flake-8-check################################ +# (plus a closing newline to avoid W391 at end of file) +identifier = 'flake-8-check' +line_length = 80 +fill_0 = (line_length // 2 - 1) - np.floor(len(identifier) / 2).astype(int) +fill_1 = (line_length // 2 - 1) - np.ceil(len(identifier) / 2).astype(int) + +separator = '# ' + '#' * fill_0 + identifier + '#' * fill_1 + '\n' +buffer_lines = 3 # sections must end with >2 blank lines to avoid E302 + +# save relevant file paths +code_file = pathlib.Path(f"{nb_file.stem}_scripted.py") +warn_file = pathlib.Path(f"{nb_file.stem}_pep8.txt") +nb_magic_file = pathlib.Path(".github/helpers/nb_flake8_magic.json") + +def check_cell_content(source): + """Verify that a cell contains any content besides whitespace or newlines""" + for line in source: + if re.search('^(?!(?:\\n)$)', line): + # a "negative lookahead" for any characters in "non-capturing group" + return True + +# save code cell contents to a script divided into blocks with the separator +code_cells = [] +with open(nb_file) as nf: + og_nb = json.load(nf) + + with open(code_file, 'w') as cf: + for i, cl in enumerate(og_nb['cells']): + if (cl['cell_type'] == 'code' + and cl['source'] + and check_cell_content(cl['source']) + ): + # only check code cells containing actual code; skip blanks + code_cells.append(i) + + # check zeroth line for comment on errors to be ignored in this + # cell. if any, generate appropriate "noqa" comment + top_line = cl['source'][0] + noqa_check = re.search('^# flake8-ignore', top_line) + noqa_comment = ('' if noqa_check is None + else ' # noqa' + top_line[noqa_check.end():]) + + for ln in cl['source']: + # comment out lines with IPython magic commands + line = ln if ln[0] != '%' else '# ' + ln + + # insert noqa comment if needed (with care for newline char) + if (noqa_comment and not line.startswith('#') + and line != '\n' and line.endswith('\n')): + line = line[:-1] + noqa_comment + elif (noqa_comment and not line.startswith('#') + and line != '\n'): + line += noqa_comment[:-1] + + cf.write(line) + cf.write('\n' * buffer_lines) + cf.write(separator) + +# without spawning a shell, run flake8 and save any PEP8 warnings to a new file +with open(warn_file, 'w') as wf: + # flake8's command line options are specified in base repo folder's .flake8 + subprocess.run(['flake8', code_file], stdout=wf) + +# read in the PEP8 warnings +with open(warn_file) as wf: + warns = wf.readlines() + +# if there are none, QUIT while we're ahead +if not warns: + print(f"{nb_file} is clean!") + sys.exit() + +# else, read in the script and find the lines that function as cell borders +with open(code_file) as cf: + script = cf.readlines() + +borderlines = [j for j, ll in enumerate(script, start=1) # 1-indexed, like file + if re.search(fr"#+{identifier}#+", ll)] + +# customize the beginning of each PEP8 warning +pre = dt.now(pytz.timezone("America/New_York")).strftime('%Y-%m-%d %H:%M:%S - INFO - ') +# pre = 'INFO:pycodestyle:' +# pre = '' + +# create dict ready to take stderr dicts and append warning messages. the nested +# defaultdict guarantees a first-level key for each cell number needed, and a +# second-level list for appending 'text' strings, and nothing extra (w/o errors!) +stderr_shared = {'name': 'stderr', 'output_type': 'stream'} +nu_output_dict = defaultdict(lambda: defaultdict(list)) + +# match the warnings' line numbers to the notebook's cells with regex and math +for script_line in warns: + # get this warning's line number in the script + wrn = script_line[re.match(code_file.name, script_line).end():] + script_line_num = int(re.search(r'(?<=:)\d+(?=:)', wrn).group()) + + # translate it into cell numbers and then to intra-cell line number + code_cell_num = np.searchsorted(borderlines, script_line_num) + all_cell_num = code_cells[code_cell_num] + borderline_num = borderlines[code_cell_num - 1] if code_cell_num > 0 else 0 + next_borderline_num = borderlines[code_cell_num] + + if script_line_num != next_borderline_num: + line_in_cell = str(script_line_num - borderline_num) + else: + # correct line number for E303 by accounting for buffer added earlier + line_in_cell = str(script_line_num - borderline_num - buffer_lines) + + # print(f"--borderline:L{borderlines[code_cell_num - 1]}," + # f"next borderline:L{borderlines[code_cell_num]}," + # f"errorline:L{script_line_num}--") + # print(f"code_cell_num {code_cell_num}, line_in_cell {line_in_cell}, " + # f"all_cell_num {all_cell_num}") + + # only keep line/column info and warning from original flake8 text. + # prepend it with the customized string chosen earlier + nu_msg = pre + re.sub(r':\d+(?=:)', line_in_cell, wrn, count=1) + + # update the defaultdict + nu_output_dict[all_cell_num].update({'name': 'stderr', + 'output_type': 'stream'}) + nu_output_dict[all_cell_num]['text'].append(nu_msg) + +# use the defaultdict's keys to learn which cells require warnings +cells_to_edit = list(nu_output_dict.keys()) +injected_nb = copy.deepcopy(og_nb) + +for num, cell in enumerate(injected_nb['cells']): + # clear any cell output, regardless of PEP8 status + if cell.get('execution_count'): + cell['execution_count'] = None + if cell.get('outputs'): + cell['outputs'] = [] + + # inject PEP8 warnings into cells marked earlier + if num in cells_to_edit: + cell['outputs'] = [nu_output_dict[num]] + +# insert cells for enabling interactive PEP8 feedback just above first code cell +# if they aren't already present +with open(nb_magic_file) as nmf: + flake8_magic_cells = json.load(nmf)['cells'] + +if all([og_nb['cells'][i].get('source') != flake8_magic_cells[0]['source'] + for i in code_cells]): + injected_nb['cells'][code_cells[0]:code_cells[0]] = flake8_magic_cells + +# save the edited notebook +with open(nb_file, 'w') as file: + json.dump(injected_nb, file, indent=1, ensure_ascii=False) + file.write("\n") # end with new line since json.dump doesn't diff --git a/.github/helpers/tech_review_instructions.md b/.github/helpers/tech_review_instructions.md new file mode 100644 index 0000000..58d0aeb --- /dev/null +++ b/.github/helpers/tech_review_instructions.md @@ -0,0 +1,44 @@ +Hello @%s, + +Thank you for submitting these changes to your notebook. Please read on for your technical review instructions. + +## Before you begin + +The technical review helps ensure that contributed notebooks a) run from top to bottom, b) follow the [PEP8 standards](https://www.python.org/dev/peps/pep-0008/) for Python code readability, and c) conform to the Institute's [style guide](https://github.com/spacetelescope/style-guides/blob/master/guides/jupyter-notebooks.md) for Jupyter Notebooks. + +I've pushed the review as a new commit in this pull request. **To view and edit the commit locally, follow these steps:** + +``` +git checkout %s +git fetch YOUR-REMOTE-FORK +git merge YOUR-REMOTE-FORK/%s +``` + +_(`YOUR-REMOTE-FORK` is your fork's online copy. It's often `origin`, but if you don't know your name for it, run `git remote -v` and choose the one whose path ends with `%s/%s.git`.)_ + +From here you can work on your branch as normal. If you have trouble with this step, please let me know before continuing. + +--- + +## Instructions + +After updating your local copy of this branch, **please open your notebook and address any warnings or errors you find**. + +If you see cells with output like this, it means some of your code doesn't follow the PEP8 standards of code readability: + +image + +_(In the example above, `INFO - 3:3: E111` means that the text entered on line 3 at index 3 caused the warning "E111". The violation is briefly described at the end of the message.)_ + +You can test that your edits satisfy the standard by installing `flake8` on the command line with: +``` +pip install flake8==3.9.2 pycodestyle_magic +``` + +Then, restart the notebook and run the following cells: + +image + +After that, edit and re-run cells with warnings until you've fixed all of them. **Please remember to delete the cells shown in the above image before pushing your changes back to this pull request.** + +**If you have questions or feedback on specific cells, click the earlier message in this thread from the "review-notebook-app" bot.** There, you can comment on specific cells and view what's changed in the new commit. I may also write comments there. Anything posted there will also be reflected in this pull request's conversational thread. diff --git a/.github/scripts/insert_failure_message.py b/.github/scripts/insert_failure_message.py new file mode 100644 index 0000000..2221e5f --- /dev/null +++ b/.github/scripts/insert_failure_message.py @@ -0,0 +1,16 @@ +# .github/scripts/insert_failure_message.py +# This script inserts a message into the generated HTML to let users know that the notebook currently fails execution +import nbformat as nbf +import sys + +notebook_filename = sys.argv[1] + +with open(notebook_filename, "r") as f: + nb = nbf.read(f, as_version=4) + +new_cell = nbf.v4.new_markdown_cell('This notebook currently fails to execute, use as reference only') +# Insert the cell at the second position in the notebook +nb.cells.insert(1, new_cell) + +with open(notebook_filename, "w") as f: + nbf.write(nb, f) diff --git a/.github/workflows/ci_buildondemand.yml b/.github/workflows/ci_buildondemand.yml new file mode 100644 index 0000000..07caeb1 --- /dev/null +++ b/.github/workflows/ci_buildondemand.yml @@ -0,0 +1,9 @@ +name: Manual Execute All Notebooks +on: + workflow_dispatch: + +jobs: + ExecuteNotebooks: + uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_scheduled.yml@v3 + with: + python-version: ${{ vars.PYTHON_VERSION }} diff --git a/.github/workflows/ci_execute_merge_generate.yml b/.github/workflows/ci_execute_merge_generate.yml new file mode 100644 index 0000000..ee04517 --- /dev/null +++ b/.github/workflows/ci_execute_merge_generate.yml @@ -0,0 +1,9 @@ +name: Manual Full Archive Execute-Store-Generate HTML +on: + workflow_dispatch: + +jobs: + GenerateHTML: + uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_build_merge_manual.yml@v3 + with: + python-version: ${{ vars.PYTHON_VERSION }} diff --git a/.github/workflows/ci_html_build.yml b/.github/workflows/ci_html_build.yml index cb88f21..bcfa7aa 100644 --- a/.github/workflows/ci_html_build.yml +++ b/.github/workflows/ci_html_build.yml @@ -1,4 +1,4 @@ -name: Build HTML +name: Build HTML on merge on: pull_request: branches: @@ -8,5 +8,11 @@ on: workflow_dispatch: jobs: - GenerateHTML: - uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_builder.yml@main + Generate_HTML: + if: github.event.pull_request.merged == true + uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_builder.yml@v3 + with: + python-version: ${{ vars.PYTHON_VERSION }} + # permissions: + # contents: write + diff --git a/.github/workflows/manual_html_build.yml b/.github/workflows/ci_manual_html_deploy.yml similarity index 51% rename from .github/workflows/manual_html_build.yml rename to .github/workflows/ci_manual_html_deploy.yml index 1fc623e..00ada86 100644 --- a/.github/workflows/manual_html_build.yml +++ b/.github/workflows/ci_manual_html_deploy.yml @@ -1,7 +1,10 @@ -name: Manual Build HTML +name: Manual HTML Deploy + on: workflow_dispatch: jobs: GenerateHTML: - uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_builder_manual.yml@main + uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_builder_manual.yml@v3 + with: + python-version: ${{ vars.PYTHON_VERSION }} diff --git a/.github/workflows/ci_scheduled.yml b/.github/workflows/ci_nightly.yml similarity index 56% rename from .github/workflows/ci_scheduled.yml rename to .github/workflows/ci_nightly.yml index f452745..c780b6a 100644 --- a/.github/workflows/ci_scheduled.yml +++ b/.github/workflows/ci_nightly.yml @@ -1,8 +1,11 @@ name: Scheduled Notebook Execution on: schedule: + #- cron: '0 3 * * *' # run at 2 AM UTC - cron: '0 0 * * 0' # midnight sunday UTC jobs: - ScheduledBuild: - uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_scheduled.yml@main \ No newline at end of file + Scheduled: + uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_scheduled.yml@v3 + with: + python-version: ${{ vars.PYTHON_VERSION }} diff --git a/.github/workflows/ci_runner.yml b/.github/workflows/ci_runner.yml index e87690d..8a30f96 100644 --- a/.github/workflows/ci_runner.yml +++ b/.github/workflows/ci_runner.yml @@ -16,10 +16,18 @@ on: - 'notebooks/**.ipynb' - '*.yml' -env: - CASJOBS_PW: ${{ secrets.CASJOBS_PW }} - CASJOBS_USERID: ${{ secrets.CASJOBS_USERID }} +#permissions: +# contents: write +# id-token: write +# issues: write +# pull-requests: write + jobs: NotebookExecutionValidation: - uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_runner.yml@main + uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_runner.yml@v3 + with: + python-version: ${{ vars.PYTHON_VERSION }} + permissions: + contents: write + diff --git a/.github/workflows/greetings.yml b/.github/workflows/greetings.yml.tmp similarity index 99% rename from .github/workflows/greetings.yml rename to .github/workflows/greetings.yml.tmp index 9f5c346..88bec98 100644 --- a/.github/workflows/greetings.yml +++ b/.github/workflows/greetings.yml.tmp @@ -1,5 +1,4 @@ name: Greetings - on: [pull_request, issues] jobs: diff --git a/.github/workflows/manual_execution.yml b/.github/workflows/manual_execution.yml deleted file mode 100644 index d8b07ba..0000000 --- a/.github/workflows/manual_execution.yml +++ /dev/null @@ -1,7 +0,0 @@ -name: Manual Notebook Execution -on: - workflow_dispatch: - -jobs: - ScheduledBuild: - uses: spacetelescope/notebook-ci-actions/.github/workflows/ci_scheduled.yml@main diff --git a/.github/workflows/notebook-pep8-check.yml.old b/.github/workflows/notebook-pep8-check.yml.old new file mode 100644 index 0000000..553d203 --- /dev/null +++ b/.github/workflows/notebook-pep8-check.yml.old @@ -0,0 +1,246 @@ +name: notebook-pep8-check +on: + # use pull_request_target instead of pull_request to allow pushes to PR branch + pull_request_target: + branches: + - main + paths: + - 'notebooks/*/*.ipynb' + types: [ labeled, opened, synchronize, reopened ] + +# looks like either each folder needs its own job or pull requests need to be +# limited to one folder at a time, otherwise requirements files will be mixed. + +env: + pep8_script: '.github/helpers/pep8_nb_checker.py' + magic_cells: '.github/helpers/nb_flake8_magic.json' + instruct_post: '.github/helpers/tech_review_instructions.md' + flake8_config: '.flake8' + bot_name: 'github-actions[bot]' + bot_email: '41898282+github-actions[bot]@users.noreply.github.com' + +permissions: + contents: write + pull-requests: write + # GITHUB_TOKEN cannot push to PR branch without these permissions + +jobs: + do-notebook-pep8-check: + if: | + contains(github.event.pull_request.labels.*.name, 'Technical Review') && + github.event.pull_request.head.user.login != 'github-actions[bot]' && + github.event.pull_request.draft == false && + github.event.pull_request.state == 'open' + + runs-on: ubuntu-latest + + steps: + - name: Event familiarization + run: | + echo "---github.event.pull_request.draft---" + echo ${{ github.event.pull_request.draft }} + + echo "---github.event.pull_request.state---" + echo ${{ github.event.pull_request.state }} + + echo "---contains(github.event.pull_request.labels.*.name, 'Technical Review')---" + echo ${{ contains(github.event.pull_request.labels.*.name, 'Technical Review') }} + + echo "---github.event.pull_request.user.login---" + echo ${{ github.event.pull_request.user.login }} + + echo "---github.event.pull_request.author_association---" + echo ${{ github.event.pull_request.author_association }} + + echo "---github.event.pull_request.head.ref---" + echo ${{ github.event.pull_request.head.ref }} + + echo "---github.event.pull_request.head.sha---" + echo ${{ github.event.pull_request.head.sha }} + + echo "---github.event.pull_request.base.ref---" + echo ${{ github.event.pull_request.base.ref }} + + echo "---github.event.pull_request.base.sha---" + echo ${{ github.event.pull_request.base.sha }} + # this is the latest commit on the base/target branch; NOT + # (necessarily) the commit from which the PR diverges! + + echo "---github.event.action---" + echo ${{ github.event.action }} + + echo "---github.event.sender.login---" + echo ${{ github.event.sender.login }} + + - name: Check out PR branch + uses: actions/checkout@v3 + with: + repository: ${{github.event.pull_request.head.repo.full_name}} + ref: ${{ github.event.pull_request.head.ref }} + fetch-depth: 0 + # (0 fetches all branches even with ref specified... don't love + # fetching everything, but don't see how else to pull base + PR) + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Get changed files + run: | + # for clarity, add an upstream remote and fetch the base/target branch + # ('origin' after checkout step is the PR branch's remote. + # if the PR branch is from a fork, 'upstream' is needed to fetch the + # base branch. if the PR and base branches share a remote, 'origin' + # and 'upstream' will point to the same repository, which is fine.) + + echo "---add upstream remote---"; + git remote add upstream "https://github.com/${{ github.event.pull_request.base.user.login }}/${{ github.event.repository.name }}.git"; + git remote -v + + echo "---fetch upstream base branch---"; + git fetch upstream ${{ github.event.pull_request.base.ref }} + + # find last common ancestor commit between target and PR branches + + last_common=$(git merge-base upstream/${{ github.event.pull_request.base.ref }} ${{ github.event.pull_request.head.ref }}) + echo "last_common" + echo $last_common + + # make an array of all notebooks added/copied/modified in PR branch + + git_diff=$(git diff --name-only --diff-filter=ACM $last_common ${{ github.event.pull_request.head.sha }} *ipynb) + echo "git_diff" + echo $git_diff + + # (yes to added, copied, modified and NO to renamed, deleted, type + # changed, unmerged, or unknown seems reasonable.) + + changed_files=($(printf '%s\n' $git_diff)) + echo "changed_files" + echo "${changed_files[@]}" + echo "changed_files=${changed_files[@]}" >> $GITHUB_ENV + + # use arrays to check that changed notebooks are in the same directory + # (note that xargs dirname can only be used in this way on Linux) + # (https://unix.stackexchange.com/a/377820 for unique check) + + changed_dirs_all=( $(printf '%s\n' "${changed_files[@]}" | xargs dirname) ) + echo "changed_dirs_all" + echo "${changed_dirs_all[@]}" + + changed_dirs_uniq=( $(printf '%s\n' "${changed_dirs_all[@]}" | sort -u) ) + echo "changed_dirs_uniq" + echo "${changed_dirs_uniq[@]}" + + if [ "${#changed_dirs_uniq[@]}" != 1 ]; then echo "one_dir=no" >> $GITHUB_ENV; fi + + - name: Auto-fail if changed files are not in same directory + if: env.one_dir + uses: actions/github-script@v5 + with: + script: | + core.setFailed('All changed files are not in the same directory.') + + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: 3.8 # JDAT recommends 3.8.10 + + - name: Install flake8 and script dependencies + if: steps.cache-step.outputs.cache-hit != 'true' + run: | + python -m pip install --upgrade pip + pip install flake8 numpy pytz + + - name: Run the PEP8 check's Python script + run: | + # ensure script's helper files come from the base branch, not the PR + # (checkout also works, though you have to unstage the file afterward) + + git restore --source=upstream/${{ github.event.pull_request.base.ref }} --worktree $pep8_script $magic_cells $instruct_post $flake8_config + + # isolate changed notebooks in order to run PEP8 check + + for ff in ${{ env.changed_files }}; do + # https://stackoverflow.com/a/6397979 + if [[ $ff == *".ipynb" ]]; then + python ${{ env.pep8_script }} -f $ff; + fi + done + + # check for changes to the notebooks in this local branch + if [[ `git status --porcelain --untracked-files=no` ]]; then echo "push=yes" >> $GITHUB_ENV; fi + shell: bash + + # if env.push is empty, the next steps are skipped and the job should pass + - name: Push any changes to PR + if: env.push + run: | + # verify identity to allow a push to which other workflows can respond + git config --local user.email $bot_email + git config --local user.name $bot_name + + # add the changed notebook(s) + git add --update *.ipynb + git status + # (script output files should be untracked) + + # list changed notebooks in commit message; push the changes + git_diff_new=($(git diff --name-only --diff-filter=M *ipynb)) + git commit -m "[BOT] Left PEP8 feedback on PR ${{ github.event.number }}'s notebooks" -m "Files: $(printf '%s\n' $git_diff_new)" + # (is tagging the PR for the best here?) + + git push origin "HEAD:${{ github.event.pull_request.head.ref }}" + + - name: Write instructional message (if new branch) + if: env.push + uses: actions/github-script@v5 + with: + script: | + async function scanAndPost() + { + const template = "${{ env.instruct_post }}"; + const branch = "${{ github.event.pull_request.head.ref }}"; + const creator = "${{ github.event.pull_request.user.login }}"; + const issueno = context.issue.number + const { owner, repo } = context.repo; + + // scan PR comments (expecting JS array of JS lists) + const comments = await github.rest.issues.listComments({ + owner: owner, // as in account that owns the repo + repo: repo, + issue_number: issueno + }); + console.log("imported comments") + //console.log(comments.data[0].user); + + // see how many were written by the bot + const botPosts = comments.data.filter(post => post.user.login === "${{ env.bot_name }}").length; + + // if any, terminate early + if (botPosts != 0) { console.log("Already commented!"); return; } + + // else, read text from template file + const fs = require("fs").promises; + const message = await fs.readFile(template, "utf8"); + console.log("read file"); + + // format message with user and PR info + const util = require("util"); + const injectedMsg = util.format(message, creator, branch, + branch, creator, repo); + console.log("formatted message"); + + // create the comment + github.rest.issues.createComment({ + issue_number: issueno, + owner: owner, + repo: repo, + body: injectedMsg + }); + // could label with github.rest.issues.addLabels... + } + scanAndPost(); + + - name: Auto-fail if changes were pushed + if: env.push + uses: actions/github-script@v5 + with: + script: core.setFailed('Notebook(s) failed PEP8 tests.') diff --git a/.github/workflows/pep8_nb_style_check.yml b/.github/workflows/pep8_nb_style_check.yml new file mode 100644 index 0000000..a9ff39a --- /dev/null +++ b/.github/workflows/pep8_nb_style_check.yml @@ -0,0 +1,14 @@ +## This workflow performs PEP8 style checks on the python code in notebook code cells. +name: PEP8 Notebook Style Check Execution + +on: + pull_request: + branches: + - main + paths: + - 'notebooks/**.ipynb' + - '*.yml' + +jobs: + Notebook_PEP8_Check: + uses: spacetelescope/notebook-ci-actions/.github/workflows/notebook_pep8check.yml@main diff --git a/.github/workflows/pep8_script_style_check.yml b/.github/workflows/pep8_script_style_check.yml new file mode 100644 index 0000000..da6910d --- /dev/null +++ b/.github/workflows/pep8_script_style_check.yml @@ -0,0 +1,14 @@ +## This workflow performs PEP8 style checks on the python code in notebook code cells. +name: PEP8 Script Style Check Execution + +on: + pull_request: + branches: + - main + paths: + - 'notebooks/**.py' + - '*.yml' + +jobs: + Script_PEP8_Check: + uses: spacetelescope/notebook-ci-actions/.github/workflows/script_pep8check.yml@main diff --git a/.github/workflows/weekly_broken_link_finder.yml b/.github/workflows/weekly_broken_link_finder.yml new file mode 100644 index 0000000..4ae3c13 --- /dev/null +++ b/.github/workflows/weekly_broken_link_finder.yml @@ -0,0 +1,10 @@ +name: Weekly broken link check +on: + schedule: + - cron: '0 4 * * 0' # 0400 UTC every Sunday + +jobs: + Scheduled: + uses: spacetelescope/notebook-ci-actions/.github/workflows/broken_link_checker.yml@main + with: + website_url: "https://spacetelescope.github.io/jdat_notebooks/" diff --git a/README.md b/README.md index 938142e..1efb9b2 100644 --- a/README.md +++ b/README.md @@ -1,3 +1 @@ - -# notebook_ci_template -Structural template for notebook CI system +# roman_notebooks diff --git a/_config.yml b/_config.yml index 5106ad8..efe5e26 100644 --- a/_config.yml +++ b/_config.yml @@ -1,40 +1,51 @@ -# Book settings -# Learn more at https://jupyterbook.org/customize/config.html +####################################################################################### +# A default configuration that will be loaded for all jupyter books +# See the documentation for help and more options: +# https://jupyterbook.org/customize/config.html -title: Notebook repository template (CHANGE THIS) -author: STScI (CHANGE AS NEEDED) -logo: logo.png (CHANGE AS NEEDED) +####################################################################################### +# Book settings +title : STScI Roman Notebooks # The title of the book. Will be placed in the left navbar. +author : STScI # The author of the book +copyright : "2024" # Copyright year to be placed in the footer +logo : stsci_logo2.png # A path to the book logo -html: - google_analytics_id: - # Force re-execution of notebooks on each build. # See https://jupyterbook.org/content/execute.html execute: - execute_notebooks: force + execute_notebooks : force # Whether to execute notebooks at build time. Must be one of ("auto", "force", "cache", "off") + timeout : -1 # timeout in seconds, -1 removes timeout restrictions + only_build_toc_files: true # Define the name of the latex output file for PDF builds latex: latex_documents: targetname: book.tex - + +# Define the name of the latex output file for PDF builds +parse: + myst_enable_extensions: + # don't forget to list any other extensions you want enabled, + # including those that are enabled by default! + - html_image + # Add a bibtex file so that we can create citations -bibtex_bibfiles: - - references.bib +#bibtex_bibfiles: +# - references.bib launch_buttons: thebe : true - colab : true - colab_url: "https://colab.research.google.com" - + binderhub_url: "" + # Information about where the book exists on the web repository: - url: (CHANGE THIS) # Online location of your book - # path_to_book: Optional path to your book, relative to the repository root + url: https://github.com/spacetelescope/roman_notebooks/ # Online location of your book + path_to_book: notebooks # Optional path to your book, relative to the repository root branch: main # Which branch of the repository should be used when creating links (optional) # Add GitHub buttons to your book # See https://jupyterbook.org/customize/config.html#add-a-link-to-your-repository html: + google_analytics_id: use_issues_button: true use_repository_button: true diff --git a/_static/table.css b/_static/main.css similarity index 99% rename from _static/table.css rename to _static/main.css index 9fca7ee..24b75fe 100644 --- a/_static/table.css +++ b/_static/main.css @@ -12,4 +12,3 @@ div[class^='highlight'] pre { overflow-y: scroll; overflow-x: scroll; } - diff --git a/_toc.yml b/_toc.yml index acf89e7..c425651 100644 --- a/_toc.yml +++ b/_toc.yml @@ -1,6 +1,8 @@ # Table of contents # Learn more at https://jupyterbook.org/customize/toc.html format: jb-book -root: index -chapters: -- file: notebooks/example_notebook/example_notebook.ipynb +root: intro +parts: + - caption: Example + chapters: + - file: notebooks/example_notebook/example_notebook.ipynb diff --git a/conf.py b/conf.py new file mode 100644 index 0000000..abaacaa --- /dev/null +++ b/conf.py @@ -0,0 +1,38 @@ +############################################################################### +# Auto-generated by `jupyter-book config` +# If you wish to continue using _config.yml, make edits to that file and +# re-generate this one. +############################################################################### +author = 'Captain Jupyter' +bibtex_bibfiles = ['references.bib'] +comments_config = {'hypothesis': False, 'utterances': False} +copyright = '2022' +exclude_patterns = ['**.ipynb_checkpoints', '.DS_Store', 'Thumbs.db', '_build'] +execution_allow_errors = False +execution_excludepatterns = [] +execution_in_temp = False +execution_timeout = 30 +extensions = ['sphinx_togglebutton', 'sphinx_copybutton', 'myst_nb', 'jupyter_book', 'sphinx_thebe', 'sphinx_comments', 'sphinx_external_toc', 'sphinx.ext.intersphinx', 'sphinx_panels', 'sphinx_book_theme', 'sphinxcontrib.bibtex', 'sphinx_jupyterbook_latex'] +external_toc_exclude_missing = False +external_toc_path = '_toc.yml' +html_baseurl = '' +html_favicon = '' +html_logo = 'Jspectra.svg' +html_sourcelink_suffix = '' +html_static_path = ['_static'] +html_theme = 'sphinx_book_theme' +html_theme_options = {'search_bar_text': 'Search this book...', 'launch_buttons': {'notebook_interface': 'classic', 'binderhub_url': 'https://mybinder.org', 'jupyterhub_url': '', 'thebe': True, 'colab_url': 'https://colab.research.google.com', 'colab': True}, 'path_to_docs': 'notebooks', 'repository_url': 'https://github.com/therealzoidberg/actions-ci-testing', 'repository_branch': 'main', 'google_analytics_id': '', 'extra_navbar': 'Powered by Jupyter Book', 'extra_footer': '
\n
\n \n \n
\n
\n', 'home_page_in_toc': True, 'use_repository_button': True, 'use_edit_page_button': False, 'use_issues_button': True} +html_title = 'STScI JDAT Notebooks' +jupyter_cache = '' +jupyter_execute_notebooks = 'force' +language = None +latex_engine = 'pdflatex' +myst_enable_extensions = ['colon_fence', 'dollarmath', 'linkify', 'substitution', 'tasklist'] +myst_url_schemes = ['mailto', 'http', 'https'] +nb_output_stderr = 'show' +numfig = True +panels_add_bootstrap_css = False +pygments_style = 'sphinx' +suppress_warnings = ['myst.domains'] +use_jupyterbook_latex = True +use_multitoc_numbering = True diff --git a/docs/about.md b/docs/about.md deleted file mode 100644 index 8dba1b1..0000000 --- a/docs/about.md +++ /dev/null @@ -1,49 +0,0 @@ -# What does this CI system do? -There are a few different actions that provide different CI functionalty. -- Validation/Execution - ci_validation.yml - - This workflow exercises the notebook to validate that it properly executes without failure, and validates the output. - - Execution of the notebook is accomplished through the use of the jupyter utility 'nbconvert' - - https://nbconvert.readthedocs.io/en/latest/ - - Validation is performed using pytest and a pytest plugin 'nbval' - - https://nbval.readthedocs.io/en/latest/ -- HTML Builder/Deployment - - Static HTML pages of rendered/executed notebooks are deployed to Github pages. - - The workflow uses "Jupyter-Book" as a backend to perform the rendering, the page inclusion as well as the overall look and feel. - - https://jupyterbook.org/en/stable/intro.html -- Nightly/Weekly Builds - - Notebooks are exercised on a scheduled cycle that can be defined in the workflow. The nightly build performs the validation and execution steps as the validation workflow, but does not regenerate HTML. - - The workflow breaks up notebooks into a matrix and executes each individually on its own worker node, errors are reported in a summary under the actions tab of the repository as well as emailed to the watchers. - - The intent is to capture issues regarding updates to requirements that cause notebook failures or any other transient environmental issues. - -## Getting started -The easiest method for creating a new repository with the CI build machinery is to use the deployable template. This will provide the required directory structure support configuration files required to operate the CI system. -- Navigate to https://github.com/TheRealZoidberg/notebook_ci_template (This will eventually be moved to STScI repos) -- On the top right side of the page, select the green button labeled "Use this template" -- Select the repository 'owner' (Your own personal github repo for example) and enter a new repository name. -- ** It is important to select "Include all branches" as github pages requires its own documents branch. -- Finally select the "Create repository from template" button. A new repository will be created with all the required files/structure. - -### Configuration -Now that a new repository has been created, there are a few more steps to finish. -- Jupyter-book configuration: _config.yml - - This configuration file contains all of the variables to customize the rendered HTML pages. The entries that should be edited are marked with '(CHANGE THIS)'. - - General settings: - - Title: The title of the notebook collection - - author: STScI or a specific author to the notebook collection - - logo: log.png - a logo that displays the mission or a special feature that defines the collection. - - execute: - - execute_notebooks: force - defines how and when notebooks are executed during HTML generation - See https://jupyterbook.org/content/execute.html - - html: - - google_analytics_id: - applies a google analytics tracking ID to every html page generated - - use_issues_button: true or false - Adds a button to report an issue directly to github - - use_repository_button: true or false - Adds a button that allows the user to directly visit the repository for the notebooks - - launch_buttons: - - thebe: true or false - Adds a button that implements thebe into the notebook, this uses a mybinder kernel in the background to allow cell execution from the static pages. - - colab: true or false - Adds a button that opens the notebook in Google colab - allowing the user to fully run the notebook on an active node. - -- Notebook Requirements (This section is currently under development, and may change) - - Typically when a notebook is added to the repository, it will come with a developer created 'requirements.txt' file. When there is a large collection of notebooks the contents of these files can contain many dupilcates of requirements but potentially all with different versions. - - When running in a CI environment it is important to keep a consistant environment. - - To do this there is a script included in the template that scans for notebooks and uses a utility 'pipreqsnb' to extract only the needed requirements from each notebook. It then creates a global requirements file in the repository root that contains all of the unique requirements from the collection, unpinned from versions, ensuring that the latest releases are used. - - To run the script, first 'pip install pipreqsnb' - and then execute the script './reqBuilder.sh' - diff --git a/docs/readme.rst b/docs/readme.rst deleted file mode 100644 index f59e40d..0000000 --- a/docs/readme.rst +++ /dev/null @@ -1 +0,0 @@ -This is the deployable template for the creation of a jupyter notebook continuious integration system that uses Github actions. diff --git a/index.rst b/index.rst deleted file mode 100644 index baaf37f..0000000 --- a/index.rst +++ /dev/null @@ -1,3 +0,0 @@ -# STScI Notebook Repository Template - -This is the template that should be used to structure your notebook repository to worth with the STScI notebook CI system. diff --git a/intro.md b/intro.md index baaf37f..10dc919 100644 --- a/intro.md +++ b/intro.md @@ -1,3 +1,3 @@ -# STScI Notebook Repository Template +### Intro to repository -This is the template that should be used to structure your notebook repository to worth with the STScI notebook CI system. +## Fill out repo information here diff --git a/markdown.md b/markdown.md deleted file mode 100644 index 5d5ae76..0000000 --- a/markdown.md +++ /dev/null @@ -1,125 +0,0 @@ -# Markdown Files - -Whether you write your book's content in Jupyter Notebooks (`.ipynb`) or -in regular markdown files (`.md`), you'll write in the same flavor of markdown -called **MyST Markdown**. - -## What is MyST? - -MyST stands for "Markedly Structured Text". It -is a slight variation on a flavor of markdown called "CommonMark" markdown, -with small syntax extensions to allow you to write **roles** and **directives** -in the Sphinx ecosystem. - -## What are roles and directives? - -Roles and directives are two of the most powerful tools in Jupyter Book. They -are kind of like functions, but written in a markup language. They both -serve a similar purpose, but **roles are written in one line**, whereas -**directives span many lines**. They both accept different kinds of inputs, -and what they do with those inputs depends on the specific role or directive -that is being called. - -### Using a directive - -At its simplest, you can insert a directive into your book's content like so: - -```` -```{mydirectivename} -My directive content -``` -```` - -This will only work if a directive with name `mydirectivename` already exists -(which it doesn't). There are many pre-defined directives associated with -Jupyter Book. For example, to insert a note box into your content, you can -use the following directive: - -```` -```{note} -Here is a note -``` -```` - -This results in: - -```{note} -Here is a note -``` - -In your built book. - -For more information on writing directives, see the -[MyST documentation](https://myst-parser.readthedocs.io/). - - -### Using a role - -Roles are very similar to directives, but they are less-complex and written -entirely on one line. You can insert a role into your book's content with -this pattern: - -``` -Some content {rolename}`and here is my role's content!` -``` - -Again, roles will only work if `rolename` is a valid role's name. For example, -the `doc` role can be used to refer to another page in your book. You can -refer directly to another page by its relative path. For example, the -role syntax `` {doc}`intro` `` will result in: {doc}`intro`. - -For more information on writing roles, see the -[MyST documentation](https://myst-parser.readthedocs.io/). - - -### Adding a citation - -You can also cite references that are stored in a `bibtex` file. For example, -the following syntax: `` {cite}`holdgraf_evidence_2014` `` will render like -this: {cite}`holdgraf_evidence_2014`. - -Moreover, you can insert a bibliography into your page with this syntax: -The `{bibliography}` directive must be used for all the `{cite}` roles to -render properly. -For example, if the references for your book are stored in `references.bib`, -then the bibliography is inserted with: - -```` -```{bibliography} -``` -```` - -Resulting in a rendered bibliography that looks like: - -```{bibliography} -``` - - -### Executing code in your markdown files - -If you'd like to include computational content inside these markdown files, -you can use MyST Markdown to define cells that will be executed when your -book is built. Jupyter Book uses *jupytext* to do this. - -First, add Jupytext metadata to the file. For example, to add Jupytext metadata -to this markdown page, run this command: - -``` -jupyter-book myst init markdown.md -``` - -Once a markdown file has Jupytext metadata in it, you can add the following -directive to run the code at build time: - -```` -```{code-cell} -print("Here is some code to execute") -``` -```` - -When your book is built, the contents of any `{code-cell}` blocks will be -executed with your default Jupyter kernel, and their outputs will be displayed -in-line with the rest of your content. - -For more information about executing computational content with Jupyter Book, -see [The MyST-NB documentation](https://myst-nb.readthedocs.io/). diff --git a/notebooks/example_notebook/example_notebook.ipynb b/notebooks/example_notebook/example_notebook.ipynb index 6b2aee2..a4652f4 100644 --- a/notebooks/example_notebook/example_notebook.ipynb +++ b/notebooks/example_notebook/example_notebook.ipynb @@ -30,12 +30,12 @@ } }, "source": [ - "## Imports\n", + "## Imports\n ", "Describe the libraries we're using here. If there's something unusual, explain what the library is, and why we need it.\n", "- *numpy* to handle array functions\n", "- *astropy.io fits* for accessing FITS files\n", "- *astropy.table Table* for creating tidy tables of the data\n", - "- *matplotlib.pyplot* for plotting data" + "- *matplotlib.pyplot* for plotting data " ] }, { diff --git a/notebooks/example_notebook/requirements.txt b/notebooks/example_notebook/requirements.txt index 24e68af..e60aa81 100644 --- a/notebooks/example_notebook/requirements.txt +++ b/notebooks/example_notebook/requirements.txt @@ -1,4 +1,4 @@ -astropy==5.1 -astroquery==0.4.6 -matplotlib==3.6.0 -numpy==1.23.4 +numpy>=1.16.4 +astroquery>=0.3.10 +astropy>=3.2.1 +matplotlib>=3.1.1 diff --git a/references.bib b/references.bib deleted file mode 100644 index 783ec6a..0000000 --- a/references.bib +++ /dev/null @@ -1,56 +0,0 @@ ---- ---- - -@inproceedings{holdgraf_evidence_2014, - address = {Brisbane, Australia, Australia}, - title = {Evidence for {Predictive} {Coding} in {Human} {Auditory} {Cortex}}, - booktitle = {International {Conference} on {Cognitive} {Neuroscience}}, - publisher = {Frontiers in Neuroscience}, - author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Knight, Robert T.}, - year = {2014} -} - -@article{holdgraf_rapid_2016, - title = {Rapid tuning shifts in human auditory cortex enhance speech intelligibility}, - volume = {7}, - issn = {2041-1723}, - url = {http://www.nature.com/doifinder/10.1038/ncomms13654}, - doi = {10.1038/ncomms13654}, - number = {May}, - journal = {Nature Communications}, - author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Rieger, Jochem W. and Crone, Nathan and Lin, Jack J. and Knight, Robert T. and Theunissen, Frédéric E.}, - year = {2016}, - pages = {13654}, - file = {Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:C\:\\Users\\chold\\Zotero\\storage\\MDQP3JWE\\Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:application/pdf} -} - -@inproceedings{holdgraf_portable_2017, - title = {Portable learning environments for hands-on computational instruction using container-and cloud-based technology to teach data science}, - volume = {Part F1287}, - isbn = {978-1-4503-5272-7}, - doi = {10.1145/3093338.3093370}, - abstract = {© 2017 ACM. There is an increasing interest in learning outside of the traditional classroom setting. This is especially true for topics covering computational tools and data science, as both are challenging to incorporate in the standard curriculum. These atypical learning environments offer new opportunities for teaching, particularly when it comes to combining conceptual knowledge with hands-on experience/expertise with methods and skills. Advances in cloud computing and containerized environments provide an attractive opportunity to improve the effciency and ease with which students can learn. This manuscript details recent advances towards using commonly-Available cloud computing services and advanced cyberinfrastructure support for improving the learning experience in bootcamp-style events. We cover the benets (and challenges) of using a server hosted remotely instead of relying on student laptops, discuss the technology that was used in order to make this possible, and give suggestions for how others could implement and improve upon this model for pedagogy and reproducibility.}, - booktitle = {{ACM} {International} {Conference} {Proceeding} {Series}}, - author = {Holdgraf, Christopher Ramsay and Culich, A. and Rokem, A. and Deniz, F. and Alegro, M. and Ushizima, D.}, - year = {2017}, - keywords = {Teaching, Bootcamps, Cloud computing, Data science, Docker, Pedagogy} -} - -@article{holdgraf_encoding_2017, - title = {Encoding and decoding models in cognitive electrophysiology}, - volume = {11}, - issn = {16625137}, - doi = {10.3389/fnsys.2017.00061}, - abstract = {© 2017 Holdgraf, Rieger, Micheli, Martin, Knight and Theunissen. Cognitive neuroscience has seen rapid growth in the size and complexity of data recorded from the human brain as well as in the computational tools available to analyze this data. This data explosion has resulted in an increased use of multivariate, model-based methods for asking neuroscience questions, allowing scientists to investigate multiple hypotheses with a single dataset, to use complex, time-varying stimuli, and to study the human brain under more naturalistic conditions. These tools come in the form of “Encoding” models, in which stimulus features are used to model brain activity, and “Decoding” models, in which neural features are used to generated a stimulus output. Here we review the current state of encoding and decoding models in cognitive electrophysiology and provide a practical guide toward conducting experiments and analyses in this emerging field. Our examples focus on using linear models in the study of human language and audition. We show how to calculate auditory receptive fields from natural sounds as well as how to decode neural recordings to predict speech. The paper aims to be a useful tutorial to these approaches, and a practical introduction to using machine learning and applied statistics to build models of neural activity. The data analytic approaches we discuss may also be applied to other sensory modalities, motor systems, and cognitive systems, and we cover some examples in these areas. In addition, a collection of Jupyter notebooks is publicly available as a complement to the material covered in this paper, providing code examples and tutorials for predictive modeling in python. The aimis to provide a practical understanding of predictivemodeling of human brain data and to propose best-practices in conducting these analyses.}, - journal = {Frontiers in Systems Neuroscience}, - author = {Holdgraf, Christopher Ramsay and Rieger, J.W. and Micheli, C. and Martin, S. and Knight, R.T. and Theunissen, F.E.}, - year = {2017}, - keywords = {Decoding models, Encoding models, Electrocorticography (ECoG), Electrophysiology/evoked potentials, Machine learning applied to neuroscience, Natural stimuli, Predictive modeling, Tutorials} -} - -@book{ruby, - title = {The Ruby Programming Language}, - author = {Flanagan, David and Matsumoto, Yukihiro}, - year = {2008}, - publisher = {O'Reilly Media} -} diff --git a/reqBuilder.sh b/reqBuilder.sh deleted file mode 100755 index 783dab7..0000000 --- a/reqBuilder.sh +++ /dev/null @@ -1,9 +0,0 @@ -#!/bin/bash - -echo Parsing requirements from ./notebooks -find ./notebooks -name "*.ipynb" | xargs -I {} pipreqsnb {} > /dev/null 2>&1 -find ./notebooks -iname "requirements.txt" -print0 | xargs -0 -I {} cat {} >> reqList.txt -cat reqList.txt | cut -f1 -d"=" | sort -u > requirements.txt -ureqs=`cat requirements.txt | wc -l` -rm reqList.txt -echo Done: Requirements file built with $ureqs unique requirements: requirements.txt diff --git a/requirements.txt b/requirements.txt deleted file mode 100644 index f3cb4ae..0000000 --- a/requirements.txt +++ /dev/null @@ -1,4 +0,0 @@ -astropy -astroquery -matplotlib -numpy diff --git a/stsci_logo2.png b/stsci_logo2.png new file mode 100644 index 0000000..eb50faa Binary files /dev/null and b/stsci_logo2.png differ