diff --git a/notebooks/NIRSPEC/MOS/JWPipeNB-NIRSpec-MOS.ipynb b/notebooks/NIRSPEC/MOS/JWPipeNB-NIRSpec-MOS.ipynb
new file mode 100644
index 0000000..fa59920
--- /dev/null
+++ b/notebooks/NIRSPEC/MOS/JWPipeNB-NIRSpec-MOS.ipynb
@@ -0,0 +1,2383 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "1242c2a8-719d-4647-aaeb-ac5f256ad2c4",
+ "metadata": {},
+ "source": [
+ " "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e64444c3-e53f-4489-8762-14250ad9eb67",
+ "metadata": {},
+ "source": [
+ "# NIRSpec MOS Pipeline Notebook"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7ddbe7fe-926b-41bf-a1fa-84802422c604",
+ "metadata": {},
+ "source": [
+ "**Authors**: Dan Coe (dcoe@stsci.edu), NIRSpec branch, with contributions from the NIRSpec team, including Elena Manjavacas, Peter Zeidler, Kayli Glidic, Melanie Clarke, James Muzerolle, Nikolay Nikolov, Chris Hayes, and Alaina Henry, who designed the ERO NIRSpec observations. \n",
+ "**Last Updated**: January 12, 2024 \n",
+ "**Pipeline Version**: 1.17.1 (Build 11.2, Context jwst_1322.pmap)\n",
+ "\n",
+ "**Purpose**: \n",
+ "End-to-end calibration with the James Webb Space Telescope (JWST) pipeline is divided into three main processing stages. This notebook provides a framework for processing generic Near-Infrared Spectrograph (NIRSpec) multi-object spectroscopy (MOS) data through [stages 1-3 of the JWST pipeline](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline/stages-of-jwst-data-processing#gsc.tab=0), including how to use associations for multi-exposure observations and how to interact and work with JWST datamodels. Data is assumed to be organized into two folders: science and associations, as specified in the paths set up below. In most cases, editing cells outside the [Configuration](#1.-Configuration) section is unnecessary unless the standard pipeline processing options or plot parameters need to be modified.\n",
+ "\n",
+ "**[Data](#3.-Demo-Mode-Setup-(ignore-if-not-using-demo-data))**: \n",
+ "This notebook is set up to use observations of galaxy cluster SMACS0723 with the G395M grism obtained by Proposal ID (PID) 2736, Observation 7. The demo data will automatically download unless disabled (i.e., to use local files instead).\n",
+ "\n",
+ "**[JWST pipeline version and CRDS context](#Set-CRDS-Context-and-Server)**: \n",
+ "This notebook was written for the above-specified pipeline version and associated build context for this version of the JWST Calibration Pipeline. Information about this and other contexts can be found in the JWST Calibration Reference Data System (CRDS [server](https://jwst-crds.stsci.edu/)). If you use different pipeline versions, please refer to the table [here](https://jwst-crds.stsci.edu/display_build_contexts/) to determine what context to use. To learn more about the differences for the pipeline, read the relevant [documentation](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline/jwst-operations-pipeline-build-information#references). \n",
+ "\n",
+ "Please note that pipeline software development is a continuous process, so results in some cases may be slightly different if a subsequent version is used. **For optimal results, users are strongly encouraged to reprocess their data using the most recent pipeline version and [associated CRDS context](https://jwst-crds.stsci.edu/display_build_contexts/), taking advantage of bug fixes and algorithm improvements.**\n",
+ "Any [known issues](https://jwst-docs.stsci.edu/known-issues-with-jwst-data/nirspec-known-issues/nirspec-mos-known-issues#NIRSpecFSKnownIssues-Resamplingof2-Dspectra&gsc.tab=0:~:text=MOS%20Known%20Issues-,NIRSpec%20MOS%20Known%20Issues,-Known%20issues%20specific) for this build are noted in the notebook. \n",
+ "\n",
+ "**Updates**: \n",
+ "This notebook is regularly updated to incorporate the latest pipeline improvements. Find the most up-to-date version of this notebook [here](https://github.com/spacetelescope/jwst-pipeline-notebooks/).\n",
+ "\n",
+ "**Recent Changes**:\n",
+ "* October 15, 2024: Converted notebook to follow standard template ([original](https://github.com/dancoe/NIRSpec/blob/main/NIRSpec%20MOS%20pipeline.ipynb)). \n",
+ "* November 4, 2024: Notebook updated to JWST pipeline version 1.16.0 (Build 11.1).\n",
+ "* January 12, 2024: Added an example for association file creation.\n",
+ "* February 19, 2025: Updated JWST pipeline version 1.17.1 (Build 11.2). Changed code to create all associations instead of using the MAST downloads.\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e6483525-d20d-491c-b310-0a3688467922",
+ "metadata": {},
+ "source": [
+ "## Table of Contents\n",
+ "\n",
+ "* [1. Configuration](#1.-Configuration)\n",
+ "* [2. Package Imports](#2.-Package-Imports)\n",
+ "* [3. Demo Mode Setup](#3.-Demo-Mode-Setup-(ignore-if-not-using-demo-data))\n",
+ "* [4. Directory Setup](#4.-Directory-Setup)\n",
+ "* [5. Stage 1: `Detector1Pipeline` (`calwebb_detector1`)](#5.-Stage-1:-Detector1Pipeline-(calwebb_detector1))\n",
+ " * [5.1 Configure `Detector1Pipeline`](#5.1-Configure-Detector1Pipeline)\n",
+ " * [5.2 Run `Detector1Pipeline`](#5.2-Run-Detector1Pipeline)\n",
+ "* [6. Stage 2: `Spec2Pipeline` (`calwebb_spec2`)](#5.-Stage-2:-Spec2Pipeline-(calwebb_spec2))\n",
+ " * [6.1 Configure `Spec2Pipeline`](#6.1-Configure-Spec2Pipeline)\n",
+ " * [6.2 Create `Spec2Pipeline` Association Files](#6.2-Create-Spec2Pipeline-Association-Files)\n",
+ " * [6.3 MSA Metadata File](#6.3-MSA-Metadata-File)\n",
+ " * [6.4 Run `Spec2Pipeline`](#6.4-Run-Spec2Pipeline)\n",
+ "* [7. Stage 3: `Spec3Pipeline` (`calwebb_spec3`)](#5.-Stage-3:-Spec3Pipeline-(calwebb_spec3))\n",
+ " * [7.1 Configure `Spec3Pipeline`](#7.1-Configure-Spec3Pipeline)\n",
+ " * [7.2 Create `Spec3Pipeline` Association Files](#7.2-Create-Spec3Pipeline-Association-Files)\n",
+ " * [7.3 Run `Spec3Pipeline`](#7.3-Run-Spec3Pipeline)\n",
+ "* [8. Visualize the Data](#8.-Visualize-the-Data)\n",
+ " * [8.1 Display `Detector1Pipeline` Products](#8.1-Display-Detector1Pipeline-Products)\n",
+ " * [8.2 Display `Spec2Pipeline` Products](#8.2-Display-Spec2Pipeline-Products)\n",
+ " * [8.3 Display `Spec3Pipeline` Products](#8.3-Display-Spec3Pipeline-Products)\n",
+ "* [9. Modifying the EXTRACT1D Reference File (as needed)](#9.-Modifying-the-EXTRACT1D-Reference-File-(as-needed))\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d3d67fa9-8623-471b-a897-9b60b1048970",
+ "metadata": {},
+ "source": [
+ "## 1. Configuration\n",
+ "\n",
+ "#### Install dependencies and parameters\n",
+ "To make sure that the pipeline version is compatible with the steps discussed below and that the required dependencies and packages get installed, you can create a fresh conda environment and install the provided requirements.txt file before starting this notebook:\n",
+ "\n",
+ " conda create -n nirspec_mos_pipeline python=3.12\n",
+ " conda activate nirspec_mos_pipeline\n",
+ " pip install -r requirements.txt\n",
+ "\n",
+ "#### Set the basic parameters to configure the notebook.\n",
+ "\n",
+ "These parameters determine what data gets used, where data is located (if already on disk), and the type of background subtraction (if any). The list of parameters includes:\n",
+ "* `demo_mode`:\n",
+ " * `True`: Downloads example data from the [Barbara A. Mikulski Archive for Space Telescopes (MAST)](https://archive.stsci.edu/) and processes it through the pipeline. All processing will occur in a local directory unless modified in [Section 3](#3.-Demo-Mode-Setup-(ignore-if-not-using-demo-data)) below.\n",
+ " * `False`: Process your own downloaded data; provide its location.
\n",
+ "* **Directories with data**:\n",
+ " * `sci_dir`: Directory where science observation data is stored.\n",
+ " * REMOVE `asn_dir`: Directory where Stage 2/3 associations are stored.
\n",
+ "* **[Backgroud subtraction methods](https://jwst-docs.stsci.edu/jwst-near-infrared-spectrograph/nirspec-observing-strategies/nirspec-background-recommended-strategies#gsc.tab=0)** (`True` = run, `False` = skip):\n",
+ " * `master_bg`: Apply master-background subtraction in `calwebb_spec2`? Uses \"blank sky\" shutters defined in the observation.\n",
+ " * `pixel_bg` : Apply pixel-to-pixel background subtraction in `calwebb_spec2`. This is the default pipeline setting. Uses nodded observations.
\n",
+ "\n",
+ "Note that `demo_mode` must be set appropriately below.\n",
+ "\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ab1960b2-75f8-420d-8673-250e370b2400",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set parameters for demo_mode, data mode directories, and processing steps.\n",
+ "\n",
+ "# -------------------------------DEMO MODE-----------------------------------\n",
+ "demo_mode = True\n",
+ "\n",
+ "if demo_mode:\n",
+ " print('Running in demonstration mode using online example data!')\n",
+ "\n",
+ "# ----------------------------User Mode Directories--------------------------\n",
+ "else: # If demo_mode = False, look for user data in these paths.\n",
+ "\n",
+ " # Set directory paths for processing specific data; adjust to your local\n",
+ " # directory setup (examples provided below).\n",
+ " basedir = os.path.abspath(os.path.join(os.getcwd(), ''))\n",
+ "\n",
+ " # Directory to science observation data; expects uncalibrated data in\n",
+ " # sci_dir/uncal/ and results in stage1, stage2, and stage3 directories.\n",
+ " sci_dir = os.path.join(basedir, 'mos_data_02736/Obs007', '')\n",
+ "\n",
+ "# ---------------------------Set Processing Steps----------------------------\n",
+ "# Individual pipeline stages can be turned on/off here. Note that a later\n",
+ "# stage won't be able to run unless data products have already been\n",
+ "# produced from the prior stage.\n",
+ "\n",
+ "# Science processing.\n",
+ "dodet1 = True # calwebb_detector1\n",
+ "dospec2 = True # calwebb_spec2\n",
+ "dospec3 = True # calwebb_spec3\n",
+ "doviz = True # Visualize calwebb outputs\n",
+ "\n",
+ "# ---------------------------Set Processing Steps----------------------------\n",
+ "# How should background subtraction be done?\n",
+ "# If none are selected, data will not be background subtracted.\n",
+ "# pixel_bg - True for nodded observations\n",
+ "# master_bg - True if using \"blank sky\" sutters in the science observation.\n",
+ "master_bg = False # Master-background subtraction in spec3.\n",
+ "pixel_bg = True # Pixel-based background subtraction in spec2."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3ee72de7-b396-4e6e-9033-cafb4977eb8f",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### Set CRDS Context and Server\n",
+ "Before importing `CRDS` and `JWST` modules, we need to configure our environment. This includes defining a CRDS cache directory in which to keep the reference files that will be used by the calibration pipeline. If the local CRDS cache directory has not been set, it will automatically be created in the home directory.
\n",
+ "[Build Context Table](https://jwst-crds.stsci.edu/display_build_contexts/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3ad8ff04-797d-496b-86f4-89eed9e75492",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# ------------------------Set CRDS context and paths------------------------\n",
+ "\n",
+ "# Each version of the calibration pipeline is associated with a specific CRDS\n",
+ "# context file. The pipeline will select the appropriate context file behind\n",
+ "# the scenes while running. However, if you wish to override the default context\n",
+ "# file and run the pipeline with a different context, you can set that using\n",
+ "# the CRDS_CONTEXT environment variable. Here we show how this is done,\n",
+ "# although we leave the line commented out in order to use the default context.\n",
+ "# If you wish to specify a different context, uncomment the line below.\n",
+ "#os.environ['CRDS_CONTEXT'] = 'jwst_1298.pmap' # CRDS context for 1.16.0\n",
+ "\n",
+ "# Set CRDS cache directory to user home if not already set.\n",
+ "if os.getenv('CRDS_PATH') is None:\n",
+ " os.environ['CRDS_PATH'] = os.path.join(os.path.expanduser('~'), 'crds_cache')\n",
+ "\n",
+ "# Check whether the CRDS server URL has been set. If not, set it.\n",
+ "if os.getenv('CRDS_SERVER_URL') is None:\n",
+ " os.environ['CRDS_SERVER_URL'] = 'https://jwst-crds.stsci.edu'\n",
+ "\n",
+ "# Output the current CRDS path and server URL in use.\n",
+ "print('CRDS local filepath:', os.environ['CRDS_PATH'])\n",
+ "print('CRDS file server:', os.environ['CRDS_SERVER_URL'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2bd611cc-a558-4424-b302-7236ea0aaefe",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "## 2. Package Imports"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "16ee3875-27e6-4b86-b70d-1c255136b9c6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Use the entire available screen width for this notebook.\n",
+ "from IPython.display import display, HTML, JSON\n",
+ "display(HTML(\"\"))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2bfec1a8-67ee-4e7c-8ae3-69d12a12f1cb",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# ----------------------General Imports----------------------\n",
+ "import time\n",
+ "import glob\n",
+ "import json\n",
+ "import itertools\n",
+ "import numpy as np\n",
+ "from itertools import combinations\n",
+ "\n",
+ "# ----------------------Astropy Imports----------------------\n",
+ "# Astropy utilities for opening FITS files, downloading demo files, etc.\n",
+ "from astropy.io import fits\n",
+ "from astropy.stats import sigma_clip\n",
+ "from astropy.visualization import ImageNormalize, ManualInterval, LogStretch\n",
+ "from astropy.visualization import LinearStretch, AsinhStretch, simple_norm\n",
+ "\n",
+ "# -------------------- Astroquery Imports ----------------------\n",
+ "from astroquery.mast import Observations\n",
+ "\n",
+ "# ----------------------Plotting Imports---------------------\n",
+ "import matplotlib.pyplot as plt\n",
+ "from matplotlib.patches import Rectangle\n",
+ "from matplotlib.collections import PatchCollection"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "784d79d4-6b6e-4472-848e-023deecbcfaa",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ "Installation instructions for the JWST pipeline found here: [JDox](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview) •\n",
+ "[ReadtheDocs](https://jwst-pipeline.readthedocs.io) •\n",
+ "[Github](https://github.com/spacetelescope/jwst)\n",
+ "\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ce773fac-c76a-49c8-8ca2-3c9169fdc17a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# ----------------------JWST Calibration Pipeline Imports----------------------\n",
+ "import jwst # Import the base JWST and CRDS packages.\n",
+ "import crds\n",
+ "from crds.client import api\n",
+ "from stpipe import crds_client\n",
+ "\n",
+ "# JWST pipelines (each encompassing many steps).\n",
+ "from jwst.pipeline import Detector1Pipeline # calwebb_detector1\n",
+ "from jwst.pipeline import Spec2Pipeline # calwebb_spec2\n",
+ "from jwst.pipeline import Spec3Pipeline # calwebb_spec3\n",
+ "from jwst.extract_1d import Extract1dStep # Extract1D Step\n",
+ "\n",
+ "# JWST pipeline utilities\n",
+ "from jwst import datamodels # JWST pipeline utilities: datamodels.\n",
+ "from jwst.associations import asn_from_list # Tools for creating association files.\n",
+ "from jwst.associations.lib.rules_level3_base import DMS_Level3_Base # Define Lvl3 ASN.\n",
+ "\n",
+ "default_context = crds.get_default_context('jwst', state='build')\n",
+ "print(\"JWST Calibration Pipeline Version = {}\".format(jwst.__version__))\n",
+ "print(f\"Default CRDS Context for JWST Version {jwst.__version__}: {default_context}\")\n",
+ "print(f\"Using CRDS Context: {os.environ.get('CRDS_CONTEXT', default_context)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fc1663f0-51d8-4ac9-824e-53d7691785aa",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### Define Convience Functions\n",
+ "\n",
+ "Function that identifies unique MSA files."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7b3c91d0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Find unique files to download.\n",
+ "def unique_files_to_array(files):\n",
+ " \"\"\"\n",
+ " Checks for unique files and adds them to an array.\n",
+ "\n",
+ " Parameters\n",
+ " ----------\n",
+ " files : list of lists\n",
+ " The input files.\n",
+ "\n",
+ " Returns\n",
+ " -------\n",
+ " list: An array of unique files.\n",
+ " \"\"\"\n",
+ "\n",
+ " unique_files = set()\n",
+ " for row in files:\n",
+ " for element in row:\n",
+ " if isinstance(element, str):\n",
+ " unique_files.add(element)\n",
+ "\n",
+ " return list(unique_files)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b936a764",
+ "metadata": {},
+ "source": [
+ "Define a function to combine the nodded exposures to use as background exposures in for the spec2 associations.\n",
+ "This funtions identify the `rate.fits` observations from the same detector and visit and create all the possible conbinations. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a13f6e24",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Identify and combine the nodded exposures\n",
+ "def group_spec2_asn_bkg_nods(files_l2, nods):\n",
+ "\n",
+ " # Function to split a string into words using underscores\n",
+ " def split_name(s):\n",
+ " return s.split('_')\n",
+ "\n",
+ " # Dictionary to store the strings that match the criteria for grouping\n",
+ " group_obs = {}\n",
+ "\n",
+ " # Process each file to group 'rate.fits' files\n",
+ " for original_name in files_l2:\n",
+ " original_dir = os.path.dirname(original_name)\n",
+ " original_name = os.path.basename(original_name)\n",
+ "\n",
+ " # Split the filename into components\n",
+ " obs_vals = split_name(original_name)\n",
+ "\n",
+ " # Check if the filename has at least 4 components for grouping\n",
+ " if len(obs_vals) >= 4 and obs_vals[-1] == 'rate.fits':\n",
+ " key = (obs_vals[0], obs_vals[1], obs_vals[3], obs_vals[-1])\n",
+ "\n",
+ " # Group files by the key derived from filename components\n",
+ " if key in group_obs:\n",
+ " group_obs[key].append(os.path.join(original_dir, original_name))\n",
+ " else:\n",
+ " group_obs[key] = [os.path.join(original_dir, original_name)]\n",
+ "\n",
+ " # Convert grouped files into a list of lists\n",
+ " matrix = list(group_obs.values())\n",
+ "\n",
+ " # Now, process each group to generate associations with background nods\n",
+ " result = []\n",
+ " for group in matrix:\n",
+ " # Extract names from file paths for node association\n",
+ " names = [file for file in group]\n",
+ "\n",
+ " # Generate associations\n",
+ " for fixed in names:\n",
+ " remaining = [os.path.join(original_dir, name) for name in names if name != fixed]\n",
+ " for pair in combinations(remaining, nods - 1):\n",
+ " combination = [os.path.join(original_dir, fixed)] + list(pair)\n",
+ " result.append(combination)\n",
+ "\n",
+ " return result"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "8379329c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Start a timer to keep track of runtime.\n",
+ "time0 = time.perf_counter()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b698c1f3-0f7f-49f9-a449-9b8ea17c5a2d",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "## 3. Demo Mode Setup (ignore if not using demo data)\n",
+ "
\n",
+ "\n",
+ "The data in this notebook is public and does not require a token. For other data sets, you may need to provide a token. For more infomation visit the\n",
+ "[astroquery](https://astroquery.readthedocs.io/en/latest/index.html) documentation.\n",
+ "\n",
+ "
\n",
+ "\n",
+ "If running in demonstration mode, set up the program information to retrieve the uncalibrated data (`_uncal.fits`) automatically from MAST using `astroquery`. MAST provides flexibility by allowing searches based on proposal ID and observation ID, rather than relying solely on filenames. More information about the JWST file naming conventions can be found [here](https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/file_naming.html).\n",
+ "\n",
+ "The MOS demo data in this notebook is from [\"JWST's First Deep Field\"](https://webbtelescope.org/contents/news-releases/2022/news-2022-035), [JWST Early Release Observation (ERO) program 2736](https://www.stsci.edu/jwst/science-execution/program-information?id=2736) and features observations of galaxy cluster SMACS0723 with NIRCam, NIRSpec, and MIRI. This program includes two identical MOS observations (7 and 8) with confirmation images taken after target acquisition (20 groups | NRSIRS2RAPID | 306 [s]). More of the program setup is briefly summarized in the table below.\n",
+ "\n",
+ "| Demo Target: SMACS0723 | | |\n",
+ "|:-----------:|:-------:|:---:|\n",
+ "| PROGRAM | 02736 | Program number |\n",
+ "| OBSERVTN | 007 | Observation number |\n",
+ "| [GRATING/FILTER](https://jwst-docs.stsci.edu/jwst-near-infrared-spectrograph/nirspec-observing-modes/nirspec-multi-object-spectroscopy#gsc.tab=0:~:text=Table%C2%A01.%20Spectral%20configurations%20available%20in%20NIRSpec%20MOS%20mode) | G395M/F290LP | λ: 2.87–5.10 μm (a medium resolution, R ~ 1000) |\n",
+ "||G235M/F170LP | λ: 1.66–3.07 μm (a medium resolution, R ~ 1000) |\n",
+ "| SUBARRAY | FULL | Subarray used (2048 x 2048) |\n",
+ "| NINTS | 2 | Number of integrations in exposure |\n",
+ "| NGROUPS | 20 | Number of groups in integration |\n",
+ "| READPATT | NRSIRS2 | Readout pattern |\n",
+ "| NOD_TYPE | 3-SHUTTER-SLITLET | Nod pattern type |\n",
+ "| NUMDTHPT | 3 | Total number of points in pattern | \n",
+ "| SRCTYAPT | UNKNOWN | Source type selected in APT |\n",
+ "| TOTAL DURATION | NOD_TYPE x NINTS x NGROUPS = 8841 [s] (~2.5 hrs) | Total duration (per grating and observation)|\n",
+ "\n",
+ "> **Note:** The presence of a physical gap between detectors affects all MOS observations (any resolution) because the spectra can be long enough to span both NIRSpec detectors. [More Info ...](https://jwst-docs.stsci.edu/jwst-near-infrared-spectrograph/nirspec-operations/nirspec-mos-operations/nirspec-mos-wavelength-ranges-and-gaps#gsc.tab=0)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e38dc1ac-7329-4369-9386-617baedf9041",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set up the program information and directories to collect\n",
+ "# the data in demo_mode.\n",
+ "\n",
+ "if demo_mode:\n",
+ "\n",
+ " print('Running in demonstration mode. '\n",
+ " 'Example data will be downloaded from MAST!')\n",
+ "\n",
+ " # NOTE:\n",
+ " # For non public data sets, you may need to provide a token.\n",
+ " # However, for security it is not recommended to enter tokens into\n",
+ " # a terminal or Jupyter notebook.\n",
+ " #Observations.login(token=\"your-token\")\n",
+ "\n",
+ " # --------------Program and observation information--------------\n",
+ " program = \"02736\"\n",
+ " sci_observtn = \"007\"\n",
+ " filters = [\"F290LP;G395M\"]\n",
+ " number_nods = 3 # Number of nodded positions\n",
+ "\n",
+ " # ----------Define the base and observation directories----------\n",
+ " basedir = os.path.join(os.getcwd(), f'mos_data_{program}')\n",
+ " sci_dir = os.path.join(basedir, f'Obs{sci_observtn}')\n",
+ " asn_dir = os.path.join(sci_dir, 'asn/') # Keep MSA files in the association directory\n",
+ " uncal_dir = os.path.join(sci_dir, 'uncal/')\n",
+ "\n",
+ " # ----Ensure directories for downloading MAST data exists--------\n",
+ " os.makedirs(uncal_dir, exist_ok=True)\n",
+ " os.makedirs(asn_dir, exist_ok=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f2643d7a-44bc-407f-81eb-0f628169258c",
+ "metadata": {},
+ "source": [
+ " Click on the following links to learn more about querying and downloading data: \n",
+ "• [Downloading data](https://astroquery.readthedocs.io/en/latest/mast/mast_obsquery.html#downloading-data) \n",
+ "• [Observations Class](https://astroquery.readthedocs.io/en/latest/api/astroquery.mast.ObservationsClass.html) \n",
+ "• [Products Field Descriptions](https://mast.stsci.edu/api/v0/_productsfields.html)
\n",
+ "\n",
+ "Compile a table of files from MAST associated with the science (SCI) observation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "51e2235c-5b36-453e-9bd6-baf9e9c0e70e",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Obtain a list of observation IDs for the specified demo program.\n",
+ "\n",
+ "if demo_mode:\n",
+ " # --------------------SCIENCE Observation--------------------\n",
+ " sci_obs_id_table = Observations.query_criteria(instrument_name=['NIRSPEC/MSA'],\n",
+ " provenance_name=[\"CALJWST\"],\n",
+ " obs_id=[f'*{program}*{sci_observtn}*'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "584d0945-2e18-41fb-81d4-222332823981",
+ "metadata": {},
+ "source": [
+ "Filter these tables to identify uncalibrated data, metadata, and association files to download from MAST.\n",
+ "\n",
+ "The demo dataset consists of six `_uncal.fits` files, each approximately 500 MB in size."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "87cbc15f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Convert visits into a list of uncalibrated data and ASN files.\n",
+ "\n",
+ "if demo_mode:\n",
+ "\n",
+ " file_criteria = {'filters': filters, 'calib_level': [1],\n",
+ " 'productSubGroupDescription': 'UNCAL'}\n",
+ "\n",
+ " # Initialize lists for science, background, and ASN files.\n",
+ " sci_downloads, msa_downloads = [], []\n",
+ "\n",
+ " pfilter = Observations.filter_products # Alias for filter_products method.\n",
+ "\n",
+ " # ----Identify uncalibrated SCIENCE files associated with each visit-----\n",
+ " for exposure in sci_obs_id_table:\n",
+ " sci_products = Observations.get_product_list(exposure)\n",
+ "\n",
+ " # Filter for full-size science files (exclude smaller confirmation images).\n",
+ " avg_sci_size = np.nanmean(sci_products['size'])\n",
+ " sci_products_avg = sci_products[sci_products['size'] > avg_sci_size]\n",
+ " sci_downloads.extend(pfilter(sci_products_avg, **file_criteria)['dataURI'])\n",
+ "\n",
+ " # Identifies association metadata files.\n",
+ " msa_files = {\n",
+ " f['productFilename'] for f in sci_products\n",
+ " if 'AUXILIARY' in f['productType'] and 'metadata' in f['description']}\n",
+ " msa_downloads.append(msa_files)\n",
+ "\n",
+ " # Filter out other observations and remove duplicates.\n",
+ " msa_downloads = unique_files_to_array(msa_downloads)\n",
+ " sci_downloads = {f for f in sci_downloads if f\"jw{program}{sci_observtn}\" in f}\n",
+ "\n",
+ " msa_mast_downloads = [\n",
+ " f\"mast:JWST/product/{f}\" for f in msa_downloads\n",
+ " if f\"jw{program}{sci_observtn}\" in f\n",
+ " ]\n",
+ "\n",
+ " print(f\"Science files selected for downloading: {len(sci_downloads)}\")\n",
+ " print(f\"MSA files selected for downloading: {len(msa_mast_downloads)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3517dfb8",
+ "metadata": {},
+ "source": [
+ "Download the data.\n",
+ "\n",
+ "
\n",
+ "Warning: If this notebook is halted during this step, the downloaded file may be incomplete, and cause crashes later on!\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f570975a-cb41-41d6-9738-d0598544a081",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Download data and place them into the appropriate directories.\n",
+ "\n",
+ "if demo_mode:\n",
+ " for file in sci_downloads:\n",
+ " sci_manifest = Observations.download_file(file, local_path=uncal_dir)\n",
+ " for file in msa_mast_downloads:\n",
+ " msa_manifest = Observations.download_file(file, local_path=asn_dir)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9c91bec9-f4a7-451e-a439-3761ee0431ad",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### Galaxies of Interest\n",
+ "There are several galaxies of interest in the demo data. Here we will look at some of them (source_ids provided below).\n",
+ "\n",
+ "* **6355**: z = 7.665 (13.0 Gyr ago)\n",
+ " * We'll look at this galaxy in this notebook.\n",
+ " * Excellent spectrum with bright lines.\n",
+ " * 5-shutter slitlet (rather than standard 3-shutter slitlet), so 1D extraction can be improved significantly.\n",
+ " * Not a multiple image of 10612 (described below), which has a similar redshift.\n",
+ "\n",
+ "* 5144: z = 6.383 (12.8 Gyr ago)\n",
+ "\n",
+ "The following galaxies have been featured in the [press release](https://webbtelescope.org/contents/news-releases/2022/news-2022-035):\n",
+ "* 4590: z = 8.498 (13.1 Gyr ago)\n",
+ "* 10612: z = 7.663 (13.0 Gyr ago)\n",
+ "* 8140: z = 5.275 (12.6 Gyr ago)\n",
+ "* 9922: z = 2.743 (11.3 Gyr ago)\n",
+ "\n",
+ "and presented and studied in papers including\n",
+ "[Katz et al. 2023](https://ui.adsabs.harvard.edu/abs/2023MNRAS.518..592K),\n",
+ "[Curti et al. 2023](https://ui.adsabs.harvard.edu/abs/2023MNRAS.518..425C),\n",
+ "[Carnall et al. 2023](https://ui.adsabs.harvard.edu/abs/2023MNRAS.518L..45C)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "09d5b068-fc29-4413-9940-95bf0e5d0388",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "## 4. Directory Setup\n",
+ "Set up detailed paths to input/output stages here."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4bf6dd2c-9501-4bb5-bddf-f1c305edd9b1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Define/create output subdirectories to keep data products organized.\n",
+ "\n",
+ "# -----------------------------Science Directories------------------------------\n",
+ "uncal_dir = os.path.join(sci_dir, 'uncal/') # Uncalibrated pipeline inputs.\n",
+ "det1_dir = os.path.join(sci_dir, 'stage1/') # calwebb_detector1 pipeline outputs.\n",
+ "spec2_dir = os.path.join(sci_dir, 'stage2/') # calwebb_spec2 pipeline outputs.\n",
+ "spec3_dir = os.path.join(sci_dir, 'stage3/') # calwebb_spec3 pipeline outputs.\n",
+ "asn_dir = os.path.join(sci_dir, 'asn/') # Associations directory\n",
+ "\n",
+ "# Creates the directories if target directory does not exist\n",
+ "os.makedirs(det1_dir, exist_ok=True)\n",
+ "os.makedirs(spec2_dir, exist_ok=True)\n",
+ "os.makedirs(spec3_dir, exist_ok=True)\n",
+ "os.makedirs(asn_dir, exist_ok=True)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ddd113f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Print out the time benchmark.\n",
+ "time1 = time.perf_counter()\n",
+ "print(f\"Runtime so far: {round((time1 - time0) / 60.0, 1):0.4f} min\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5cf9f2f5-e3bd-4348-b150-7e607aec6a15",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "## 5. Stage 1: `Detector1Pipeline` (`calwebb_detector1`)\n",
+ "In this section, we process the data through the `calwebb_detector1` pipeline to create Stage 1 [data products](https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/science_products.html).\n",
+ "\n",
+ "* **Input**: Raw exposure (`_uncal.fits`) containing original data from all detector readouts (ncols x nrows x ngroups x nintegrations).\n",
+ "* **Output**: Uncalibrated countrate (slope) image in units of DN/s:\n",
+ " * `_rate.fits`: A single countrate image averaged over multiple integrations (if available).\n",
+ " * `_rateints.fits`: Countrate images for each integration, saved in multiple extensions.\n",
+ "\n",
+ "The `Detector1Pipeline` applies basic detector-level corrections on a group-by-group basis, followed by ramp fitting for all exposure types, commonly referred to as \"ramps-to-slopes\" processing.\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d295a974-db84-4410-8737-a8553975713b",
+ "metadata": {},
+ "source": [
+ "### 5.1 Configure `Detector1Pipeline`\n",
+ "\n",
+ "The `Detector1Pipeline` has the following steps available for NIRSpec MOS:\n",
+ "\n",
+ "* `group_scale` : Rescales pixel values to correct for improper onboard frame averaging.\n",
+ "* `dq_init` : Initializes the data quality (DQ) flags for the input data.\n",
+ "* `saturation` : Flags pixels at or below the A/D floor or above the saturation threshold.\n",
+ "* `superbias` : Subtracts the superbias reference file from the input data.\n",
+ "* `refpix` : Use reference pixels to correct bias drifts.\n",
+ "* `linearity` : Applies a correction for non-linear detector response.\n",
+ "* `dark_current` : Subtracts the dark current reference file from the input data.\n",
+ "* `jump` : Performs CR/jump detection on each ramp integration within an exposure.\n",
+ "* `clean_flicker_noise`: Removes flicker (1/f) noise from calibrated ramp images (similar to `nsclean` in spec2).\n",
+ "* `ramp_fit` : Determines the mean count rate (counts per second) for each pixel by performing a linear fit to the input data.\n",
+ "* `gain_scale` : Corrects pixel values for non-standard gain settings, primarily in NIRSpec subarray data.\n",
+ "\n",
+ "For more information about each step and a full list of step arguments, please refer to the official documentation: [JDox](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview/stages-of-jwst-data-processing/calwebb_detector1) and\n",
+ "[ReadtheDocs](https://jwst-pipeline.readthedocs.io/en/stable/jwst/pipeline/calwebb_detector1.html)\n",
+ "\n",
+ "Below, we set up a dictionary that defines how the `Detector1Pipeline` should be configured for MOS data. We follow the [CEERS NIRSpec reduction parameters](https://web.corral.tacc.utexas.edu/ceersdata/DR07/NIRSpec/README_NIRSpec_DR0.7.txt) to improve the rejection of cosmic rays and snowballs during the `jump` step."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "174679be-3230-4da6-899a-975f06e84e60",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "To override specific steps and reference files, use the examples below.\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6f0fb783-d564-4d99-8465-b3dffab91ab5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set up a dictionary to define how the Detector1 pipeline should be configured.\n",
+ "\n",
+ "# -------------------------Boilerplate dictionary setup-------------------------\n",
+ "det1dict = {}\n",
+ "det1dict['group_scale'], det1dict['dq_init'], det1dict['saturation'] = {}, {}, {}\n",
+ "det1dict['superbias'], det1dict['refpix'] = {}, {}\n",
+ "det1dict['linearity'], det1dict['dark_current'], det1dict['jump'] = {}, {}, {}\n",
+ "det1dict['clean_flicker_noise'], det1dict['ramp_fit'] = {}, {}\n",
+ "det1dict['gain_scale'] = {}\n",
+ "\n",
+ "# ---------------------------Override reference files---------------------------\n",
+ "\n",
+ "# Overrides for various reference files (example).\n",
+ "# Files should be in the base local directory or provide full path.\n",
+ "#det1dict['dq_init']['override_mask'] = 'myfile.fits' # Bad pixel mask\n",
+ "#det1dict['superbias']['override_superbias'] = 'myfile.fits' # Bias subtraction\n",
+ "#det1dict['dark_current']['override_dark'] = 'myfile.fits' # Dark current subtraction\n",
+ "\n",
+ "# -----------------------------Set step parameters------------------------------\n",
+ "\n",
+ "# Overrides for whether or not certain steps should be skipped (example).\n",
+ "det1dict['linearity']['skip'] = False # This is the default.\n",
+ "\n",
+ "# Turn on multi-core processing (off by default).\n",
+ "# Choose what fraction of cores to use (quarter, half, or all).\n",
+ "det1dict['jump']['maximum_cores'] = 'half'\n",
+ "#det1dict['ramp_fit']['maximum_cores'] = 'half'"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3df7c594",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Turn on detection of cosmic ray snowballs (on by default).\n",
+ "# and change some parameters\n",
+ "det1dict['jump']['expand_large_events'] = True\n",
+ "det1dict['jump']['expand_factor'] = 3 # (default 2)\n",
+ "det1dict['jump']['min_sat_area'] = 15 # (default 1).\n",
+ "det1dict['jump']['min_jump_area'] = 15 # (default 5)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "90649607-669c-4083-a67a-796017f5d3a8",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ "Many exposures are affected by artifacts known as [snowballs](https://jwst-docs.stsci.edu/known-issues-with-jwst-data/shower-and-snowball-artifacts#gsc.tab=0) caused by large cosmic ray events. These artifacts are particularly significant in deep exposures with long integration times, with an estimated rate of one snowball per detector (FULL FRAME) per 20 seconds. To expand the number of pixels flagged as jumps around large cosmic ray events, set `expand_large_events` to True. An `expand_factor` of 3 works well for NIRSpec observations to cover most snowballs.\n",
+ "\n",
+ "
\n",
+ "\n",
+ "JWST detector readout electronics (a.k.a. SIDECAR ASICs) generate significant 1/f noise during detector operations and signal digitization. This noise manifests as faint banding along the detector's slow axis and varies from column to column. For NIRSpec data, the primary pipeline algorithm to address 1/f noise is `nsclean` in the `Spec2Pipeline` (Rauscher 2023) but is off by default.\n",
+ "\n",
+ "An additional 1/f noise-cleaning algorithm, `clean_flicker_noise`, has been implemented at the group stage in the `Detector1Pipeline`. This step is also off by default.\n",
+ "\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1cefe74b-6e76-433f-badd-b1c02aeba44f",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### 5.2 Run `Detector1Pipeline`\n",
+ "Run the science files through the `calwebb_detector1` pipeline using the `.call()` method.\n",
+ "\n",
+ "We use `.call()` instead of `.run()` to ensure that the latest default parameters set via reference files in CRDS are applied ([ReadtheDocs](https://jwst-pipeline.readthedocs.io/en/latest/jwst/stpipe/call_via_run.html)).\n",
+ "\n",
+ "This stage takes approximately 25 minutes to process six `_uncal.fits` files (~4 minutes per file) and generate `_rate.fits` files."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d10e8475-88f3-4951-be41-56d8bdbcf7cd",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Final list of UNCAL files ready for Stage 1 processing.\n",
+ "uncal_sci = sorted(glob.glob(uncal_dir + '*uncal.fits'))\n",
+ "print(f\"Science UNCAL Files:\\n{'-' * 20}\\n\" + \"\\n\".join(uncal_sci))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4bb44c6a-1b9f-4b50-b949-47fa66605fc5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "time_det1 = time.perf_counter() # Tracks runtime for Stage 1."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "675d2709-2bc6-4adb-b094-afa33b5991a6",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "# Run Stage 1 pipeline using the custom det1dict dictionary.\n",
+ "\n",
+ "if dodet1:\n",
+ " # --------------------------Science UNCAL files--------------------------\n",
+ " for uncal_file in uncal_sci:\n",
+ " print(f\"Applying Stage 1 Corrections & Calibrations to: \"\n",
+ " f\"{os.path.basename(uncal_file)}\")\n",
+ "\n",
+ " det1_result = Detector1Pipeline.call(uncal_file,\n",
+ " save_results=True,\n",
+ " steps=det1dict,\n",
+ " output_dir=det1_dir)\n",
+ " print(\"Stage 1 has been completed! \\n\")\n",
+ "else:\n",
+ " print('Skipping Detector1 processing for SCI data.')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d7f05296-faeb-4dec-a6ef-454bc2cec4a3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Print output result details:\n",
+ "#det1_result.__dict__ # View entire contents.\n",
+ "#det1_result.meta.filename\n",
+ "#det1_result.data.shape"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "469a887d-ece7-4904-9684-02fca7102dfc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Print out the time benchmark.\n",
+ "time2 = time.perf_counter()\n",
+ "print(f\"Runtime so far: {round((time2 - time0) / 60.0, 1):0.4f} min\")\n",
+ "print(f\"Runtime for Stage 1: {round((time2 - time_det1) / 60.0, 1):0.4f} min\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c55218d9-ece7-45aa-b226-f34f587d93c3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Final list of RATE[INTS] files ready for Stage 2 processing.\n",
+ "rate_sci = sorted(glob.glob(det1_dir + '*_rate*.fits'))\n",
+ "print(f\"SCIENCE | RATE[INTS] Files:\\n{'-' * 20}\\n\" + \"\\n\".join(rate_sci))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b8f0186f-a4e9-4384-b591-54bf2438c198",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "## 6. Stage 2: `Spec2Pipeline` (`calwebb_spec2`)\n",
+ "In this section, we process our countrate (slope) image products from Stage 1 (`calwebb_detector1`) through the Spec2 (`calwebb_spec2`) pipeline to create Stage 2 [data products](https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/science_products.html).\n",
+ "\n",
+ "* **Input**: A single countrate (slope) image (`_rate[ints].fits`) or an association file listing multiple inputs.\n",
+ "* **Output**: Calibrated products (rectified and unrectified) and 1D spectra.\n",
+ ">* `_cal[ints].fits`: Calibrated 2D (unrectified) spectra (ncols x nrows).\n",
+ ">* `_s2d.fits`: Resampled (rectified) 2D spectra (ncols x nrows).\n",
+ ">* `_x1d[ints].fits`: Extracted 1D spectroscopic data (wavelength vs. flux).\n",
+ "\n",
+ "In Stage 2, each exposure (or association) and detector produces a single file, with multiple extensions corresponding to each source.\n",
+ "\n",
+ "The `Spec2Pipeline` applies additional instrumental corrections and calibrations (e.g., slit loss, path loss, etc.,) to countrate products that result in a fully calibrated individual exposure (per nod/dither position). The `Spec2Pipeline` also converts countrate products from units of DN/s to flux (Jy) for point sources and surface brightness (MJy/sr) for extended sources.\n",
+ "\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6123e939-b3be-4dfe-b444-d927c2baa65e",
+ "metadata": {},
+ "source": [
+ "### 6.1 Configure `Spec2Pipeline`\n",
+ "The `Spec2Pipeline` has the following steps available for NIRSpec MOS:\n",
+ "\n",
+ "* `assign_wcs`: Associates a WCS object with each exposure.\n",
+ "* `msaflagopen`: Flags pixels in NIRSpec exposures affected by MSA shutters stuck in the open position.\n",
+ "* `nsclean`: Cleans 1/f noise.\n",
+ "* `imprint`: Removes patterns caused by the MSA structure in NIRSpec MOS and IFU exposures.\n",
+ "* `bkg_subtract`: Performs image subtraction for background removal.\n",
+ "* `extract_2d` : Extracts 2D arrays from spectral images.\n",
+ "* `srctype`: Determines whether a spectroscopic source should be classified as a point or extended object.\n",
+ "* `master_background` : Subtracts background signal from 2D spectroscopic data using a 1D master background spectrum.\n",
+ "* `wavecorr` : Updates wavelength assignments for FS and MOS point sources that are offset in the dispersion direction within their slit.\n",
+ "* `flat_field`: Applies flat-field corrections to the input science dataset.\n",
+ "* `pathloss`: Calculates and applies corrections for signal loss in spectroscopic data.\n",
+ "* `barshadow` : Calculates the correction to NIRSpec MOS data for extended sources affected by the bar that separates adjacent microshutters.\n",
+ "* `photom`: Applies photometric calibrations to convert data from countrate to surface brightness or flux density.\n",
+ "* `pixel_replace`: Interpolates and estimates flux values for pixels flagged as DO_NOT_USE in 2D extracted spectra.\n",
+ "* `resample_spec`: Resamples each input 2D spectral image using WCS and distortion information.\n",
+ "* `extract_1d`: Extracts a 1D signal from 2D or 3D datasets.\n",
+ "\n",
+ "For more information about each step and a full list of step arguments, please refer to the official documentation: [JDox](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview/stages-of-jwst-data-processing/calwebb_spec2) and\n",
+ "[ReadtheDocs](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_spec2.html)\n",
+ "\n",
+ "Below, we set up a dictionary that defines how the `Spec2Pipeline` should be configured for MOS data."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e02e9e60-92d4-4037-8c91-7e0f7abf3766",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "If pixel-to-pixel or master background subtraction was chosen above, it will be applied during this stage.\n",
+ "To override specific steps and reference files, use the examples below.\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7544bd22",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Select source to inspect.\n",
+ "# Set None to proccess all sources.\n",
+ "source_id, slit_name = None, None\n",
+ "\n",
+ "if demo_mode:\n",
+ " # Galaxies of interest:\n",
+ " source_id, slit_name = 6355, '72'\n",
+ " # source_id, slit_name = 5144, '51'\n",
+ " # source_id, slit_name = 4590, '55'\n",
+ " # source_id, slit_name = 10612, '69'\n",
+ " # source_id, slit_name = 8140, '16'\n",
+ " # source_id, slit_name = 9922, '64'\n",
+ "\n",
+ "# If running master background subtraction,\n",
+ "# make sure we don't restrict to one slit.\n",
+ "elif not master_bg:\n",
+ " source_id, slit_name = None, None"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "32ba737b-90ff-4044-92fa-e7084ce1a102",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set up a dictionary to define how the Spec2 pipeline should be configured.\n",
+ "\n",
+ "# -------------------------Boilerplate dictionary setup-------------------------\n",
+ "spec2dict = {}\n",
+ "spec2dict['assign_wcs'], spec2dict['nsclean'] = {}, {}\n",
+ "spec2dict['master_background_mos'] = {}\n",
+ "spec2dict['barshadow'], spec2dict['extract_2d'] = {}, {}\n",
+ "spec2dict['bkg_subtract'], spec2dict['srctype'], spec2dict['wavecorr'] = {}, {}, {}\n",
+ "spec2dict['flat_field'], spec2dict['pathloss'] = {}, {}\n",
+ "spec2dict['photom'], spec2dict['pixel_replace'] = {}, {}\n",
+ "spec2dict['resample_spec'], spec2dict['extract_1d'] = {}, {}\n",
+ "\n",
+ "# ---------------------------Override reference files---------------------------\n",
+ "\n",
+ "# Overrides for various reference files (example).\n",
+ "# Files should be in the base local directory or provide full path.\n",
+ "#spec2dict['extract_1d']['override_extract1d'] = 'myfile.json'\n",
+ "\n",
+ "# -----------------------------Set step parameters------------------------------\n",
+ "\n",
+ "# Overrides for whether or not certain steps should be skipped (example).\n",
+ "spec2dict['bkg_subtract']['skip'] = not pixel_bg # Runs if pixel-to-pixel bkg selected.\n",
+ "spec2dict['master_background_mos']['skip'] = not master_bg # Runs if masterbg selected.\n",
+ "\n",
+ "# Run pixel replacement code to extrapolate values for otherwise bad pixels\n",
+ "# This can help mitigate 5-10% negative dips in spectra of bright sources.\n",
+ "# Use the 'fit_profile' algorithm.\n",
+ "#spec2dict['pixel_replace']['skip'] = False\n",
+ "#spec2dict['pixel_replace']['n_adjacent_cols'] = 5\n",
+ "#spec2dict['pixel_replace']['algorithm'] = 'fit_profile'\n",
+ "\n",
+ "# Extract a specific source; saves on processing time.\n",
+ "if slit_name is not None:\n",
+ " spec2dict['extract_2d']['slit_name'] = slit_name\n",
+ "\n",
+ "# Resample weight_type.\n",
+ "spec2dict['resample_spec']['weight_type'] = 'ivm'"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "80827b43-e0e7-4826-bb65-d27e4f7a1346",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ "Resampling 2D spectra (`resample_spec` step)can sometimes introduce artificial noise and reduce the signal-to-noise ratio (SNR) in the resulting 1D spectra when using `weight_type='ivm'` ([known issue](https://jwst-docs.stsci.edu/known-issues-with-jwst-data/nirspec-known-issues/nirspec-mos-known-issues#NIRSpecFSKnownIssues-Resamplingof2-Dspectra&gsc.tab=0:~:text=of%20the%20shutter.-,Outlier%20detection%20and%20resampling,-The%20curvature%20of)). The default is now set to 'exptime'. Consider the following when selecting a `weight_type`:\n",
+ "* **'ivm'**: Inverse variant scaling based on read noise (VAR_RNOISE), ideal for rejecting outliers and better suited for faint sources.\n",
+ "* **'exptime'**: Uses exposure time for scaling, improving SNR for bright sources.\n",
+ "\n",
+ "
\n",
+ "\n",
+ "To correct for 1/f noise with `nsclean` in Stage 2, see the **MOS_NSClean_example** demo notebook for MOS data [here](https://github.com/spacetelescope/jdat_notebooks/tree/main/notebooks/NIRSpec/NIRSpec_NSClean).\n",
+ "\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "59ac83d8-3787-4116-bf02-728abbd3341a",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### 6.2 Create `Spec2Pipeline` Association Files\n",
+ "[Association (ASN) files](https://jwst-pipeline.readthedocs.io/en/stable/jwst/associations/overview.html) define the relationships between multiple exposures, allowing them to get processed as a set rather than individually. Processing an ASN file enables the exposures to be calibrated, archived, retrieved, and reprocessed as a set rather than as individual objects.\n",
+ "\n",
+ "[Stage 2 ASN files](https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/level2_asn_technical.html) for MOS data can include `science` and `background` exposure types. A Stage 2 ASN file requires at least one `science` file but can contain multiple `background` files that enable pixel-to-pixel background subtraction in `calwebb_spec2`.\n",
+ "\n",
+ "There are two types of associations:\n",
+ "* **Observation**: Groups data within a single observation.\n",
+ "* **Candidate**: Groups data from one or multiple observations within the same proposal.\n",
+ "\n",
+ "In `demo_mode`, this notebook downloads Stage 2 ASN files directly from MAST. \n",
+ "\n",
+ "
\n",
+ "\n",
+ "Background subtraction may not be correctly applied if more than *one* `science` file is included in the association.\n",
+ "\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e5e3f9d8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Using the `rate.fits` files output of Detector1 creates the list of data that\n",
+ "# will be used to create Spec2 associations that nodded observations for pixel_background subtrationc\n",
+ "\n",
+ "if dospec2:\n",
+ " new_asn = group_spec2_asn_bkg_nods(rate_sci, number_nods)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "6f255788",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create the assoiciation with the output of Stage 1\n",
+ "# Assumes node_type = 3-SHUTTER-SLITLET\n",
+ "# Define root name of leve2 association\n",
+ "spec2_asn = []\n",
+ "\n",
+ "if dospec2:\n",
+ " # Three exposures per association\n",
+ " exposures = ['science', 'background', 'background']\n",
+ "\n",
+ " # Create associations for the SHUTTER-SLITLET and background\n",
+ " # correction\n",
+ " for asn in new_asn:\n",
+ "\n",
+ " # Create the name of the stage2 product\n",
+ " basename = os.path.basename(asn[0])\n",
+ " root_index = basename.rfind('_')\n",
+ " new_name = basename[0:root_index]\n",
+ "\n",
+ " associations = asn_from_list.asn_from_list(asn,\n",
+ " product_name=new_name)\n",
+ "\n",
+ " associations.data['asn_type'] = 'spec2'\n",
+ " associations.data['program'] = program if 'program' in globals() else \"9999\"\n",
+ "\n",
+ " for i, product in enumerate(associations.data['products'], 1): \n",
+ "\n",
+ " # Separate association for each combination of level2 products\n",
+ " for j, name in enumerate(associations.data['products'], 1):\n",
+ " for k, member in enumerate(name['members'], 1):\n",
+ " member['exptype'] = exposures[k - 1]\n",
+ " asn_filename, serialized = associations.dump(format=\"json\")\n",
+ " spec2_single_asn = os.path.join(asn_dir, asn_filename)\n",
+ "\n",
+ " with open(spec2_single_asn, \"w\") as fd:\n",
+ " fd.write(serialized)\n",
+ "\n",
+ " if isinstance(spec2_single_asn, str):\n",
+ " spec2_asn.append(spec2_single_asn)\n",
+ "\n",
+ " print(f\"Stage 2 ASN Files:\\n{'-'}\\n\" + \"\\n\".join(spec2_asn))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5d1b6ef-40be-48a8-a2d5-bbacd3b1a527",
+ "metadata": {},
+ "source": [
+ "If you wish to edit the MSA metafile, a detailed example notebook is forthcoming ([draft](https://github.com/dancoe/NIRSpec/blob/main/NIRSpec%20MOS%20MSA%20metafile.ipynb)) ..."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "01d0d7b4-dec1-47c7-b8dd-72392a8af6ec",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### 6.4 Run `Spec2Pipeline`\n",
+ "Run the science files and, if available, any background files through the `calwebb_spec2` pipeline using the `.call()` method."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ed273d58-b841-4ac9-bd11-fd95955c34e9",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "\n",
+ "### 6.3 MSA Metadata File\n",
+ "\n",
+ "While it doesn’t contain actual science data, the NIRSpec [MSA metadata file](https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/msa_metadata.html) is a crucial component of calibration processing for NIRSpec MOS exposures through the `calwebb_spec2` pipeline. The MSA metadata contains:\n",
+ "* Information on which MSA shutters (slitlets) are open, which sources they observe (if any), and whether they should be used for science or background in each dithered exposure.\n",
+ "* Source details, including RA, Dec, and whether they should be treated as point sources (stellarity > 0.75) or extended (uniform illumination) for path loss corrections.\n",
+ "\n",
+ "All MSA configurations for an observing program may be stored in a single MSA metadata file.\n",
+ "\n",
+ "The MSA metadata file is stored in the ASN directory."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53465286-79be-4979-a8ed-d0636147d9d2",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "Perform pixel-to-pixel or master background subtraction (if desired) here in Stage 2.\n",
+ "
\n",
+ "\n",
+ "Please note that the demo source data is technically defined and processed as an extended source, even though the plots above are presented in units of Jy (for point sources). For extended sources, the extraction box for the 1D spectra defaults to the center of the slitlet. In these cases, you may need to manually adjust the position of the extraction box.\n",
+ "\n",
+ "Additionally, with point source data, you may sometimes observe that the extraction region is not always perfectly centered on the source. Recent improvements in resampling might exacerbate some off-centered cases, and in certain situations, the source may be completely missed. The workaround for this issue is found in [Section 9](#9.-Modifying-the-EXTRACT1D-Reference-File-(as-needed)).\n",
+ "\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "18e4ce68-4e4b-4cb8-83f0-7bdabecff874",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "\n",
+ "## 7. Stage 3: `Spec3Pipeline` (`calwebb_spec3`)\n",
+ "In this section, we process our calibrated spectra from Stage 2 (`calwebb_spec2`) through the Spec3 (`calwebb_spec3`) pipeline to create Stage 3 [data products](https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/science_products.html).\n",
+ "\n",
+ "* **Input**: An ASN file that lists multiple calibrated exposures (`_cal.fits`) in addition to any background exposures (`_x1d.fits`).\n",
+ "* **Output**: A single calibrated product (rectified and unrectified) and 1D spectrum. These data products have units of MJy/sr (or Jy for extracted point-source spectra).\n",
+ ">* `_cal.fits`: Calibrated 2D (unrectified) spectra (ncols x nrows).\n",
+ ">* `_crf.fits`: Calibrated 2D (unrectified) spectra whose DQ array has been updated to flag pixels detected as outliers (ncols x nrows).\n",
+ ">* `_s2d.fits`: Resampled (rectified) 2D spectra (ncols x nrows).\n",
+ ">* `_x1d.fits`: Extracted 1D spectroscopic data.\n",
+ "\n",
+ "In Stage 3, single files are created for each source, one extension in the file.\n",
+ "\n",
+ "The `Spec3Pipeline` performs additional corrections (e.g., outlier detection) and combines calibrated data from multiple exposures (e.g. a dither/nod pattern) into a single 2D spectral product, as well as a combined 1D spectrum."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "819c62b8-fab1-48cd-b6f3-dbfc7bec3248",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### 7.1 Configure `Spec3Pipeline`\n",
+ "The `Spec3Pipeline` has the following steps available for NIRSpec MOS:\n",
+ "\n",
+ "> * `assign_mtwcs`: Modifies the WCS output frame in each exposure of a Moving Target (MT) observation association.\n",
+ "> * `outlier_detection` : Identification of bad pixels or cosmic-rays that remain in each of the input images.\n",
+ "> * `pixel_replace`: Interpolates and estimates flux values for pixels flagged as DO_NOT_USE in 2D extracted spectra.\n",
+ "> * `resample_spec`: Resamples each input 2D spectral image using WCS and distortion information.\n",
+ "> * `extract_1d`: Extracts a 1D signal from 2D or 3D datasets.\n",
+ "\n",
+ "For more information about each step and a full list of step arguments, please refer to the official documentation: [JDox](https://jwst-docs.stsci.edu/jwst-science-calibration-pipeline-overview/stages-of-jwst-data-processing/calwebb_spec3) and\n",
+ "[ReadtheDocs](https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_spec3.html)\n",
+ "\n",
+ "Below, we set up a dictionary that defines how the `Spec3Pipeline` should be configured for MOS data.\n",
+ "\n",
+ "
\n",
+ "To override specific steps and reference files, use the examples below.\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d07f59c6-8115-48d5-a58a-e204a586b0ff",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set up a dictionary to define how the Spec3 pipeline should be configured.\n",
+ "\n",
+ "# -------------------------Boilerplate dictionary setup-------------------------\n",
+ "spec3dict = {}\n",
+ "spec3dict['assign_mtwcs'], spec3dict['outlier_detection'] = {}, {}\n",
+ "spec3dict['pixel_replace'], spec3dict['resample_spec'] = {}, {}\n",
+ "spec3dict['extract_1d'] = {}\n",
+ "\n",
+ "# ---------------------------Override reference files---------------------------\n",
+ "\n",
+ "# Overrides for various reference files.\n",
+ "# Files should be in the base local directory or provide full path.\n",
+ "#spec3dict['extract_1d']['override_extract1d'] = 'myfile.json'\n",
+ "\n",
+ "# -----------------------------Set step parameters------------------------------\n",
+ "\n",
+ "# Overrides for whether or not certain steps should be skipped (example).\n",
+ "spec3dict['outlier_detection']['skip'] = False\n",
+ "\n",
+ "# Run pixel replacement code to extrapolate values for otherwise bad pixels.\n",
+ "# This can help mitigate 5-10% negative dips in spectra of bright sources.\n",
+ "# Use the 'fit_profile' algorithm.\n",
+ "#spec3dict['pixel_replace']['skip'] = False\n",
+ "#spec3dict['pixel_replace']['n_adjacent_cols'] = 5\n",
+ "#spec3dict['pixel_replace']['algorithm'] = 'fit_profile'\n",
+ "\n",
+ "# Resample weight_type.\n",
+ "#spec3dict['resample_spec']['weight_type'] = 'exptime'"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6b4ee5eb-7a0c-450a-8786-08cff5fc3f7d",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### 7.2 Create `Spec3Pipeline` Association Files\n",
+ "[Stage 3 ASN files](https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/level3_asn_technical.html) for MOS data includes `science` exposure types. A Stage 3 ASN file requires at least one `science` file, although there is usually more than one. **Note that the science exposures should be in the `_cal.fits` format.**\n",
+ "\n",
+ "In this notebook the Stage 3 ASN files are created manually (as shown below). Note, Stage 3 output products will use a root name specified by the `product_name` field in the association file. For this tutorial, the root name is `mos3_association`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7d1b9c65",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "if filters[0]:\n",
+ " obs_pars = filters[0].split(';')\n",
+ " filter = obs_pars[0]\n",
+ " grating = obs_pars[1]\n",
+ "else:\n",
+ " with fits.open(sci_cal[0]) as hduf:\n",
+ " hdr = hduf[0].header\n",
+ " filter = hdr['FILTER']\n",
+ " grating = hdr['GRATING']\n",
+ "\n",
+ "name_spec3 = os.path.basename(sci_cal[0]).split(\"_\")\n",
+ "new_name = '_'.join(name_spec3[0:3]) + '_' + filter + '-' + grating\n",
+ "\n",
+ "associations = asn_from_list.asn_from_list(sci_cal,\n",
+ " rule=DMS_Level3_Base,\n",
+ " product_name=new_name)\n",
+ "\n",
+ "associations.data['asn_type'] = 'spec3'\n",
+ "associations.data['program'] = program if 'program' in globals() else \"9999\"\n",
+ "\n",
+ "# Format association as .json file.\n",
+ "asn_filename, serialized = associations.dump(format=\"json\")\n",
+ "print(asn_filename)\n",
+ "\n",
+ "# Write out association file.\n",
+ "mos_asn = os.path.join(asn_dir, asn_filename)\n",
+ "with open(mos_asn, \"w\") as fd:\n",
+ " fd.write(serialized)\n",
+ "\n",
+ "if isinstance(mos_asn, str):\n",
+ " mos_asn = [mos_asn]\n",
+ "print(f\"Stage 3 ASN Files:\\n{'-' * 20}\\n\" + \"\\n\".join(mos_asn))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8d667c0b",
+ "metadata": {},
+ "source": [
+ "Check that the association files for Stage 3"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1a4b5aaf-113f-4ce4-a25f-6b62b400bbca",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "# Open an ASN file as an example.\n",
+ "# Check that file paths have been correctly updated.\n",
+ "if dospec3:\n",
+ " if isinstance(mos_asn, str):\n",
+ " with open(mos_asn, 'r') as f_obj:\n",
+ " asnfile_data = json.load(f_obj)\n",
+ " elif isinstance(mos_asn, list):\n",
+ " with open(mos_asn[0], 'r') as f_obj:\n",
+ " asnfile_data = json.load(f_obj)\n",
+ " display(JSON(asnfile_data, expanded=True))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cc26a5cf-d3ea-4243-9ca5-58db81861293",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### 7.3 Run `Spec3Pipeline`\n",
+ "Run the science files through the `calwebb_spec3` pipeline using the `.call()` method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "358b5e3d-2609-403d-b130-d5ed56670e4f",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "time_spec3 = time.perf_counter()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4610719b-a91b-4c7f-93a0-657d40977aa5",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "# Run Stage 3 pipeline using the custom spec3dict dictionary.\n",
+ "\n",
+ "if dospec3:\n",
+ " for s3_asn in mos_asn:\n",
+ " print(f\"Applying Stage 3 Corrections & Calibrations to: \"\n",
+ " f\"{os.path.basename(s3_asn)}\")\n",
+ " spec3_result = Spec3Pipeline.call(s3_asn,\n",
+ " save_results=True,\n",
+ " steps=spec3dict,\n",
+ " output_dir=spec3_dir)\n",
+ " print(\"Stage 3 has been completed! \\n\")\n",
+ "else:\n",
+ " print(\"Skipping Stage 3. \\n\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c69d91d8-0a6a-43e7-a04c-989222f046c6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Print output result details:\n",
+ "#spec3_result.__dict__ # View entire contents.\n",
+ "#spec3_result.meta.filename\n",
+ "#spec3_result.data.shape"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "9f96ea59-c090-4345-83e8-cb0fa84704ca",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Print out the time benchmarks.\n",
+ "time4 = time.perf_counter()\n",
+ "print(f\"Runtime so far: {round((time4 - time0) / 60.0, 1):0.4f} min\")\n",
+ "print(f\"Runtime for Spec3: {round((time4 - time_spec3) / 60.0, 1):0.4f} min\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "82dbc13e-cfc2-45d2-b54e-c496acf5cee4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# List the Stage 3 products.\n",
+ "\n",
+ "stage3_cal = sorted(glob.glob(spec3_dir + '*_cal.fits'))\n",
+ "stage3_s2d = sorted(glob.glob(spec3_dir + '*_s2d.fits'))\n",
+ "stage3_x1d = sorted(glob.glob(spec3_dir + '*_x1d.fits'))\n",
+ "\n",
+ "print(f\"Stage 3 CAL Products:\\n{'-' * 20}\\n\" + \"\\n\".join(stage3_cal))\n",
+ "print(f\"Stage 3 S3D Products:\\n{'-' * 20}\\n\" + \"\\n\".join(stage3_s2d))\n",
+ "print(f\"Stage 3 X1D Products:\\n{'-' * 20}\\n\" + \"\\n\".join(stage3_x1d))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "32708697",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "## 8. Visualize the Data\n",
+ "Define convenience funcitons for visualization.\n",
+ "\n",
+ "Function to display Stage 1 products."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3bf8385c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def display_rate(rates,\n",
+ " slits_models=[],\n",
+ " integration=0,\n",
+ " extname='data',\n",
+ " cmap='viridis',\n",
+ " bad_color=(1, 0.7, 0.7),\n",
+ " vmin=None,\n",
+ " vmax=None,\n",
+ " scale='asinh',\n",
+ " aspect='auto',\n",
+ " title_prefix=None,\n",
+ " title_path=False,\n",
+ " save_plot=False):\n",
+ " \"\"\"\n",
+ " Display countrate images.\n",
+ "\n",
+ " Parameters\n",
+ " ----------\n",
+ " rates : list of str\n",
+ " A list of RATE[INTS] files to be displayed.\n",
+ " slits_models : list of str, optional\n",
+ " A list of CAL[INTS] or S2D files containing the slit models.\n",
+ " If provided, slit cutouts will be overlaid on the countrate images.\n",
+ " integration : {None, 'min', int}, optional\n",
+ " Specifies the integration to use for multi-integration data.\n",
+ " If 'min', the minimum value across all integrations is used.\n",
+ " If an integer, the specific integration index is used (default 0).\n",
+ " extname : str, optional\n",
+ " The name of the data extension to extract from ('data', 'dq', etc.).\n",
+ " cmap : str, optional\n",
+ " Colormap to use for displaying the image. Default is 'viridis'.\n",
+ " bad_color : tuple of float, optional\n",
+ " Color to use for NaN pixels. Default is light red (1, 0.7, 0.7).\n",
+ " vmin : float, optional\n",
+ " Minimum value for color scaling. If None, determined from the data.\n",
+ " vmax : float, optional\n",
+ " Maximum value for color scaling. If None, determined from the data.\n",
+ " scale : {'linear', 'log', 'asinh'}, optional\n",
+ " Scale to use for the image normalization. Default is 'asinh'.\n",
+ " aspect : str, optional\n",
+ " Aspect ratio of the plot. Default is 'auto'.\n",
+ " title_prefix : str, optional\n",
+ " Optional prefix for the plot title.\n",
+ " title_path : bool, optional\n",
+ " If True, uses the full file path for the title;\n",
+ " otherwise, uses the basename. Default is False.\n",
+ " save_plot : bool, optional\n",
+ " If True, saves the plot as a PNG file. Default is False.\n",
+ "\n",
+ " Returns\n",
+ " -------\n",
+ " None.\n",
+ " \"\"\"\n",
+ "\n",
+ " # -------------------------------Check Inputs-------------------------------\n",
+ " rates = [rates] if isinstance(rates, str) else rates\n",
+ " slits_models = [slits_models] if isinstance(slits_models, str) else slits_models\n",
+ " nrates = len(rates)\n",
+ "\n",
+ " # ------------------------------Set up figures------------------------------\n",
+ " fig, axes = plt.subplots(nrates, 1, figsize=(12, 12 * nrates),\n",
+ " sharex=True, height_ratios=[1] * nrates)\n",
+ " fig.subplots_adjust(hspace=0.2, wspace=0.2)\n",
+ " axes = [axes] if nrates == 1 else axes\n",
+ "\n",
+ " cmap = plt.get_cmap(cmap) # Set up colormap and bad pixel color.\n",
+ " cmap.set_bad(bad_color, 1.0)\n",
+ "\n",
+ " # ---------------------------Plot countrate image---------------------------\n",
+ " for i, (rate, cal) in enumerate(itertools.zip_longest(rates,\n",
+ " slits_models,\n",
+ " fillvalue=None)):\n",
+ "\n",
+ " # -------------------Open files as JWST datamodels-------------------\n",
+ " model = datamodels.open(rate)\n",
+ " slits_model = datamodels.open(cal) if cal else None\n",
+ "\n",
+ " # -----------------------Extract the 2D/3D data----------------------\n",
+ " data_2d = getattr(model, extname)\n",
+ " if data_2d.ndim == 3: # Handle multi-integration data.\n",
+ " if integration == 'min':\n",
+ " data_2d = np.nanmin(data_2d, axis=0)\n",
+ " elif isinstance(integration, int) and 0 <= integration < data_2d.shape[0]:\n",
+ " data_2d = data_2d[integration]\n",
+ " else:\n",
+ " raise ValueError(f\"Invalid integration '{integration}' for 3D data.\")\n",
+ "\n",
+ " # ---------------------------Scale the data-------------------------\n",
+ " sigma_clipped_data = sigma_clip(data_2d, sigma=5, maxiters=3)\n",
+ " vmin = np.nanmin(sigma_clipped_data) if vmin is None else vmin\n",
+ " vmax = np.nanmax(sigma_clipped_data) if vmax is None else vmax\n",
+ " stretch_map = {'log': LogStretch(), 'linear': LinearStretch(),\n",
+ " 'asinh': AsinhStretch()}\n",
+ " if scale in stretch_map:\n",
+ " norm = ImageNormalize(sigma_clipped_data,\n",
+ " interval=ManualInterval(vmin=vmin, vmax=vmax),\n",
+ " stretch=stretch_map[scale])\n",
+ " else:\n",
+ " norm = simple_norm(sigma_clipped_data, vmin=vmin, vmax=vmax)\n",
+ "\n",
+ " # -----------------Draw slits and label source ids------------------\n",
+ " # slits_model can be s2d/cal from spec2 - contains slit models for all sources.\n",
+ " if slits_model:\n",
+ " slit_patches = []\n",
+ " for slit in slits_model.slits:\n",
+ " slit_patch = Rectangle((slit.xstart, slit.ystart),\n",
+ " slit.xsize, slit.ysize)\n",
+ " slit_patches.append(slit_patch)\n",
+ " y = slit.ystart + slit.ysize / 2\n",
+ " ha = 'right' if 'nrs1' in rate else 'left'\n",
+ " plt.text(slit.xstart, y, slit.source_id, color='w', ha=ha, va='center',\n",
+ " fontsize=7, path_effects=[], weight='bold')\n",
+ " axes[i].add_collection(PatchCollection(slit_patches, ec='r', fc='None'))\n",
+ "\n",
+ " # ----------------Plot the countrate image & colorbar---------------\n",
+ " plt.subplots_adjust(left=0.05, right=0.85)\n",
+ " im = axes[i].imshow(data_2d, origin='lower', cmap=cmap,\n",
+ " norm=norm, aspect=aspect, interpolation='nearest')\n",
+ " units = model.meta.bunit_data\n",
+ " cbar_ax = fig.add_axes([axes[i].get_position().x1 + 0.02,\n",
+ " axes[i].get_position().y0, 0.02,\n",
+ " axes[i].get_position().height])\n",
+ " cbar = fig.colorbar(im, cax=cbar_ax)\n",
+ " cbar.set_label(units, fontsize=12)\n",
+ "\n",
+ " # -----------------Construct title and axis labels------------------\n",
+ " filename = model.meta.filename\n",
+ " title = (f\"{title_prefix + ' ' if title_prefix else ''}\"\n",
+ " f\"{filename if title_path else os.path.basename(filename)}\")\n",
+ " if integration is not None:\n",
+ " title = title.replace('rateints', f'rateints[{integration}]')\n",
+ " axes[i].set_title(title, fontsize=14)\n",
+ " axes[i].set_xlabel(\"Pixel Column\", fontsize=12)\n",
+ " axes[i].set_ylabel(\"Pixel Row\", fontsize=12)\n",
+ "\n",
+ " # -------------------------Save the figure?-------------------------\n",
+ " if save_plot:\n",
+ " save_plot = rate.replace('fits', 'png')\n",
+ " if integration:\n",
+ " save_plot = save_plot.replace('.png', '%s.png' % integration)\n",
+ " fig.savefig(save_plot, dpi=200)\n",
+ "\n",
+ " fig.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e7324faa",
+ "metadata": {},
+ "source": [
+ "Function to create plots of the Stage 2/3 spectra."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "bdc4ffd3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def display_spectra(spectra,\n",
+ " compare_x1d=None,\n",
+ " compare_mast=None,\n",
+ " integration=None,\n",
+ " extname='data',\n",
+ " source_id=None,\n",
+ " source_type=None,\n",
+ " expand_wavelength_gap=True,\n",
+ " plot_resample=True,\n",
+ " plot_errors=False,\n",
+ " cmap='viridis',\n",
+ " bad_color=(1, 0.7, 0.7),\n",
+ " aspect='auto',\n",
+ " vmin=None,\n",
+ " vmax=None,\n",
+ " scale='asinh',\n",
+ " title_prefix=None,\n",
+ " title_path=False,\n",
+ " y_limits=None,\n",
+ " is_stage3=False):\n",
+ "\n",
+ " \"\"\"\n",
+ " Display 2D and 1D spectra (Stage 2/3).\n",
+ "\n",
+ " Parameters\n",
+ " ----------\n",
+ " spectra : list of str\n",
+ " A list of data products (e.g., CAL, S2D, X1D files).\n",
+ " compare_x1d : list of str, optional\n",
+ " A list of 1D spectra for comparison (X1D files).\n",
+ " compare_mast : list of str, optional\n",
+ " A list of 1D spectra from MAST for comparison (X1D files).\n",
+ " integration : {None, 'min', int}, optional\n",
+ " Specifies the integration to use for multi-integration data.\n",
+ " If 'min', the minimum value across all integrations is used.\n",
+ " If an integer, the specific integration index is used (default 0).\n",
+ " extname : str, optional\n",
+ " The name of the data extension to extract ('data', 'dq', etc.).\n",
+ " source_id : int or none, optional\n",
+ " Identifier for the source/slit to be displayed. Default is None.\n",
+ " source_type : str, optional\n",
+ " Override data source type ('POINT' or 'EXTENDED').\n",
+ " expand_wavelength_gap : bool, optional\n",
+ " If True, expands gaps in the wavelength data for better visualization.\n",
+ " plot_resample : bool, optional\n",
+ " If True, plots resampled (S2D) data products;\n",
+ " otherwise, plots calibrated (CAL) data. Default is True.\n",
+ " plot_errors : bool, optional\n",
+ " If True, plots the error bands for the 1D spectra. Default is False.\n",
+ " cmap : str, optional\n",
+ " Colormap to use for displaying the images. Default is 'viridis'.\n",
+ " bad_color : tuple of float, optional\n",
+ " Color to use for bad pixels. Default is light red (1, 0.7, 0.7).\n",
+ " aspect : str, optional\n",
+ " Aspect ratio of the plot. Default is 'auto'.\n",
+ " vmin : float, optional\n",
+ " Minimum value for color scaling. If None, determined from the data.\n",
+ " vmax : float, optional\n",
+ " Maximum value for color scaling. If None, determined from the data.\n",
+ " scale : {'linear', 'log', 'asinh'}, optional\n",
+ " Scale to use for the image normalization. Default is 'asinh'.\n",
+ " title_prefix : str, optional\n",
+ " Optional prefix for the plot title.\n",
+ " title_path : bool, optional\n",
+ " If True, uses the full file path for the title;\n",
+ " otherwise, uses the basename. Default is False.\n",
+ " y_limits : tuple of float, optional\n",
+ " Limits for the y-axis of the 1D spectrum plot.\n",
+ " If None, limits are determined from the data.\n",
+ " is_stage3 : bool, optional\n",
+ " Plot stage 3 products? Default is False.\n",
+ "\n",
+ " Returns\n",
+ " -------\n",
+ " None.\n",
+ " \"\"\"\n",
+ "\n",
+ " # ---------------------------------Check Inputs---------------------------------\n",
+ " spectra = [spectra] if isinstance(spectra, str) else spectra\n",
+ " compare_x1d = [compare_x1d] if isinstance(compare_x1d, str) else compare_x1d\n",
+ " compare_mast = [compare_mast] if isinstance(compare_mast, str) else compare_mast\n",
+ "\n",
+ " # Assign a default source_id if one was not supplied.\n",
+ " if source_id is None:\n",
+ " ftype = \"cal\"\n",
+ " if plot_resample:\n",
+ " ftype = \"s2d\"\n",
+ " for file in spectra:\n",
+ " if ftype in file:\n",
+ " source_id = datamodels.open(file)[0].slits[0].source_id\n",
+ " break\n",
+ "\n",
+ " src_str = str(source_id)\n",
+ "\n",
+ " # Plot stage 3 products?\n",
+ " if is_stage3:\n",
+ "\n",
+ " # Stage 3 products should include the source_id in the filename.\n",
+ " # Sort based on filename rather than open all.\n",
+ " def filter_prod(products, source_id):\n",
+ " \"\"\"Filter products based on the source_id.\"\"\"\n",
+ "\n",
+ " return [\n",
+ " f for f in products\n",
+ " if src_str.lower() in f and ('FXD_SLIT' not in fits.getheader(f, ext=0) or fits.getheader(f, ext=0)['FXD_SLIT'].lower() == src_str.lower())]\n",
+ "\n",
+ " spectra = filter_prod(spectra, source_id)\n",
+ " compare_x1d = filter_prod(compare_x1d, source_id) if compare_x1d else None\n",
+ " compare_mast = filter_prod(compare_mast, source_id) if compare_mast else None\n",
+ "\n",
+ " ftypes = {ftype: [f for f in spectra\n",
+ " if ftype in f] for ftype in [\"cal\", \"s2d\", \"x1d\"]}\n",
+ " products = sorted(ftypes['s2d']) if plot_resample else sorted(ftypes['cal'])\n",
+ " if not products:\n",
+ " raise ValueError(\"No valid data products found for plotting.\")\n",
+ "\n",
+ " # --------------------------------Set up figures-------------------------------\n",
+ " total_plots = len(products) + bool(ftypes['x1d'])\n",
+ " height_ratios = [1] * len(products) + ([3] if bool(ftypes['x1d']) else [])\n",
+ " fig, axes = plt.subplots(total_plots, 1, figsize=(15, 5 * total_plots),\n",
+ " sharex=False, height_ratios=height_ratios)\n",
+ " fig.subplots_adjust(hspace=0.2, wspace=0.2)\n",
+ " ax2d, ax1d = (axes[:-1], axes[-1]) if bool(ftypes['x1d']) else (axes, None)\n",
+ "\n",
+ " cmap = plt.get_cmap(cmap) # Set up colormap and bad pixel color.\n",
+ " cmap.set_bad(bad_color, 1.0)\n",
+ " colors = plt.get_cmap('tab10').colors\n",
+ " color_cycle = itertools.cycle(colors)\n",
+ "\n",
+ " # ---------------------------------Plot spectra--------------------------------\n",
+ " for i, product in enumerate(products):\n",
+ " model = datamodels.open(product) # Open files as JWST datamodels.\n",
+ "\n",
+ " # Extract the correct 2D source spectrum if there are multiple.\n",
+ " slit_m = model\n",
+ " if 'slits' in model:\n",
+ " slits = model.slits\n",
+ " slit_m = next((s for s in slits\n",
+ " if getattr(s, 'name', None) == source_id), None)\n",
+ " slit_m = slit_m or next((s for s in model.slits\n",
+ " if s.source_id == source_id), None)\n",
+ " if not slit_m:\n",
+ " print(f\"'{source_id}' not found/invalid.\")\n",
+ " print(f\"Available source_ids: {[s.source_id for s in slits][:5]}\")\n",
+ " break\n",
+ "\n",
+ " # Check if 'fixed_slit' exists, otherwise fall back to 'slitlet_id'\n",
+ " slit_name = (f\"SLIT: {getattr(slit_m, 'name', None) or slit_m.slitlet_id}, \"\n",
+ " f\"SOURCE: {getattr(slit_m, 'source_id', '')}\")\n",
+ "\n",
+ " # -----------------------Extract the 2D/3D data----------------------\n",
+ " data_2d = getattr(slit_m, extname)\n",
+ " if data_2d.ndim == 3: # Handle multi-integration data.\n",
+ " if integration == 'min':\n",
+ " data_2d = np.nanmin(data_2d, axis=0)\n",
+ " elif isinstance(integration, int) and 0 <= integration < data_2d.shape[0]:\n",
+ " data_2d = data_2d[integration]\n",
+ " else:\n",
+ " raise ValueError(f\"Invalid integration '{integration}' for 3D data.\")\n",
+ "\n",
+ " # -----------Convert from pixels to wavelength (x-axis)--------------\n",
+ " wcsobj = slit_m.meta.wcs # Obtaining the WCS object from the meta data.\n",
+ " y, x = np.mgrid[:slit_m.data.shape[0], :slit_m.data.shape[1]]\n",
+ " # Coordinate transform from detector space (pixels) to sky (RA, DEC).\n",
+ " det2sky = wcsobj.get_transform('detector', 'world')\n",
+ " ra, dec, s2dwave = det2sky(x, y) # RA/Dec, wavelength (microns) for each pixel.\n",
+ " s2dwaves = s2dwave[0, :] # Single row since this is the rectified spectrum.\n",
+ " x_arr = np.arange(0, slit_m.data.shape[1], int(len(slit_m.data[1]) / 4))\n",
+ " wav = np.round(s2dwaves[x_arr], 2) # Populating the wavelength array.\n",
+ " ax2d[i].set_xticks(x_arr, wav)\n",
+ "\n",
+ " # xticks = np.arange(np.ceil(wave_1d[0]), wave_1d[-1], 0.2)\n",
+ " # xtick_pos = np.interp(xticks, wave_1d, np.arange(num_waves))\n",
+ " # ax1d.set_xticks(xtick_pos)\n",
+ " # ax1d.set_xticklabels([f'{xtick:.1f}' for xtick in xticks])\n",
+ "\n",
+ " # ---------------------------Scale the data-------------------------\n",
+ " sigma_clipped_data = sigma_clip(data_2d, sigma=5, maxiters=3)\n",
+ " vmin = np.nanmin(sigma_clipped_data) if vmin is None else vmin\n",
+ " vmax = np.nanmax(sigma_clipped_data) if vmax is None else vmax\n",
+ " stretch_map = {'log': LogStretch(), 'linear': LinearStretch(),\n",
+ " 'asinh': AsinhStretch()}\n",
+ " if scale in stretch_map:\n",
+ " norm = ImageNormalize(sigma_clipped_data,\n",
+ " interval=ManualInterval(vmin=vmin, vmax=vmax),\n",
+ " stretch=stretch_map[scale])\n",
+ " else:\n",
+ " norm = simple_norm(sigma_clipped_data, vmin=vmin, vmax=vmax)\n",
+ "\n",
+ " # -------------------------Plot 1D Spectra-------------------------\n",
+ " for k, (prods_1d, prefix) in enumerate([(sorted(ftypes['x1d']),\n",
+ " f'{title_prefix} '),\n",
+ " (compare_x1d, 'RE-EXTRACTION '),\n",
+ " (compare_mast, 'MAST ')]):\n",
+ " if prods_1d:\n",
+ " model_1d = datamodels.open(prods_1d[i])\n",
+ " specs = model_1d.spec\n",
+ " spec = next((s for s in specs if\n",
+ " getattr(s, 'name', None) == source_id), None)\n",
+ " spec = spec or next((s for s in specs\n",
+ " if s.source_id == source_id), None)\n",
+ "\n",
+ " if spec:\n",
+ " tab = spec.spec_table\n",
+ " source_type = source_type if source_type else slit_m.source_type\n",
+ " wave = tab.WAVELENGTH\n",
+ " flux = tab.FLUX if source_type == 'POINT' else tab.SURF_BRIGHT\n",
+ " errs = tab.FLUX_ERROR if source_type == 'POINT' else tab.SB_ERROR\n",
+ "\n",
+ " # Expand the array to visualize the wavelength gap.\n",
+ " if expand_wavelength_gap:\n",
+ " dx1d_wave = wave[1:] - wave[:-1]\n",
+ " igap = np.argmax(dx1d_wave)\n",
+ " dx_replace = (dx1d_wave[igap - 1] + dx1d_wave[igap + 1]) / 2.\n",
+ " nfill = int(np.round(np.nanmax(dx1d_wave) / dx_replace))\n",
+ "\n",
+ " if nfill > 1:\n",
+ " print(nfill)\n",
+ " print(f\"Expanding wavelength gap {wave[igap]:.2f} \"\n",
+ " f\"-- {wave[igap + 1]:.2f} μm\")\n",
+ "\n",
+ " wave_fill = np.mgrid[wave[igap]:wave[igap + 1]:(nfill + 1) * 1j]\n",
+ " wave = np.concatenate([wave[:igap + 1],\n",
+ " wave_fill[1:-1],\n",
+ " wave[igap + 1:]])\n",
+ "\n",
+ " if prefix != 'RE-EXTRACTION ':\n",
+ " num_rows, num_waves = data_2d.shape\n",
+ " fill_2d = np.zeros(shape=(num_rows, nfill - 1)) * np.nan\n",
+ " data_2d = np.concatenate([data_2d[:, :igap + 1],\n",
+ " fill_2d, data_2d[:, igap + 1:]],\n",
+ " axis=1)\n",
+ "\n",
+ " fill = np.zeros(shape=(nfill - 1)) * np.nan\n",
+ " flux = np.concatenate([flux[:igap + 1], fill, flux[igap + 1:]])\n",
+ " errs = np.concatenate([errs[:igap + 1], fill, errs[igap + 1:]])\n",
+ " else:\n",
+ " nfill = 0\n",
+ "\n",
+ " # ----------------Construct legends and annotations-----------------\n",
+ " detector = slit_m.meta.instrument.detector\n",
+ " ffilter = slit_m.meta.instrument.filter\n",
+ " grating = slit_m.meta.instrument.grating\n",
+ " dither = model.meta.dither.position_number\n",
+ " label_2d = f'{grating}/{ffilter}'\n",
+ " label_1d = f'{detector} ({grating}/{ffilter})'\n",
+ " if not is_stage3:\n",
+ " label_2d = f'Dither/Nod {dither} ({label_2d})'\n",
+ " label_1d = (f'{prefix} Dither/Nod {dither} {label_1d}')\n",
+ " else:\n",
+ " label_1d = f'{prefix}{label_1d}'\n",
+ " ax2d[i].annotate(label_2d, xy=(1, 1), xycoords='axes fraction',\n",
+ " xytext=(-10, -10), textcoords='offset points',\n",
+ " bbox=dict(boxstyle=\"round,pad=0.3\",\n",
+ " edgecolor='white',\n",
+ " facecolor='white', alpha=0.8),\n",
+ " fontsize=12, ha='right', va='top')\n",
+ "\n",
+ " title_2d = (f\"{title_prefix + ' ' if title_prefix else ''}\"\n",
+ " f\"{model.meta.filename} | {slit_name}\")\n",
+ " if integration:\n",
+ " title_2d = title_2d.replace('.fits', f'[{integration}].fits')\n",
+ " ax2d[i].set_title(title_2d, fontsize=14)\n",
+ " if not bool(ftypes['x1d']):\n",
+ " ax2d[i].set_xlabel(\"Wavelength (μm)\", fontsize=12)\n",
+ " ax2d[i].set_ylabel(\"Pixel Row\", fontsize=12)\n",
+ " ax2d[i].legend(fontsize=12)\n",
+ "\n",
+ " # ------------------------------------------------------------------\n",
+ "\n",
+ " num_waves = len(wave)\n",
+ " color = next(color_cycle)\n",
+ " ax1d.step(wave, flux, lw=1, label=label_1d, color=color)\n",
+ " if plot_errors:\n",
+ " ax1d.fill_between(np.arange(num_waves), flux - errs,\n",
+ " flux + errs, color='grey', alpha=0.3)\n",
+ " ax1d.legend(fontsize=12)\n",
+ " ax1d.set_title(f\"{title_prefix + ' ' if title_prefix else ''}\"\n",
+ " f\"Extracted 1D Spectra | {slit_name}\", fontsize=14)\n",
+ " ax1d.set_ylabel(\"Flux (Jy)\" if source_type == 'POINT'\n",
+ " else \"Surface Brightness (MJy/sr)\", fontsize=12)\n",
+ " ax1d.set_xlabel(\"Wavelength (μm)\", fontsize=12)\n",
+ "\n",
+ " ax1d.set_ylim(y_limits or (np.nanpercentile(flux, 1),\n",
+ " np.nanpercentile(flux, 99.5)))\n",
+ "\n",
+ " # --------------------Plot the 2D spectra & colorbar---------------\n",
+ " plt.subplots_adjust(left=0.05, right=0.85)\n",
+ " if k == 0:\n",
+ " im = ax2d[i].imshow(data_2d, origin='lower',\n",
+ " cmap=cmap, norm=norm,\n",
+ " aspect=aspect, interpolation='nearest')\n",
+ " units = slit_m.meta.bunit_data\n",
+ " cbar_ax = fig.add_axes([ax2d[i].get_position().x1 + 0.02,\n",
+ " ax2d[i].get_position().y0, 0.02,\n",
+ " ax2d[i].get_position().height])\n",
+ " cbar = fig.colorbar(im, cax=cbar_ax)\n",
+ " cbar.set_label(units, fontsize=12)\n",
+ "\n",
+ " # ----------------------Add extraction region---------------------\n",
+ " ystart, ystop, xstart, xstop = (spec.extraction_ystart - 1,\n",
+ " spec.extraction_ystop - 1,\n",
+ " spec.extraction_xstart - 1,\n",
+ " spec.extraction_xstop - 1)\n",
+ " extract_width = ystop - ystart + 1\n",
+ " box = Rectangle((xstart, ystart), xstop - xstart + nfill,\n",
+ " extract_width, fc='None', ec=color,\n",
+ " lw=2, label=prefix)\n",
+ " ax2d[i].add_patch(box)\n",
+ " ax2d[i].legend()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e009f5ed-d91e-4c78-9cee-4ee3b9838e7a",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### 8.1 Display `Detector1Pipeline` Products\n",
+ "Inspect the Stage 1 slope products. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "438d448c-e635-4bec-ae71-9cf6f9d36f62",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "rate_file = rate_sci[-1] # Show the last rate file, as an example.\n",
+ "display_rate(rate_file, vmin=-0.003, vmax=0.022, scale='linear',\n",
+ " title_prefix='REPROCESSED') # , extname='dq')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f32383ef",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "\n",
+ "### 8.2 Display `Spec2Pipeline` Products\n",
+ "Inspect the Stage 2 calibrated spectra. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b4ce6910",
+ "metadata": {},
+ "source": [
+ "Draw boxes around the extraction regions for each source in a `_rate.fits` file using the slit information from the corresponding Stage 2 calibrated products (`_cal.fits` or `_s2d.fits`). These boxes should be large enough to accommodate the curved spectral traces. While neighboring boxes may overlap, the spectra themselves do not."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "71c6001f",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [],
+ "source": [
+ "# Display stage 2 products.\n",
+ "display_spectra(sci_s2d + sci_x1d, source_id=source_id, scale='log',\n",
+ " source_type='POINT', vmin=0, vmax=3, title_prefix='REPROCESSED')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53da1cc7",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "### 8.3 Display `Spec3Pipeline` Products\n",
+ "Inspect the Stage 3 combined calibrated spectra. \n",
+ "\n",
+ "
\n",
+ "You need to provide the slit ID and name for the plot to display. Below we provide values that work with the demo data.\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c4c3ee6d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Select source to inspect. Valid for demo_mode.\n",
+ "if slit_name is None:\n",
+ " # Using a dummy slit for ploting.\n",
+ " source_id, slit_name = 6355, '72'\n",
+ " print(f\"Warning: plotting slit for source ID: \"\n",
+ " f\"{source_id} and slit name: {slit_name}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "1c052958",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Display stage 3 products.\n",
+ "display_spectra(stage3_s2d + stage3_x1d, source_id=source_id, scale='log',\n",
+ " source_type='POINT', vmin=0, vmax=3,\n",
+ " title_prefix='REPROCESSED', is_stage3=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "36842508",
+ "metadata": {},
+ "source": [
+ "Note in the demo data, the default extraction region misses the positive signal for this 5-shutter slitlet target and instead extracts negative signal from one of the nod subtractions. A workaround in [Section 9](#9.-Modifying-the-EXTRACT1D-Reference-File-(as-needed)) is provided."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2c029301-87de-48ba-9513-8b37e0400f4c",
+ "metadata": {},
+ "source": [
+ "---\n",
+ "\n",
+ "## 9. Modifying the EXTRACT1D Reference File (as needed)\n",
+ "[extract_1d](https://jwst-pipeline.readthedocs.io/en/latest/jwst/extract_1d) •\n",
+ "[Editing JSON reference file](https://jwst-pipeline.readthedocs.io/en/latest/jwst/extract_1d/reference_files.html#editing-json-reference-file-format-for-non-ifu-data)\n",
+ "\n",
+ "The `extract_1d` step's `use_source_pos` parameter in Stage 2 generally centers the 1D extraction box on the actual source location effectively and thus doesn't usually require manual adjustment. However, in some cases, adjusting the position of the extraction box by modifying the EXTRACT1D reference file may be useful. The following section demonstrates how to modify which rows in the 2D spectrum (S2D) are used for extracting the 1D spectrum (X1D).\n",
+ "\n",
+ "The EXTRACT1D reference file, along with several other parameter files, can be found in the `CRDS_PATH` directory. While some files, like `.json` files, can be manually edited, we modify them using Python.\n",
+ "\n",
+ "
\n",
+ "\n",
+ "**Warning**: Currently, there is no aperture correction in place for NIRSpec, so the `extract_width` parameter **MUST** remain unchanged (6 pixels wide) to ensure proper flux calibration! The extraction box limits (`ystart` and `ystop`) can be modified; however, if `ystart` and `ystop` do not match the `extract_width`, the `extract_width` takes precedence and is applied symmetrically around the midpoint between `ystart` and `ystop`.\n",
+ "\n",
+ "
![\"Space](\"https://raw.githubusercontent.com/spacetelescope/notebooks/master/assets/stsci_pri_combo_mark_horizonal_white_bkgd.png\")
\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e64444c3-e53f-4489-8762-14250ad9eb67",
+ "metadata": {},
+ "source": [
+ "# NIRSpec MOS Pipeline Notebook"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7ddbe7fe-926b-41bf-a1fa-84802422c604",
+ "metadata": {},
+ "source": [
+ "**Authors**: Dan Coe (dcoe@stsci.edu), NIRSpec branch, with contributions from the NIRSpec team, including Elena Manjavacas, Peter Zeidler, Kayli Glidic, Melanie Clarke, James Muzerolle, Nikolay Nikolov, Chris Hayes, and Alaina Henry, who designed the ERO NIRSpec observations.