JWST Build 7.7 rc1

ami

• Update code to use two new input parameters: psf_offset,rotation_search [#5548]

• Update code and unit tests to use new ami_analyze algorithms [#5390]

• Update ami_analyze to extract a SUB80 subarray from full-frame images [#5437]

assign_wcs

• Add nrs_verify to the NIRSpec exposure list [#5403]

• Enable resample_spec for NIRSpec line lamp exposures [#5484]

• Added SIP approximation to WCS for imaging modes. FITS WCS keywords added to meta.wcsinfo. [#5507]

• Fix bug where subarray bounding boxes were 1 pixel too small. [#5543]

• Mark Nirspec slits which project on less than one pixel as invalid. [#5554]

associations

• Add new dither keyword subpxpts to constraints [#5525]

• Add further constraints to rule Asn_Lv2NRSLAMPSpectral such that associations
  are created only when LAMP is on and OPMODE indicates a valid optical path. [#5496]

• Restrict association creation based on optical path for NIRSpec Fixed-slit and IFU [#5504]

• Asn_Lv3SpecAux: Add optical element constraint [#5479]

• Add utility asn_gather [#5468]

• Do not allow target acqs to be considered TSO [#5385]

• Add NRS_VERIFY to the list of target acq/confirmation images [#5395]

cube_build

• When making SINGLE type cubes for outlier detection or mrs_imatch data not in the
  appropriate channel/grating is skipped [#5347]

• If outlier detection has flagged all the data on a input file as DO_NOT_USE, then
  skip the file in creating an ifucube [*5347]

• Refactor DataTypes handling of ModelContainer. [#5409]

datamodels

• Skip serializing None in datamodels to be compatible with asdf>=2.8 [#5371]

• Implement full class deprecator decorator and use for MIRIRampModel [#5382]

• Add NRS_VERIFY to the core schema as an allowed EXP_TYPE [#5395]

• Remove logging from DataModel.close [#5413]

• Updated keyword schemas for EXP_TYPE and MODULE, to keep in sync with the
  JWST Keyword Dictionary [#5452]

• Added flatfield and photom correction arrays to slit data models [#5460]

• Move core jwst.datamodels code to stdatamodels package and add it as
  an install dependency [#5433]

• Update schemas to include new allowed SUBARRAY values for FGS ASIC tuning
  modes [#5531]

• Add meta.visit.pointing_engdb_quality entry to correspond to ENGQLPTG keyword [#5556]

• Update Moving Target CHEBY table extension schema for changes to column
  definitions in the JWSTKD and SDP [#5558]

extract_1d

• Fixed bug involving the determination of source RA/Dec for resampled Slit
  data. [#5353]

• Updated to use an EXTRACT1D reference file for NIRCam TSGRISM exposures;
  added step param "bkg_fit" to allow for mean and median options in background
  computation, in addition to the existing polynomial fit; fixed bug in
  background computation that was preventing background subtraction from
  ever happening. [#5414]

• Fixed bug involving the processing of WFSS observations when there's only
  one spectrum instance for a given source. [#5439]

fits_generator

• Addressed deprecated get_children method of XML parser. Changed type of PATTSIZE from
  float to string in templates. [#5536]

flatfield

• Fixed bug in sending NIRSpec AUTOWAVE exposures to the spectroscopic
  processing branch. [#5356]

• Updated branch logic to handle NRS_LAMP exposures as spectroscopic. [#5370]

• Updated NIRSpec fixed-slit processing to compute and save correction
  values for both point and uniform sources in the primary slit when it
  contains a point source, in order to support master background corrections.
  [#5462]

jump

• Fixed bug in the minimum number of groups per integration for the jump
  detection step by changing it from 3 to 5. [#5376]

• Various rework to reduce memory usage and increase readability. [#5404]

master_background

• Update the NIRSpec MOS master background logic to only proceed with processing
  after verifying that there are both background and source slits available in
  the input dataset. [#5370]

outlier_detection

• Implement memory check in resample to prevent huge arrays [#5354]

photom

• Updated NIRSpec fixed-slit processing to compute and save correction
  values for both point and uniform sources in the primary slit when it
  contains a point source, in order to support master background corrections.
  [#5463]

pipeline

• Update Image3Pipeline to allow sky subtraction when input contains
  only one image (group). [#5423]
• Enable resample_spec for NIRSpec line lamp exposures in Spec2Pipeline [#5484]

ramp_fitting

• Update to store output as an IFUImageModel for NIRSpec AUTOWAVE exposures
  using the IFU mode. [#5356]

• Update to add 'DO_NOT_USE' DQ flag to pixels with all groups flagged as
  saturated. [#5367]

resample

• Implement memory check in resample to prevent huge arrays [#5354]

• Add pixel_scale_ratio parameter to allow finer output grid. [#5389]

• Enable resample_spec for NIRSpec line lamp exposures [#5484]

reset

• Turn the step back on for the calwebb_detector1 pipeline [#5485]

saturation

• Set saturation threshold to A-to-D limit of 65535 for pixels flagged with
  NO_SAT_CHECK in the saturation reference file, instead of skipping any
  test of those pixels. [#5394]
• Flag groups values below A/D floor (0 DN) (#5422)

set_telescope_pointing

• Add logging of the found quaternion information [#5495]
• Handle cases where engineering database's pointing mnemonics are all zero over the requested time range [#5540]
• Set value of keyword ENGQLPTG to CALCULATED or PLANNED depending on whether pointing telemetry was used to
  update the WCS [#5556]

skymatch

• Fix a bug in skymatch that would result in a crash when skymethod
  contains 'global' and the single image group's sky cannot be computed
  (e.g., because all pixels are flagged as "bad"). [#5440]

stpipe

• Implement utility function all_steps and fix crds reference file retrieval for non-datamodels [#5492]

tso_photometry

• Place aperture using header keywords XREF_SCI and YREF_SCI instead of
  CRPIX1 and CRPIX2 [#5533]

• Fixed the flux units in the output photometry catalog. [#5529]

tweakreg

• Add support for the new fitgeom mode: 'rshift' that can fit only
  for shifts and a rotation. [#5475]

wfs_combine

• Add checking for bad pixels by using DO_NOT_USE rather than DQ>0. [#5500, #5519]

white_light

• Add support for step parameters min_wavelength and max_wavelength to modify
  the wavelength region over which the flux integration is calculated. [#5501]

JWST Build 7.6

Release Notes

The JWST calibration pipeline (v0.17.1) has been made available on PyPi and Github. A high-level list of updates in this build is given below.

Association generator rules have been updated to:

• Fix incorrect product names for level-3 IFU products

Updates to Level-2a processing for all exposure modes include:

• Enabled creation of the "rateints" product for all exposures, even when NINTS=1

Updates to Level-2b processing for all exposure modes include:

• Enabled application of the "filteroffset" correction for NIRCam and NIRISS imaging modes in the assign_wcs step
• The flat-field step has been updated to set DQ='DO_NOT_USE' for reference file pixels that have values of NaN or zero, as well as reference file pixels flagged as 'NO_FLAT'.

Updates to Level-2b spectroscopic (calwebb_spec2) processing include:

• The ability to perform master background subtraction for NIRSpec MOS exposures
• The ability to process NIRSpec internal lamp exposures
• Improved calculation of bounding boxes for grism images in the assign_wcs step
• Updated the assign_wcs step to allow for user-settable extraction heights in grism (WFSS) images
• The ability to apply minimal processing of MIRI MRS TSO exposures
• Changed the default weighting in the cube_build step to "emsm"
• The ability to perform aperture corrections for point source extracted spectra in the extract_1d step
• Bug fixes in flat-fielding for NIRSpec fixed-slit exposures

Updates to the Level-3 imaging (calwebb_image3) processing include:

• The outlier_detection step has been fixed so that it does not subtract background from the science data

Updates to the Level-3 spectroscopic (calwebb_spec3) processing include:

• Implemented corrections to master background subtraction for NIRSpec IFU exposures
• Updated the master_background step to allow for saving of the 2D master background data

Updates to the Level-3 Coronagraphic (calwebb_coron3) pipeline include:

• Implemented bad pixel replacement in the align_refs step
  Updates to the Level-3 TSO (calwebb_tso3) pipeline include:

• Fixed a bug in the computation of integration time stamps

Complete change log:

associations

• Add product name override to the IFUGratingBkg class, to prevent the default
  "clear" suffix showing up in NIRSpec IFU product names. [#5326]

barshadow

• Implement using a user-supplied correction which overrides all references. [#5302]

• Implement applying the inverse operation. [#5302]

blendmeta

• Do not close files that were not opened by blendmodels [#5299]

cube_build

• If every wavelength plane of the IFU cube contains 0 data, cube_build is skipped [#5294]

• Remove "clear" suffix from MIRI MRS product name templates [#5326]

flat_field

• Update how the flat field reference dq mask is used for NIRSpec MOS data [#5284]

• Implement providing a user-supplied flat field which overrides all references. [#5302]

• Implement applying the inverse operation. [#5302]

master_backround

• Create new step MasterBackgroundNrsSlits step to handle NIRSpec MOS data in Spec2Pipeline [#5317]

• Implement option to save the 2d version of the calculated master background [#5317]

outlier_detection

• Fix bug where background was being subtracted on the input data [#4858]

pathloss

• Implement using a user-supplied correction which overrides all references. [#5302]

• Implement applying the inverse operation. [#5302]

photom

• Implement using a user-supplied correction which overrides all references. [#5302]

• Implement applying the inverse operation. [#5302]

pipeline

• Spec3Pipeline check whether master background subtraction has already occurred. [#5308]

• Implement master background subtraction in Spec2Pipeline for NIRSpec MOS data. [#5302]

• Include the per-slit failure traceback in any RuntimeError raised in Spec2Pipeline. [#5315]

ramp_fitting

• Reinstate copying of INT_TIMES table to output rateints product for TSO exposures. [#5321]

tso_photometry

• Fix a bug in the computation of integration time stamps when the INT_TIMES
  table is not available. [#5318]

JWST Build 7.6 rc1

Release Notes

align_refs

• Add bad pixel replacement for target and psf images [#4973]

assign_mtwcs

• Skip the step if any input MT_RA/DEC keyword values are missing. [#5015]

assign_wcs

• Enabled filteroffset correction for NIRISS and NIRCAM imaging modes. [#5018, #5027]

• Pass an optional input_frame parameter in assign_wcs.util.wcs_from_footprintss. [#5120]

• Improved calculation of bounding boxes in grism images. [#5122]

• Added two new optional parameters to utils.cerate_grism_bbox - wfss_extract_half_height
  and wavelength_range. [#5140]

• Shifted the bounding box of a resampled WCS by - 0.5 px to account for the
  center of the pixel. [#5241]

• Enable NIRSpec lamp processing in calspec2 pipeline. [#5267]

associations

• Update diagrams in documentation to change sloper to detector1. [#4986]

• Update level-3 rules to exclude IFU exposures from calwebb_tso3 associations. [#5202]

• Fix formatting error in Asn_IFUGrating product name construction. [#5231]

barshadow

• Correct bar shadow parity bug for yslit. [#5095]

combine_1d

• Skip spectra that are degenerate when combining [#5037]

cube_build

• Changed default weighting to 'emsm'. [#5277]

• Fixed formatting of NIRSpec s3d output product names. [#5231]

• Modified NIRSpec blotting to the find min and max ra and dec for each slice and only
  invert those values on slice that fall in range [#5144]

• Changed default weighting back to 'msm' until NIRSPEC cube pars ref file contains emsm info [#5134]

• Added checks read from cube pars reference file that parameters have valid data [#5134]

• Change the name of default cube type from world to skyalign [#4974]

• Add ifualign cubes to be cubes rotated on sky to align with ifu instrument plane [#4974]

• Change the name of MIRI alpha-beta cube type to internal_cal [#4974]

• Add ability to make NIRSpec internal_cal ifu cubes aligned with slicer plane [#4974]

• Change default weighting from msm to emsm [#4974]

• NIRSpec IFU cubes built from all wavelengths rather than those defined in cube par ref file [#4974]

• Removed wavelength planes that contained only 0 data. These planes are edge cases [#4974]

datamodels

• Add iscopy to ModelContainer init [#5256]

• Re-enable FITS-hash by default. [#5191]

• Add blend rule for keywords DETECTOR and MODULE. [#4998]

• Add methods Model.info and Model.search. [#4660]

• Trim MT_RA, MT_DEC keyword comments to fit within FITS record. [#4994]

• Add enum list and default value of 'NONE' for meta.instrument.lamp_mode [#5022]

• Add TIMEUNIT keyword to schemas. [#5109]

• Split pathloss object into pathloss_ps and pathloss_un in schemas. [#5112]

• Add "PERSISTENCE" DQ flag definition. [#5137]

• Fix nonsensical premature closing of FITS file of a DataModel. [#4930]

• Add a hash set/check to DataModel I/O to check whether schema traversal is necessary. [#5110]

• Update underlying MultiExposureModel from the SourceModelContainer models. [#5154]

• Add new MIRI LRS dither patterns to PATTTYPE enum list. [#5254]

extract_1d

• Implement aperture corrections in the Extract1dStep. [#4902]

• Fix bug in creating a polynomial fit used in background extraction. [#4970]

• Recheck the input model container in run_extract1d to select the correct processing [#5076]

• Rework/refactor many functions for style and readability. [#5079]

• Checks subwcs and new_slit variables exist before trying to delete them. [#5093]

• Parameter mmag_extract is now propagated to the extraction routine. [#5122]

• Updated the logic for when and how to use the source position to offset the
  location of the extraction regions specified in the EXTRACT1D reference file. [#5157]

• Fixed the conversion of flux to surface brightness for IFU extended source case [#5201]

• Fixed bugs in aperture correction for NIRSpec multi-slit modes. [#5260]

extract_2d

• Check that subwcs and new_slit variables exist before trying to delete them [#5093]

• Move NIRSpec wavecorr routines to the wavecorr step. [#5133]

• Added a new optional integer parameter to extract_2d (wfss_extract_half_height)
  which allows a user to specify the extraction height in the
  cross-dispersion direction for WFSS mode. [#5140]

flat_field

• For NIRSpec BOTS and ALLSLITS add the slit start corner to the subarray start corner
  when determining what region of the flat_field reference files to extract. [#5269]

• Enable NIRSpec lamp processing in calspec2 pipeline. [#5267]

fringe

• Update the fringe step to handle 3D inputs for MIRI MRS TSO mode. [#5202]

master_background

• Fix open files bug [#4995]

• Update to include pathloss corrections to NIRSpec IFU background [#5125]

mrs_imatch

• MRSIMatchStep to create its ModelContainers with iscopy=True [#5256]

outlier_detection

• Update median filter to use numpy's nanmedian. [#5114]

• Fix outlier_detection bug when saving intermediate results. [#5108]

• Update logic to correctly handle input CubeModel that have only
  1 integration. [#5211]

pathloss

• Fix bug in NIRSpec IFU data that causes valid pixel dq flags to set to
  NON-SCIENCE in the region of an overlapping bounding box slice [#5047]

• Update to save both point source and uniform source 2D pathloss correction
  arrays to output. [#5112]

persistence

• Flag pixels with high persistence using "PERSISTENCE" DQ flag instead
  of "DO_NOT_USE". [#5137]

pipeline

• Refactor the Image3Pipeline to use stpipe infrastructure. [#4990]

• Fix Coron3Pipeline to blend headers just from each input science model,
  not every integration. [#5007]

• Fix open files bug in get_config_from_reference class method, and in
  Spec2Pipeline, Spec3Pipeline and tso3. [#4995]

• Update calwebb_tso3 to do more robust checking of input data type.
  [#5107]

• Update the Spec2Pipeline to include the new wavecorr step and put
  srctype before wavecorr. [#5133]

• Update the Spec2Pipeline to skip extract_1d for IFU data that
  have not had a cube built (e.g. MIRI MRS TSO), and update the
  calwebb_tso-spec2.cfg configuration to turn on the fringe step
  and turn off cube_build for MIRI MRS TSO. [#5202]

• Update the Coron3Pipeline logic to correctly handle inputs that have
  only 1 integration. [#5211]

• Refactor Spec2Pipeline for execution logic and step flow isolation [#5214]

• Update Ami3Pipeline to only process psf and science members from the
  input ASN. [#5243]

• Enable NIRSpec lamp processing in calspec2 pipeline. [#5267]

photom

• Fix bug in NIRSpec IFU data that causes valid pixel dq flags to set to
  NON-SCIENCE in the region of an overlapping bounding box slice [#5047]

ramp_fitting

• Add multi-processing capability. [#4815]

• Fix crash when DRPFRMS1 is not set [#5096]

• Update to always create the rateints product, even when NINTS=1. [#5211]

resample_spec

• Fix artifacts in resampled NIRSpec slit data caused by NaNs in the WCS [#5217]

source_catalog

• Use gwcs.WCS instead of FITS WCS. [#5120]

• Changed the type of column is_star from float to bool. [#5140]

• Implemented algorithm for determining whether a source is a star.
  [#5234]

stpipe

• Limit reference file prefetch to the first "science" exptype
  when a pipeline has an association as input. [#5031]

• Remove further sloper references. [#4989]

• Enable prefetch of pars reference files for associations. [#5249]

transforms

• Wrap first spherical angle ("RA") at 360 degrees in the forward V23ToSky
  transformation and to 180 degrees for the inverse transformation ("V2").
  This is now done using models defined in astropy and gwcs packages
  replacing V23ToSky model in JWST's WCS pipeline. [#5206]

wavecorr

• Implemented the wavecorr step by pulling routines from the
  extract_2d step. [#5133]

JWST build 7.5

Release Notes

Highlights

The Level-1b script "set_telescope_pointing" has been updated to insert moving target coordinate keywords into level-1b product headers when the coordinate data is available in a moving target position table extension provided by SDP.

Association generator rules have been updated to:

• Limit NIRSpec IFU level 3 associations to contain data from a single grating+filter combination
• Produce level 2b and 3 associations for nodded observations for all spectroscopic modes
• Exclude dark exposures from all associations

Updates to Level-2a processing for all exposure modes include:

• Significant reduction in the memory footprint for processing of all exposure types

Updates to Level-2b processing for all exposure modes include:

• Populate ASNPOOL and ASNTABLE keywords for all products using an ASN to drive processing

Updates to Level-2b spectroscopic (calwebb_spec2) processing include:

• Bug fixes and refactoring of WCS transforms for WFSS exposures

Updates to the Level-3 imaging (calwebb_image3) processing include:

• Only load science and background members from an input ASN file
• The new capability to align image WCS to the Gaia catalog
• Added new capability to align moving target exposures in the moving target coordinate frame
• Source catalog generation has been completely rewritten to include new types of source measurements
• Updates to several steps to make them more robust to garbage data
• Many bug fixes to the creation of combined/resampled NIRSpec and MIRI LRS spectra

Updates to the Level-3 spectroscopic (calwebb_spec3) processing include:

• Update level-2b source-based products to include metadata from input exposures
• 1-D extracted products for WFSS mode now contain all data for a given source in a single x1d product
• Combined 1-D products for WFSS mode use the new product type suffix "c1d"
• Updated mrs_imatch to make it more robust to data that is out of specification

Updates to the Level-3 WFS&C (calwebb_wfs-image3) pipeline include:

• Updated the ASNTABLE keyword in all combined (wfscmb) image products

Full Release notes

assign_wcs

• Update keyword and attribute usage around SkyObject to reflect updated keywords. [#4943]
• Refactor PPS origin of NIRSpec MOS shutters from top left to bottom left. [#4959]
  associations
• Modify NIRSpec IFU level-3 ASN rules to include only one grating per association [#4926]

calwebb_coron3

• Update coron3 for new outlier detection application [#4968]

datamodels

• Add to_container to CubeModel to convert a cube to a list of images [#4968]
• Add getarray_noinit to DataModel to access arrays without causing initialization [#4968]
• Limit looping over HDU's while resolving arrays in schema [#4951]
• Relax asdf requirement and use validator flag when asdf 2.6.x is installed [#4905]
• Updated core schema to include recent Keyword Dictionary changes (remove TIME-END; add TDB-BEG, TDB-MID, TDB-END, XPOSURE, TELAPSE) [#4925]
• Populate meta.asn.table_name when an association is loaded into a ModelContainer. [#4873]

extract_1d

Add aperture correction in extract_1d processing. [#4902]

lib

Update SkyObject keys. [#4943]

mrs_imatch

Fix mrs_imatch to avoid calls to sigma_clipped_stats with all-zero arrays. [#4944]

photom

Fix flux units in photom for MultiSlit cases. [#4958]

pipeline

Updated calwebb_image3 pipeline to only load science and background member types from an input ASN. [#4937]
Updated the calwebb_spec2 pipeline to only use the basename of the source catalog file when updating the source_catalogue keyword for WFSS inputs. [#4940]

rscd

Fixed bug when the READPATT/SUBARRAY data is not found in RSCD reference file [#4934]

source_catalog

Add more concentration indices and update step docs. [#4906, #4908]
Added fallback background estimation method to make background estimation moare robust. [#4929]
Fixed the nearest-neighbor code to handle the case of exactly one detected source. [#4929]
Update abmag error calculation. [#4945]
Exit gracefully if APCORR ref file is missing. [#4948]

tweakreg

Added align_to_gaia processing as an option [#4599]