raster: Read raster for mask from env variable #2392

wenzeslaus · 2022-05-21T02:31:23Z

Use environment variable GRASS_MASK to obtain the name of the raster map to use as raster map for auto-masking. The name defaults to MASK in the current mapset. Otherwise, the behavior follows the same rules.

Mask can be in another mapset. If it does not exist, mask is not applied. The direct masking rules for values (zero and null is masked out) apply (as when using g.copy raster=xxx,MASK).

It works with r.mask, i.e., r.mask gets the name from the environment variable. While setting and unsetting can be done only by manipulating the environment variable, r.mask will use whatever is the mask name set by the environment variable or the default name. This allows to fully use the r.mask capabilities without a need to reimplement them somewhere else. It will also keep the workflows with and without the variable same, so all the r.mask documentation applies to the advanced case of using the variable. If the mask set by the environment is in a different mapset, r.mask will fail.

An alternative would be to implement this without r.mask, i.e., r.mask does not get the name from the environment variable. Setting and unsetting this mask would be done only by manipulating the environment variable. This would be analogous to GRASS_REGION which cannot be manipulated by g.region and similar to g.copy used for MASK which is not easily mixed with r.mask. However, while GRASS_REGION says what the computational region should be, GRASS_MASK merely says what the mask name is regardless of its presence. We could fail if the raster is not present, but it seems more natural to behave the same as with the default name (MASK). This discussion then leads to the current implementation where r.mask respects whatever GRASS_MASK says.

r.fillnulls and r.in.wms are Python scripts which need to handle mask in a special way. The new functionality makes the implementation simpler and it makes the tools more robust as they don't touch the global mask state anymore. The new implementation is included here.

C library:

Use separate function for name of the mask.
Support arbitrary mask name in the library (updates r.mask.status behavior).
Create a new internal function to separate testing of the presence and testing of the reclassification.

Python library:

MaskManager: Add Python context manager for mask env variable and start tests.
Add MaskManager to the package init file.

Tools:

r.mask: Support user-provided mask name in r.mask.
r.fillnulls: Use a custom mask (but don't set it) instead of moving around the current user mask.
r.in.wms: Refactor into a function and use with-statement.

Tests:

Use env var in r.mapcalc test to disable masking instead of managing MASK file.
Use new mask handling for raster md5 sh test and sync the const test.
Add tests for different for the various situations which can occur with mask.
Use a global variable for the default mask name in tests.

Documentation:

r.mask: Document the new behavior in a basic way and center documentation around the r.mask tool and raster mask rather than a raster called MASK.
r.mask: Rewrite whole section about different masks.
Extent parallelization notebook.
Update best practices for managing mask and document best practices more specifically for Python.
Add GRASS_MASK to env vars doc.
Use raster mask, not MASK in r.mask documentation in code.
Replace MASK by mask in source code comments.

As part of the Markdown transition, it updates the Markdown doc with new content, but also the originally modified HTML files.

Review or revise code in this PR:

scripts/r.fillnulls/r.fillnulls.py (manipulates to handl mask in its own way)
scripts/r.in.wms/wms_base.py (manipulates to have custom mask)

Code changes implemented elsewhere:

raster/r.kappa/mask.c (mask to text including categories) r.report: Work with any mask name (also for r.kappa) #4633
raster/r.report/maskinfo.c (mask to text including categories) r.report: Work with any mask name (also for r.kappa) #4633
raster/r.random.cells/init.c () raster: Work with any mask name (r.surf.contour, r.random.cells, r.random.surface) #4634
raster/r.random.surface/init.c (opens by name) raster: Work with any mask name (r.surf.contour, r.random.cells, r.random.surface) #4634
raster/r.surf.contour/main.c (opens by name) raster: Work with any mask name (r.surf.contour, r.random.cells, r.random.surface) #4634
raster/r.volume/main.c (opens by name) r.volume: Work with any mask name #4632

Use variable:

testsuite/raster/raster_md5test.sh (in this PR)
raster/r.mapcalc/testsuite/const_map_test.sh (in this PR)

Use r.mask.status tool:

python/grass/grassdb/history.py wxGUI: Use r.mask.status to get info about mask #4519
gui/wxpython/core/gconsole.py wxGUI: Use r.mask.status to get info about mask #4519

Update doc inside this PR:

Update doc outside of this PR:

Changes needed:

lib/init/grass.py init: Use mask, not MASK in shell prompt #5003

Use environment variable GRASS_MASK to obtain the name of the raster map to use as raster map for auto-masking. Mask can be in another mapset. If it does not exist, mask is not applied. The direct masking rules for values (zero and null is masked out) apply (as when using `g.copy raster=xxx,MASK`). This is implemented as an alternative to using r.mask, i.e., r.mask does not get the name from the environment variable. Setting and unsetting is done by manipulating the environment variable. This is analogous to GRASS_REGION which cannot be manipulated by g.region and similar to g.copy used for MASK which cannot be mixed with r.mask -r.

landam · 2023-11-20T17:28:16Z

@wenzeslaus Please provide details on what is missing in order to review this PR?

On many places (more than covered here), 2D raster mask is called MASK conflating the concept (mask) and the implementation (MASK raster). Users using the r.mask tool may not interact with the underlying raster directly, so there is no reason to use MASK over mask. This is leaving many places as they are with MASK. Some will be better revisited with or after OSGeo#2390 and OSGeo#2392 when a more comprehensive solution is available. This fixes and keeps in sync wording r.null and r.external, and moves r.circle comment documenting the interface to a flag description.

To avoid asking about presence of the MASK raster, add a library function which checks for presence of the raster hiding its name in the library. This prepares way for OSGeo#2392. This also changes the message from using MASK to simply mask. I'm open to suggestion on wording of 'mask is present' versus 'mask is active' etc.

On many places (more than covered here), 2D raster mask is called MASK conflating the concept (mask) and the implementation (MASK raster). Users using the r.mask tool may not interact with the underlying raster directly, so there is no reason to use MASK over mask. This is changing MASK to mask and modifies related wording. However, this is also leaving many places as they are with MASK. Some will be better revisited with or after #2390 and #2392 when a more comprehensive solution is available. This fixes and keeps in sync wording in r.null and r.external, and moves r.circle comment documenting the interface to a flag description.

…MASK file.

wenzeslaus · 2025-02-24T21:03:21Z

Documentation updated both to reflect the latest review and to use Markdown. This is now ready for a new round of reviews. Please, review the .md files.

nilason · 2025-02-25T15:14:51Z

CMake fails with:

test_r_mask stderr

FE
======================================================================
ERROR: tearDownClass (__main__.TestRMask)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "etc/python/grass/gunittest/case.py", line 1318, in runModule
    module.run()
  File "etc/python/grass/pygrass/modules/interface/module.py", line 833, in run
    self.wait()
  File "etc/python/grass/pygrass/modules/interface/module.py", line 854, in wait
    raise CalledModuleError(
grass.exceptions.CalledModuleError: Module run `r.mask maskcats=* layer=1 -r` ended with an error.
The subprocess ended with a non-zero return code: 1. See the following errors:
b'Traceback (most recent call last):\n  File "scripts/r.mask", line 252, in <module>\n    main()\n  File "scripts/r.mask", line 101, in main\n    mask_status = gs.parse_command("r.mask.status", format="json")\n  File "etc/python/grass/script/core.py", line 612, in parse_command\n    res = read_command(*args, **kwargs)\n  File "etc/python/grass/script/core.py", line 550, in read_command\n    process = pipe_command(*args, **kwargs)\n  File "etc/python/grass/script/core.py", line 516, in pipe_command\n    return start_command(*args, **kwargs)\n  File "etc/python/grass/script/core.py", line 437, in start_command\n    return Popen(args, **popts)\n  File "etc/python/grass/script/core.py", line 84, in __init__\n    subprocess.Popen.__init__(self, args, **kwargs)\n  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__\n    self._execute_child(args, executable, preexec_fn, close_fds,\n  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'r.mask.status\'\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/r.mask/testsuite/test_r_mask.py", line 30, in tearDownClass
    cls.runModule("r.mask", flags="r")
  File "etc/python/grass/gunittest/case.py", line 1335, in runModule
    raise CalledModuleError(
grass.exceptions.CalledModuleError: Module run `r.mask(maskcats='*', layer='1', flags='r')` ended with an error.
The subprocess ended with a non-zero return code: 1. See the following errors:
Traceback (most recent call last):
  File "scripts/r.mask", line 252, in <module>
    main()
  File "scripts/r.mask", line 101, in main
    mask_status = gs.parse_command("r.mask.status", format="json")
  File "etc/python/grass/script/core.py", line 612, in parse_command
    res = read_command(*args, **kwargs)
  File "etc/python/grass/script/core.py", line 550, in read_command
    process = pipe_command(*args, **kwargs)
  File "etc/python/grass/script/core.py", line 516, in pipe_command
    return start_command(*args, **kwargs)
  File "etc/python/grass/script/core.py", line 437, in start_command
    return Popen(args, **popts)
  File "etc/python/grass/script/core.py", line 84, in __init__
    subprocess.Popen.__init__(self, args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'r.mask.status'


======================================================================
FAIL: test_mask (__main__.TestRMask)
Mask test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "etc/python/grass/gunittest/case.py", line 1390, in assertModule
    module.run()
  File "etc/python/grass/pygrass/modules/interface/module.py", line 833, in run
    self.wait()
  File "etc/python/grass/pygrass/modules/interface/module.py", line 854, in wait
    raise CalledModuleError(
grass.exceptions.CalledModuleError: Module run `r.mask raster=lakes maskcats=* layer=1` ended with an error.
The subprocess ended with a non-zero return code: 1. See the following errors:
b'Traceback (most recent call last):\n  File "scripts/r.mask", line 252, in <module>\n    main()\n  File "scripts/r.mask", line 101, in main\n    mask_status = gs.parse_command("r.mask.status", format="json")\n  File "etc/python/grass/script/core.py", line 612, in parse_command\n    res = read_command(*args, **kwargs)\n  File "etc/python/grass/script/core.py", line 550, in read_command\n    process = pipe_command(*args, **kwargs)\n  File "etc/python/grass/script/core.py", line 516, in pipe_command\n    return start_command(*args, **kwargs)\n  File "etc/python/grass/script/core.py", line 437, in start_command\n    return Popen(args, **popts)\n  File "etc/python/grass/script/core.py", line 84, in __init__\n    subprocess.Popen.__init__(self, args, **kwargs)\n  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__\n    self._execute_child(args, executable, preexec_fn, close_fds,\n  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child\n    raise child_exception_type(errno_num, err_msg, err_filename)\nFileNotFoundError: [Errno 2] No such file or directory: \'r.mask.status\'\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "scripts/r.mask/testsuite/test_r_mask.py", line 35, in test_mask
    self.assertModule(module)
  File "etc/python/grass/gunittest/case.py", line 1410, in assertModule
    self.fail(self._formatMessage(msg, stdmsg))
AssertionError: Running <r.mask> module ended with non-zero return code (1)
Called (Python): r.mask(raster='lakes', maskcats='*', layer='1')
Called (Bash): r.mask raster=lakes maskcats=* layer=1
See the following errors:
Traceback (most recent call last):
  File "scripts/r.mask", line 252, in <module>
    main()
  File "scripts/r.mask", line 101, in main
    mask_status = gs.parse_command("r.mask.status", format="json")
  File "etc/python/grass/script/core.py", line 612, in parse_command
    res = read_command(*args, **kwargs)
  File "etc/python/grass/script/core.py", line 550, in read_command
    process = pipe_command(*args, **kwargs)
  File "etc/python/grass/script/core.py", line 516, in pipe_command
    return start_command(*args, **kwargs)
  File "etc/python/grass/script/core.py", line 437, in start_command
    return Popen(args, **popts)
  File "etc/python/grass/script/core.py", line 84, in __init__
    subprocess.Popen.__init__(self, args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'r.mask.status'


----------------------------------------------------------------------
Ran 1 test in 0.429s
FAILED (failures=1, errors=1)

nilason · 2025-02-25T15:17:30Z

There is testreport as artefact to the cmake workflow, a bit tedious to compare. Perhaps better to use a test config file instead of per cent success.

nilason · 2025-02-25T15:20:12Z

r.mask.status needs to be added to https://github.com/OSGeo/grass/blob/main/raster/CMakeLists.txt

nilason · 2025-02-25T18:50:22Z

r.mask.status will be added with #5211

wenzeslaus · 2025-02-25T18:57:59Z

Thanks @nilason. I just assumed config is used for tests, so I thought all the unrelated failures are not on main. Fixed now. I saw that earlier, but it was great to experience that no boilerplate is needed for adding a tool to CMake.

nilason · 2025-02-25T19:04:36Z

048918c will not be enough, see #5211. I don't mind if you bring it in here (and I can close that). CI's slow now.

wenzeslaus · 2025-02-26T16:44:29Z

This and 21 other PRs made a small revolution in how raster mask can be used in GRASS (multiple masks, masks in other mapsets, set and unset without r.mask or creating and deleting raster) as well as in how mask is presented to the user (no more MASK in the doc to refer to "all things mask"). This will be part of upcoming v8.5, so please test.

Basic documentation and tests provided, see especially:

wenzeslaus added this to the 8.4.0 milestone May 24, 2022

wenzeslaus added enhancement New feature or request C Related code is in C labels May 25, 2022

wenzeslaus modified the milestones: 8.3.0, 8.4.0 Feb 10, 2023

Merge remote-tracking branch 'upstream/main' into mask-env-var

4f7b08a

landam assigned wenzeslaus Nov 20, 2023

wenzeslaus mentioned this pull request Mar 14, 2024

[Bug] Display commands for d.mon can take a long time *unexpectedly* to update the cmd file when first run. #3481

Closed

wenzeslaus modified the milestones: 8.4.0, 8.5.0 Apr 26, 2024

wenzeslaus mentioned this pull request Sep 26, 2024

r.mask.status: Check mask status through a tool #2390

Merged

wenzeslaus mentioned this pull request Sep 27, 2024

doc: Use lowercase for 2D raster mask #4401

Merged

Merge remote-tracking branch 'upstream/main' into mask-env-var

1f9a63c

wenzeslaus mentioned this pull request Sep 27, 2024

raster: Add library function to ask for mask presence #4402

Merged

wenzeslaus added 4 commits October 11, 2024 15:08

Use separate function for name of the mask

36f0e67

Merge remote-tracking branch 'upstream/main' into mask-env-var

a9fe1d4

Improve doc

2d987b9

Merge remote-tracking branch 'upstream/main' into mask-env-var

a8b23f6

github-actions bot added raster Related to raster data processing HTML Related code is in HTML libraries module docs labels Oct 11, 2024

Use env var in r.mapcalc test to disable masking instead of managing …

7065da2

…MASK file.

github-actions bot added the tests Related to Test Suite label Oct 14, 2024

Update the Markdown doc with new content

ffa26a5

github-actions bot added the markdown Related to markdown, markdown files label Feb 24, 2025

Be more clear about import tool being an example

a211d14

wenzeslaus requested a review from veroandreo February 24, 2025 21:00

wenzeslaus mentioned this pull request Feb 25, 2025

[WIP] CMake HOW-TOs and Known Issues #5192

Open

veroandreo previously approved these changes Feb 25, 2025

View reviewed changes

Merge branch 'main' into mask-env-var

204bab7

wenzeslaus added 2 commits February 25, 2025 13:53

Add missing r.mask.status entry for CMake

048918c

Merge remote-tracking branch 'fork/mask-env-var' into mask-env-var

327877e

wenzeslaus dismissed veroandreo’s stale review via 327877e February 25, 2025 18:54

github-actions bot added the CMake label Feb 25, 2025

wenzeslaus enabled auto-merge (squash) February 25, 2025 19:21

wenzeslaus mentioned this pull request Feb 25, 2025

cmake: build 'r.mask.status' #5211

Merged

Merge branch 'main' into mask-env-var

04bdd53

wenzeslaus disabled auto-merge February 26, 2025 04:45

wenzeslaus enabled auto-merge (squash) February 26, 2025 04:45

wenzeslaus added 2 commits February 26, 2025 08:40

Wrap long lines reported by Super Linter check

cf35c95

Merge remote-tracking branch 'fork/mask-env-var' into mask-env-var

88ac878

nilason approved these changes Feb 26, 2025

View reviewed changes

wenzeslaus merged commit 512bd6d into OSGeo:main Feb 26, 2025
28 checks passed

wenzeslaus deleted the mask-env-var branch February 26, 2025 15:14

wenzeslaus removed the request for review from petrasovaa February 26, 2025 16:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raster: Read raster for mask from env variable #2392

raster: Read raster for mask from env variable #2392

wenzeslaus commented May 21, 2022 •

edited

Loading

landam commented Nov 20, 2023

wenzeslaus commented Feb 24, 2025

nilason commented Feb 25, 2025

nilason commented Feb 25, 2025

nilason commented Feb 25, 2025

nilason commented Feb 25, 2025

wenzeslaus commented Feb 25, 2025

nilason commented Feb 25, 2025

wenzeslaus commented Feb 26, 2025

raster: Read raster for mask from env variable #2392

raster: Read raster for mask from env variable #2392

Conversation

wenzeslaus commented May 21, 2022 • edited Loading

landam commented Nov 20, 2023

wenzeslaus commented Feb 24, 2025

nilason commented Feb 25, 2025

nilason commented Feb 25, 2025

nilason commented Feb 25, 2025

nilason commented Feb 25, 2025

wenzeslaus commented Feb 25, 2025

nilason commented Feb 25, 2025

wenzeslaus commented Feb 26, 2025

wenzeslaus commented May 21, 2022 •

edited

Loading