Populating json colorbar file with environmental variables

[jwarner8]

Currently, json entries are fixed for specific variables (where cmap is defined, along with min/max value for colorbar). Some diagnostics much need adaptive colorbar ranges, such as the age of air diagnostic. This colorbar will extend from 0 to the length of the forecast, defined as $CSET_ANALYSIS_PERIOD. Is there a way we can use $CSET_ANALYSIS_PERIOD in the json and when reading this in the plot operator, it fills this with the appropriate environment value?

also addressed here: #1139

[Sylviabohnenstengel]

@jwarner8 @jfrost-mo For variables on i.e. pressure levels we also want to define pressure level dependent min and max values for colorbars. For this we could use the templated $PLEVEL variable set in the rose-suite.conf file and have a structure in the colorbar.json file along these lines:
"surface_upward_latent_heat_flux": {
"max": 250.0,
"min": -250.0,
"pressure_levels": {
"1000hPa": {
"max": 200.0,
"min": -200.0
}
}
"pressure_levels": {
"850hPa": {
"max": 300.0,
"min": -300.0
},

We then need to modify _colorbar_definiton.py to link the $PLEVEL from the rose-suite.conf GUI or cylc/yaml file with the predefined pressure level min max in the colorfile. This is the bit I um unclear about still how to achieve that. For model levels we would want a bit more intelligent interpolation I assume?

_plot_and_save_spatial_plot for instances populates using the variable name. Can we use the pressure level information if existent from this call to select the colour min max based on the variable information.

Specify the color bar

cmap, levels, norm = _colorbar_map_levels(cube.name())

[jwarner8]

I created a separate issue for this in #1139.
If we have pressure dependent min/max values for each variable, the json is going to get large very quickly, and will end up being modified anyway (850 geopotential will look very different over UK vs tropical Africa).
More generally I've found cube.name unreliable, as LFRIc and UM might differ in their .name, so we might need to revisit how it uses the json look up table. We could set an attribute in the cube after filtering to be equal to the $VARNAME, as the stash dict LFRic namelist is the definitive source of variable names

[Sylviabohnenstengel]

Thanks, I did not see #1139. Agree with the concern around a huge json file. To make this more intelligent we would need to be able to set the vmin and vmax level dependent based on all runs that we load. I assume this would require cylc broadcasting and is probably over the top. But I do not see a better way at the moment...

[jwarner8]

I agree, the only thing I could think of was to know all the model datas min/max for all case studies for that variable/plev, but that requires processing all the data before plotting which would be a very different design.

I guess most people will plot spatial plots on some standard pressure levels 850, 500, 200, so we could just cater for these in the json.

[jfrost-mo]

OFFICIAL The current colourbar method has been good to get us started, but I don’t think it is a long term solution. The way we are doing colour bars currently requires too much manual configuration and doesn’t have any sensible defaults for unknown variables. We should definitely have a think about how to do it better, though we always have the constraint of not being able to look at the data before deciding. (Or could we somehow…?) OFFICIAL

[jwarner8]

I wonder if there is a solution where the workflow reads the data needed for all case studies, filters, does appropriate processing and saves output as per recipes, and does all plotting tasks at the end. This would give us two advantages; ability to determine suitable colorbar ranges for plots at the end, and also reducing duplication of processing when doing case study aggregation. Implementation however, not so clear...

[jfrost-mo]

With the way CSET is currently designed any synchronisation between tasks would be very expensive, as it would have to go via the filesystem. If we were to change CSET to instead run on a single node (or at least a node per variable) then we would able to much more efficiently communicate between the jobs, and something like you suggest would become possible.