Huge Prediction

[mmann1123]

@jgrss I have about 30 time series features covering a large geographic area (northern Tanzania). Each time series feature is currently stored in non-uniform grids - some features have a large number of tiles, while others only have a few (see below).

I have trained a ML model based on sample data and now want to predict back to the rasters - but given the large size and non-uniform nature of the grids, I am having trouble deciding how best to handle this.

Previously, i had created a "prediction stack" of all the features, so in this case it would 32 bands and then ran:

# predict to stack
def user_func(w, block, model):
    pred_shape = list(block.shape)
    X = block.reshape(pred_shape[0], -1).T
    pred_shape[0] = 1
    y_hat = model.predict(X)
    X_reshaped = y_hat.T.reshape(pred_shape)
    return w, X_reshaped

gw.apply(
    "outputs/pred_stack.tif",
    f"outputs/final_model_rf{len(select_images)}.tif",
    user_func,
    args=(pipeline_performance,),
    n_jobs=16,
    count=1,
)

So I guess its a three part question: 1) can i create VRTs to help read in the non-uniform image grids and read the VRTs into geowombat? , 2) assuming I have read in the data, is my best bet to create a series of uniform "prediction" tiles, 3) is apply likely the fastest low-memory option for prediction?

I know that's a lot, but any thoughts would be helpful.

List of time series features (N=32):

'B11_abs_energy',
 'B11_maximum',
 'B11_mean',
 'B11_mean_change',
 'B11_mean_second_derivative_central',
 'B11_median',
 'B11_minimum',
 'B11_quantile_q_0.05',
 'B11_quantile_q_0.95',
 'B11_standard_deviation',
 'B12_abs_energy',
 'B12_absolute_sum_of_changes',
 'B12_maximum',
 'B12_mean_change',
 'B12_mean_second_derivative_central',
 'B12_median',
 'B12_minimum',
 'B12_quantile_q_0.05',
 'B12_quantile_q_0.95',
 'B12_standard_deviation',
 'B2_minimum',
 'B6_abs_energy',
 'B6_quantile_q_0.05',
 'EVI_doy_of_minimum_dates',
 'EVI_mean_change',
 'EVI_median',
 'EVI_minimum',
 'EVI_quantile_q_0.05',
 'EVI_standard_deviation',
 'hue_median',
 'hue_minimum',
 'hue_quantile_q_0.05'}

Where each image is broken into a non-uniform grid for instance:

{'EVI_minimum': 
['../features/EVI/EVI_minimum_0000046592-0000046592.tif',
 '../features/EVI/EVI_minimum_0000046592-0000000000.tif',
 '../features/EVI/EVI_minimum_0000000000-0000000000-part2.tif',
 '../features/EVI/EVI_minimum_0000046592-0000093184.tif',
 '../features/EVI/EVI_minimum_0000000000-0000046592-part1.tif',
 '../features/EVI/EVI_minimum_0000000000-0000046592-part2.tif',
 '../features/EVI/EVI_minimum_0000000000-0000000000-part1.tif',
 '../features/EVI/EVI_minimum_0000000000-0000093184.tif']}

and

{'hue_quantile_q_0.05': 
['../features/hue/hue_quantile_q_0.05_0000000000-0000000000.tif',
  '../features/hue/hue_quantile_q_0.05_0000000000-0000046592.tif']}