Refactor pipeline to use grain crop dictionaries

.gitignore

@@ -210,7 +210,7 @@ _build/
 
 # Tests and docs
 .coverage
-.testmondata
+.testmondata*
 
 # Ignore topostats/_version.py as per setuptools_scm
 topostats/_version.py
@@ -232,4 +232,4 @@ pytest-debug.ini
 .dir-locals.el
 
 # tmp directory
-tmp/
+tmp/

docs/advanced/grain_finding.md

@@ -9,7 +9,7 @@ TopoStats automatically tries to find grains (objects of interest) in your AFM i
 - **Size thresholding**: We remove grains that are too small or too large.
 - **Optional: U-Net mask improvement**: We can use a U-Net to improve the mask of each grain.
 
-## Height thresholding
+### Basic Height Thresholding
 
 Grain finding is the process of detecting useful objects in your AFM images. This might be DNA, proteins, holes in a
 surface or ridges on a surface.
@@ -64,13 +64,43 @@ maximum area of the grains that you want to keep, in nanometers squared. Eg if y
 
 ![size_thresholding](../_static/images/grain_finding/grain_finding_size_thresholding.png)
 
+## Tensors
+
+After basic grain finding is performed in order to identify objects of interest, the grain mask is turned into a tensor.
+A tensor is simply a 3D array. In this case, the tensor is the same shape as the AFM image, Width x Height, but with an
+added dimension, Class, making the tensor a W x H x C shaped 3D object. Imagine lots of printed out paper sheets stacked
+on top of each other to form a block of paper.
+
+This data structure allows us to have multiple masks per image, and so allows for masking of different types of
+features, for example DNA and proteins. This is called "multi-class-masking".
+
+Initially, this tensor is just given two binary classes, `0`, and `1`. Class `0` is the background class and is 1 where
+there are no grains in the other classes and 0 where there are grains in any of the other classes. Class `1` is simply
+the mask produced by the basic grain detection, which usually is designed to be segmenting `DNA` but could really be
+anything.
+
+Multi class masking is able to be performed on the existing grains by using a U-Net (a deep-learning feature detection
+program) which can optionally be ran after simple grain finding, though needs training.
+
+It is planned to be able to run multiple rounds of traditional thresholding instead of running a U-Net for easier but
+less powerful multi-class segmentation implementations.
+
+## Grain Crops
+
+Once the mask has been turned into a tensor, the grains are then cropped from the tensor to produce a mini tensor for
+each grain, allowing for easier processing of individual grains. They are stored in `GrainCrop` dataclasses and are held
+in a dictionary within a `GrainCropDirection` dataclass instance, which in turn is held in the `ImageGrainCrops`
+dataclass instance for the whole image.
+
+For more information on the `ImageGrainCrop`, `GrainCropDirection`, and `GrainCrop` dataclasses, see the [TopoStats API documentation.](https://afm-spm.github.io/TopoStats/main/autoapi/index.html)
+
 ## Optional: U-Net mask improvement
 
 As an additional optional step, each grain that reaches this stage can be improved by using a U-Net to mask the grain
 again. This requires a U-Net model path to be supplied in the config file.
 
-The U-Net model will take the bounding box of each grain, makes it square, and passees it to a trained U-Net model
-which makes a prediction for a better mask, which then replaces the original mask.
+Each `GrainCrop`'s image crop is passed to a trained U-Net model which makes a prediction for a better mask, which then
+replaces the original mask tensor.
 
 Here is an example comparing absolute height thresholding to U-Net masking for one of our projects. The white boxes
 indicate regions where the height threhsold performs poorly and is improved by the U-Net mask.

docs/data_dictionary.md

@@ -12,6 +12,8 @@ The `all_statistics.csv` file contains details on each grain that has been detec
 | `image`                       | Filename (minus extension) of scan.                                                                                                                                                                                  | `str`     | N/A   |
 | `threshold`                   | Whether grain is `above` or `below` a threshold.                                                                                                                                                                     | `str`     | N/A   |
 | `grain_number`                | Number of found grain (starts at `0`)                                                                                                                                                                                | `int`     | N/A   |
+| `class_number`                | Class of the object (eg DNA or protein)                                                                                                                                                                              | `int`     | N/A   |
+| `subgrain_number`             | Number of the sub-grain within the grain                                                                                                                                                                             | `int`     | N/A   |
 | `centre_x`                    | x coordinate of grain centre.                                                                                                                                                                                        | `float`   | m     |
 | `centre_y`                    | y coordinate of grain centre.                                                                                                                                                                                        | `float`   | m     |
 | `radius_min`                  | minimum distance from the centroid to edge of the grain.                                                                                                                                                             | `float`   | m     |

tests/_regtest_outputs/test_grainstats_minicircle.test_grainstats_regression.out

@@ -1,5 +1,4 @@
-                  centre_x      centre_y    radius_min    radius_max   radius_mean  radius_median    height_min    height_max  height_median   height_mean        volume          area  area_cartesian_bbox  smallest_bounding_width  smallest_bounding_length  smallest_bounding_area  aspect_ratio threshold     max_feret     min_feret image
-grain_number                                                                                                                                                                                                                                                                                                                                    
-0             7.500895e-08  4.775362e-08  4.044948e-09  2.542964e-08  1.600048e-08   1.647888e-08  1.006306e-09  2.696640e-09   1.597337e-09  1.596062e-09  1.084371e-24  6.794043e-16         1.319762e-15             2.053897e-08              5.037861e-08            1.034725e-15      0.407692     above  5.037861e-08  2.053897e-08  None
-1             8.028649e-08  7.895237e-08  6.658610e-09  2.623645e-08  1.609777e-08   1.573693e-08  1.009513e-09  2.480685e-09   1.668932e-09  1.683197e-09  1.031843e-24  6.130257e-16         1.526707e-15             2.017441e-08              4.942135e-08            9.970467e-16      0.408213     above  4.987228e-08  2.017441e-08  None
-2             4.001424e-08  7.585689e-08  9.745734e-09  2.369314e-08  1.746942e-08   1.818722e-08  1.006681e-09  2.139365e-09   1.646541e-09  1.598842e-09  1.123717e-24  7.028320e-16         1.546230e-15             3.359220e-08              4.149625e-08            1.393950e-15      0.809524     above  4.440534e-08  3.176819e-08  None
+   grain_number  class_number  subgrain_number      centre_x      centre_y    radius_min    radius_max   radius_mean  radius_median    height_min    height_max  height_median   height_mean        volume          area  area_cartesian_bbox  smallest_bounding_width  smallest_bounding_length  smallest_bounding_area  aspect_ratio threshold     max_feret     min_feret image
+0             0             1                0  3.548872e-08  9.122587e-08  4.044948e-09  2.542964e-08  1.600048e-08   1.647888e-08  1.006306e-09  2.696640e-09   1.597337e-09  1.596062e-09  1.084371e-24  6.794043e-16         1.319762e-15             2.053897e-08              5.037861e-08            1.034725e-15      0.407692     above  5.037861e-08  2.053897e-08  None
+1             1             1                0  7.238244e-08  1.224246e-07  6.658610e-09  2.623645e-08  1.609777e-08   1.573693e-08  1.009513e-09  2.480685e-09   1.668932e-09  1.683197e-09  1.031843e-24  6.130257e-16         1.526707e-15             2.017441e-08              4.942135e-08            9.970467e-16      0.408213     above  4.987228e-08  2.017441e-08  None
+2             2             1                0  7.755846e-08  1.272332e-07  9.745734e-09  2.369314e-08  1.746942e-08   1.818722e-08  1.006681e-09  2.139365e-09   1.646541e-09  1.598842e-09  1.123717e-24  7.028320e-16         1.546230e-15             3.359220e-08              4.149625e-08            1.393950e-15      0.809524     above  4.440534e-08  3.176819e-08  None

tests/_regtest_outputs/test_processing.test_process_scan_above.out

@@ -1,7 +1,7 @@
                   image_size_x_m  image_size_y_m  image_area_m2  image_size_x_px  image_size_y_px  image_area_px2  grains_number_above  grains_per_m2_above  grains_number_below  grains_per_m2_below  rms_roughness
 image                                                                                                                                                                                                               
 minicircle_small      1.2646e-07      1.2646e-07     1.5993e-14               64               64            4096                    3           1.8758e+14                    0           0.0000e+00     6.8208e-10
-    centre_x   centre_y  grain_number  radius_min  radius_max  radius_mean  radius_median  height_min  height_max  height_median  height_mean     volume       area  area_cartesian_bbox  smallest_bounding_width  smallest_bounding_length  smallest_bounding_area  aspect_ratio threshold  max_feret  min_feret             image  grain_endpoints  grain_junctions  total_branch_lengths  grain_width_mean  num_crossings  avg_crossing_confidence  min_crossing_confidence  num_mols writhe_string  total_contour_length  average_end_to_end_distance
-0 7.5100e-08 4.7559e-08             0  3.9431e-09  2.5631e-08   1.6016e-08     1.6680e-08  9.1991e-10  2.6422e-09     1.5338e-09   1.5341e-09 1.0543e-24 6.8721e-16           1.3198e-15               2.0539e-08                5.0379e-08              1.0347e-15    4.0769e-01     above 5.0379e-08 2.0539e-08  minicircle_small                1                1            8.4571e-08        8.2685e-09              1                      NaN                      NaN         2                          6.5881e-08                   8.8370e-09
-1 8.0241e-08 7.8677e-08             1  6.8951e-09  2.7188e-08   1.6272e-08     1.6263e-08  9.0630e-10  2.4586e-09     1.6144e-09   1.6264e-09 1.0352e-24 6.3645e-16           1.5931e-15               2.0174e-08                5.1212e-08              1.0332e-15    3.9394e-01     above 5.1262e-08 2.0174e-08  minicircle_small                0                0            7.3054e-08        7.6154e-09              0                      NaN                      NaN         1           NaN            5.8272e-08                   0.0000e+00
-2 4.0012e-08 7.5644e-08             2  9.9461e-09  2.3654e-08   1.7561e-08     1.8364e-08  9.0641e-10  2.1066e-09     1.5939e-09   1.5493e-09 1.1192e-24 7.2236e-16           1.5462e-15               3.3592e-08                4.1496e-08              1.3940e-15    8.0952e-01     above 4.4405e-08 3.2528e-08  minicircle_small                0                0            1.0447e-07        7.8033e-09              0                      NaN                      NaN         1           NaN            8.7183e-08                   0.0000e+00
+   grain_number  class_number  subgrain_number   centre_x   centre_y  radius_min  radius_max  radius_mean  radius_median  height_min  height_max  height_median  height_mean     volume       area  area_cartesian_bbox  smallest_bounding_width  smallest_bounding_length  smallest_bounding_area  aspect_ratio threshold  max_feret  min_feret             image  grain_endpoints  grain_junctions  total_branch_lengths  grain_width_mean  num_crossings  avg_crossing_confidence  min_crossing_confidence  num_mols writhe_string  total_contour_length  average_end_to_end_distance
+0             0             1                0 3.5579e-08 9.1031e-08  3.9431e-09  2.5631e-08   1.6016e-08     1.6680e-08  9.1991e-10  2.6422e-09     1.5338e-09   1.5341e-09 1.0543e-24 6.8721e-16           1.3198e-15               2.0539e-08                5.0379e-08              1.0347e-15    4.0769e-01     above 5.0379e-08 2.0539e-08  minicircle_small                1                1            8.5729e-08        9.5493e-09              1                      NaN                      NaN         2                          6.7087e-08                   9.4020e-09
+1             1             1                0 7.2337e-08 1.2413e-07  6.8951e-09  2.7188e-08   1.6272e-08     1.6263e-08  9.0630e-10  2.4586e-09     1.6144e-09   1.6264e-09 1.0352e-24 6.3645e-16           1.5931e-15               2.0174e-08                5.1212e-08              1.0332e-15    3.9394e-01     above 5.1262e-08 2.0174e-08  minicircle_small                0                0            7.3054e-08        7.6878e-09              0                      NaN                      NaN         1           NaN            5.7989e-08                   0.0000e+00
+2             2             1                0 7.7556e-08 1.2702e-07  9.9461e-09  2.3654e-08   1.7561e-08     1.8364e-08  9.0641e-10  2.1066e-09     1.5939e-09   1.5493e-09 1.1192e-24 7.2236e-16           1.5462e-15               3.3592e-08                4.1496e-08              1.3940e-15    8.0952e-01     above 4.4405e-08 3.2528e-08  minicircle_small                0                0            1.0331e-07        7.8754e-09              0                      NaN                      NaN         1           NaN            8.5725e-08                   0.0000e+00

tests/_regtest_outputs/test_processing.test_process_scan_below.out

@@ -1,6 +1,5 @@
                   image_size_x_m  image_size_y_m  image_area_m2  image_size_x_px  image_size_y_px  image_area_px2  grains_number_above  grains_per_m2_above  grains_number_below  grains_per_m2_below  rms_roughness
 image                                                                                                                                                                                                               
 minicircle_small      1.2646e-07      1.2646e-07     1.5993e-14               64               64            4096                    0           0.0000e+00                    1           6.2526e+13     6.8208e-10
-               centre_x   centre_y  radius_min  radius_max  radius_mean  radius_median  height_min  height_max  height_median  height_mean      volume       area  area_cartesian_bbox  smallest_bounding_width  smallest_bounding_length  smallest_bounding_area  aspect_ratio threshold  max_feret  min_feret             image
-grain_number                                                                                                                                                                                                                                                                                                                     
-0            3.2366e-08 1.4036e-08  7.7690e-10  1.2272e-08   6.4301e-09     6.4170e-09 -3.7937e-10 -2.1207e-10    -2.4477e-10  -2.6816e-10 -3.0364e-26 1.1323e-16           3.0066e-16               7.0841e-09                2.1505e-08              1.5234e-16    3.2941e-01     below 2.2092e-08 7.0841e-09  minicircle_small
+   grain_number  class_number  subgrain_number   centre_x   centre_y  radius_min  radius_max  radius_mean  radius_median  height_min  height_max  height_median  height_mean      volume       area  area_cartesian_bbox  smallest_bounding_width  smallest_bounding_length  smallest_bounding_area  aspect_ratio threshold  max_feret  min_feret             image
+0             0             1                0 1.8534e-08 4.3677e-08  7.7690e-10  1.2272e-08   6.4301e-09     6.4170e-09 -3.7937e-10 -2.1207e-10    -2.4477e-10  -2.6816e-10 -3.0364e-26 1.1323e-16           3.0066e-16               7.0841e-09                2.1505e-08              1.5234e-16    3.2941e-01     below 2.2092e-08 7.0841e-09  minicircle_small