MPI tests using > 15 GB memory

[quantumsteve]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. git commit hash being built
2. cmake command
3. full program/test invocation command
4. additional steps

asgard-unit-mpi-gxx and asgard-unit-mpi-gxx-scalapack currently run out of memory on the CI machines. Iincreased the memory size from 15GB to 144GB as a (temporarily?) workaround, suspect there is something over allocating memory or a test that should be scaled down or moved to a different label that runs less often.

Expected behavior
A clear and concise description of what you expected to happen.

MPI tests pass on a container with 15GB RAM.

System:

• system name [e.g. fusiont5, summit]
• modules loaded [e.g. output of module list]
• other systems where this is reproducible [e.g. "my laptop", "none"]

Additional context
Add any other context about the problem here.

Reproduced locally with docker run -m 15000m -it cpu /bin/bash

[quantumsteve]

Found while troubleshooting #624.

[quantumsteve]

failing tests are both continuity_6

asgard/src/distribution_tests.cpp

Line 647 in 028b142

SECTION("even but not square, large")

asgard/src/time_advance_tests.cpp

Line 678 in 028b142

SECTION("continuity6, explicit/non-uniform level, degree 4, sparse grid")

[mkstoyanov]

Did a few tests, I think that 32GB will be enough, but it is still too much for CI.

For some reason, when we create a 6D problem, we are unreasonable amount of memory for something. I wonder if this is an issue with the hash-map taking too much space. If I'm right, this will be an issue across the board, it's just that the problem needs to run 4 copies (since the test uses 4 mpi ranks) and all of our workstations have lots more than that.

[mkstoyanov]

Tested again on develop, it goes nowhere near 16GB. I suspect the problem was resolved in #743