Handle large ingests

[jonseddon]

There's an ERA5 dataset with over 500,000 files in it. This can't be ingested in the length of a standard batch system job. There are many ways to speed this:

• use parallelism to read and checksum multiple files at once
• allow ingestions to resume
• allow ingestions to be split into smaller chunks
• do all variables need to go into the same dataset?