Add option for prefetch factor of data loader to config

[shengshiqi-google]

Is your feature request related to a problem? Please describe.

As you know, PyTorch data loader has option for prefetch factor: https://pytorch.org/docs/stable/data.html
This is in addition to num_workers. The default is 2.

Describe the solution you'd like

It would be nice if we can specify this through the YAML. Perhaps this can be helpful if the storage latency is high and we'd want more prefetch to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.
We tried increasing num_workers, which can help and reduce latency.
However, increasing num_workers also increase CPU Memory usage.

Additional context

NeMo 24.07

[yaoyu-33]

which model you are training? we are rotating out the yaml/ nemo1 way of configuration.
Check our 2.0 api: https://docs.nvidia.com/nemo-framework/user-guide/latest/nemo-2.0/index.html .

By default your prefetch with PTL data loader would be num_workers * prefetch_factor(2), do you see it's a bottleneck?