You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm finding that IRISSpectrograph, or read_iris_spectrograph_level2_fits() takes a very long time to load and has a very high memory usage. Loading a moderate size IRIS raster in my laptop, which only has 8 GB of RAM, leads to lots of swapping and the machine nearly crawls to a halt.
Testing in different machines, I find the RAM that read_iris_spectrograph_level2_fits() uses is between 3-4 times the total size of the IRIS level 2 FITS file. A 606 MB file took about 1.5 GB of RAM, while a 2.6 GB file took 10GB. This is very inefficient, not to mention slow (even if the machine has enough RAM).
I'm finding that
IRISSpectrograph
, orread_iris_spectrograph_level2_fits()
takes a very long time to load and has a very high memory usage. Loading a moderate size IRIS raster in my laptop, which only has 8 GB of RAM, leads to lots of swapping and the machine nearly crawls to a halt.Testing in different machines, I find the RAM that
read_iris_spectrograph_level2_fits()
uses is between 3-4 times the total size of the IRIS level 2 FITS file. A 606 MB file took about 1.5 GB of RAM, while a 2.6 GB file took 10GB. This is very inefficient, not to mention slow (even if the machine has enough RAM).Looking in the code, I suspect these lines are the most problematic:
https://github.com/sunpy/irispy/blob/bbd194205c1dd815230925ad676a286f572bb28c/irispy/spectrograph.py#L562-L564
There should be a way to load it with
memmap
, and ideally it should be the default.In addition, the uncertainty will perform many operations on all data points from the file:
https://github.com/sunpy/irispy/blob/bbd194205c1dd815230925ad676a286f572bb28c/irispy/spectrograph.py#L596-L598
This is a huge burden for something of marginal value.
The loading of
IRISSpectrograph
should be as lightweight as possible. People may use it just to look at what's inside.Original issue: sunpy/sunraster#82
The text was updated successfully, but these errors were encountered: