Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make loading IRISSpectrograph more memory efficient #14

Open
nabobalis opened this issue Mar 8, 2022 · 0 comments
Open

Make loading IRISSpectrograph more memory efficient #14

nabobalis opened this issue Mar 8, 2022 · 0 comments

Comments

@nabobalis
Copy link
Collaborator

I'm finding that IRISSpectrograph, or read_iris_spectrograph_level2_fits() takes a very long time to load and has a very high memory usage. Loading a moderate size IRIS raster in my laptop, which only has 8 GB of RAM, leads to lots of swapping and the machine nearly crawls to a halt.

Testing in different machines, I find the RAM that read_iris_spectrograph_level2_fits() uses is between 3-4 times the total size of the IRIS level 2 FITS file. A 606 MB file took about 1.5 GB of RAM, while a 2.6 GB file took 10GB. This is very inefficient, not to mention slow (even if the machine has enough RAM).

Looking in the code, I suspect these lines are the most problematic:
https://github.com/sunpy/irispy/blob/bbd194205c1dd815230925ad676a286f572bb28c/irispy/spectrograph.py#L562-L564

There should be a way to load it with memmap, and ideally it should be the default.

In addition, the uncertainty will perform many operations on all data points from the file:

https://github.com/sunpy/irispy/blob/bbd194205c1dd815230925ad676a286f572bb28c/irispy/spectrograph.py#L596-L598

This is a huge burden for something of marginal value.

The loading of IRISSpectrograph should be as lightweight as possible. People may use it just to look at what's inside.

Original issue: sunpy/sunraster#82

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant