Understanding RAM usage in 2.X

[maran]

I manage multiple torrent instances, both Deluge and qBittorrent. I've recently upgraded all instances to libtorrent 2.X and I'm trying to understand the new memory usage patterns.

Right now for instance I have a Deluge process, measured using top, that runs 50.4g of RES and 50.1g of SHR memory. Now some quick googling seems to imply that I can use RES - SHR to get the 'real' memory usage.

I've also looked at smem which according to the man page does something similar to what I expected RES-SHR to do: smem reports physical memory usage, taking shared memory pages into account. Unshared memory is reported as the USS (Unique Set Size)

However this USS number was again using the same amount of GB.

  PID User     Command                         Swap      USS      PSS      RSS 
2697832 testtest  /home/testtest/venvs/de     79.1M    57.4G    57.4G    57.5G

I would love to truly understand what's happening here. Why is the memory usage in 2.0 so much higher, at least on paper. Can my OS easily free this memory if it's required by an other, more important process, on the server? Or will this cause OOM at one point?

Can I somehow return to the 1.X behavior where memory usage was always a lot lower?

[arvidn]

The vast majority of that memory is disk cache. There are two kinds of disk cache, dirty and clean. Any clean disk cache pages are likely very easy to reclaim by the OS and I would not expect any OOM kills because of it. The dirty pages are those that still need to be flushed to disk and can only be reclaimed once actually written to its backing storage. There are various options to make libtorrent hint the OS to flush dirty pages frequently.

That said, many people are experiencing issues with this, where the OS is not flushing dirty pages or evicting clean pages aggressively enough, to the detriment of more urgently needed pages.

There is no way to get the exact 1.2 behavior, but there are tweaks that might help, things like disable_os_cache or write_through cache settings. I've started working on additional improvements, specifically aimed at reducing read-back when hashing pieces. But I think I will probably end up forward-porting the 1.2 behavior anyway. It's just that I've been very busy lately.