wrapper around jemalloc to track allocator usage by thread

[alexpyattaev]

a simple wrapper around jemalloc to track allocator usage by thread name in metrics.
Idea is to get a better idea why node crashes when OOM occurs (at least which threads were allocating memory).
This is for dev use only.

Problem

If/when agave starts leaking memory (or just clogging up some channel) it may be tricky to find where memory allocations are happening that cause the crash. Tracking per-pool allocations is not a replacement for valgrind, but has the advantage of fairly small overhead & integration into metrics.

Summary of Changes

Added feature-flag gated custom wrapper around jemalloc that tracks memory usage, grouped by thread pool name.

[gregcusack]

this seems like it's going to be a big help! just a few comments/questions. Thank you!

[gregcusack]

Where are you finding these thread names? I can't seem to find a few of them? EDIT: idk why it selected 4 lines. Meant to just select FetchStage

[alexpyattaev]

These are from my experiments with thread manager. Quite possible that some of those I got wrong and/or missed some. Listing all threads in agave is pretty much impossible in its current form. But this is no big deal as long as we get the main pools right. Arguably, we could cut this list down to top 10 and it would be equally useful.

[gregcusack]

ya agree that we don't need to capture every single thread. But I do want to make sure the ones that we have hard coded here are actual thread names. when I search in the codebase for threads names like FetchStage or solClusterInfo I don't find any

[alexpyattaev]

Ok I've double-checked, now all names should be legit.

[gregcusack]

ok so maybe i don't fully understand how thread names work in rust. I know we define thread names in the code base like: solSigVerTpuVot defined here:

agave/core/src/tpu.rs

Lines 227 to 235 in 85e8f86

	let vote_sigverify_stage = {
	let verifier = TransactionSigVerifier::new_reject_non_vote(tpu_vote_sender);
	SigVerifyStage::new(
	vote_packet_receiver,
	verifier,
	"solSigVerTpuVot",
	"tpu-vote-verifier",
	)
	};

.

But in the PR, you have thread names like: solVoteSigVerTpu. But I can't find any thread names in the code base that match solVoteSigVerTpu. So, my question is, if solVoteSigVerTpu exists as a thread at runtime, how is this thread named?

[alexpyattaev]

No magic transform, just got the grep script wrong. 2 slipped through.