Some counters with more complex programming requirements don't work with libpfc

[travisdowns]

Some counters have more complex programming requirements, e.g., setting two different MSRs.

Unfortunately, this includes pretty interesting counters such as the L1I_MISS counters. See here for a concrete example and a nasty workaround.

It would be awesome if libpfc supported these (maybe I'll do it someday if no one gets to it).

[obilaniu]

I did some of the more tedious work related to supporting a new MSR, and exposed it for reading, though not writing (because /sys/module/pfc/msr is read-only for now).

I suppose what's left is that if a counter gets programmed with an L1I_MISS event code, then there should be a conditional pfcWRMSR() invocation within the kernel module.

I still need to think of a way to systematize all of these MSR accesses. More broadly, I feel that the kernel module, kernel-userland ABI and libpfc.h API are all overdue for a refactoring. But it's hard to take into account all hardware generations' quirks, events lists, restrictions on which counter may count what, ... and to then encode this within both the kernel and userland.

Then there is the issue of how to deal with counter virtualization. Making the counter values follow the thread would remove the limitation of pinning the thread to just one core, but would be a massive undertaking to support and I've no idea how to even begin.

[travisdowns]

Indeed, I don't suggest a general solution because I think "it's complicated".

Personally I wouldn't embark on the path of trying to virtualize counters: this already happens with perf_events so all of the code is already there and in almost every kernel for the last X years (X is perhaps 5ish?).

Presumably perf also has a way to program these more complex counters: indeed the events work fine when you use perf (and in fact my current workaround is to use perf on the comment line to program the extra MSRs, and then use libpfc since the MSRs keep their values). So we are again running up against the fact that perf_events already handles a lot of what you might consider potential enhancements to libpfc, so presumably you want to keep filling the niche that perf_events doesn't handle.

[travisdowns]

I suppose what's left is that if a counter gets programmed with an L1I_MISS event code, then there should be a conditional pfcWRMSR() invocation within the kernel module.

Yes, I think so - and the API also needs a way to accept this additional info (since I don't think it all gets packed into the one raw code).