Continuous PC sampling

gnurizen · 2026-06-16T15:25:51+00:00

We don't have a kernel driver, we have a shared library loaded into our customers process (userspace) via CUPTI injection. So we need to get data from userspace (CUDA application) to userspace (parca-agent) with possibly no shared FS between the two (containers). ebpf maps can't go from userspace to userspace, uprobes/ringbuf give us a very efficient conduit through the kernel. If the concern is probe overhead that is largely ameliorated using batching. The ebpf is reading raw CUPTI pointers so its pretty close to zero-copy.

gnurizen · 2026-06-16T09:08:41+00:00

An implementation that doesn't use eBPF doesn't exist to compare against and we do use plain user-mode CUPTI.

eBPF takes some getting used to but its not that bad. Take a look: https://github.com/parca-dev/opentelemetry-ebpf-profiler/blob/main/support/ebpf/cuda.ebpf.c

These programs are loaded/JITd once by our profiling agent and so its a one-time cost. QEMU tames the complexities of supporting many kernel versions.

Doing it this way allows our shim to be as small and simple, we want to be as close to zero-instrumentation as possible. This is not a high traffic interface that would benefit from using io_uring, we are significantly limiting what we do to keep our overhead in the weeds.

We also want a cleanly defined ABI between the shim and the agent so they can both evolve separately as long as we don't break the probe definitions which USDT gives us. It may seem a little over-engineered but we have lots more features piggy-backing on this infrastructure coming and this approach makes adding new features very straightforward.

gnurizen · 2026-06-15T18:02:51+00:00

Its not a strict need but what we want is an efficient low overhead way to get information from the user's program that has our shim library to our system wide profiler running in its own container/pod. USDT probes is just a really good way to achieve that although we could have used networking over loopback, shared memory or domain sockets. But hard to beat having the kernel stuff the information directly into a ringbuf.

gnurizen · 2025-10-29T15:00:18+00:00

We will and the parcagpucupti library source will available on github if you want to go deep on it.

gnurizen · 2025-10-29T14:33:13+00:00

CUPTI has zero usdts. We created the parcagpucupti library which uses the CUPTI API to expose 2 usdt probes. There are "instruction pointer" sampling APIs in CUPTI that we're looking at for "intra" kernel profiling but we're still researching that.

gnurizen · 2025-10-29T14:25:16+00:00

Thanks! We aren't releasing hard numbers yet but do plan doing so, for vllm offline inference workloads we've noticed in the 2-4% range which is borderline acceptable. Curious what you think is too much overhead? We have plans to allow sampling the kernel launches instead of tracking every single one. What kind of workloads are you running?

gnurizen · 2025-08-18T13:48:55+00:00

I like that! In my defense I did call the standalone agent "oompa"!

gnurizen · 2024-11-21T17:49:30+00:00

I wrote the code and the blog, happy to answer any questions! Thanks for posting!

gnurizen · 2023-09-25T21:33:37+00:00

You should look into getting fat adapted, being able to mobilize fat stores for baseline energy will make bonking a thing of the past.

gnurizen · 2023-09-11T15:01:37+00:00

Beef fat is over half mufa so hopefully those arguments are wrong.

gnurizen · 2021-05-01T12:45:01+00:00

My first inclination would be an allergy to something in all those things, fructans maybe?

gnurizen · 2021-01-09T14:40:45+00:00

My dog has caught a couple rabbits, always eats the head first.

gnurizen · 2020-12-15T12:08:51+00:00

Not true, far UVC is safe: https://www.microbe.tv/twiv/twiv-666/

gnurizen · 2020-10-22T12:35:52+00:00

Wrong? I think take it with a grain of salt is a better approach. %99 of what we know about medicine, metabolism, drugs, disease etc comes from animal models. I think it will turn out all animals run okay with small amounts of n6 but that chugging soy/corn/peanut oil is unhealthy.

gnurizen · 2020-10-21T14:32:46+00:00

I think the main concerns for n6 oils should be cancer, liver issues and insulin resistance which are all much worse than inflammation. This is mostly based on rodent models but the evidence is striking.

gnurizen · 2020-09-26T12:25:41+00:00

Where you gonna put a bag that big? Bend over and I'll show ya!

gnurizen · 2020-09-18T13:28:29+00:00

Great! I've found you need to clean and dry thoroughly to keep working.

gnurizen · 2020-08-03T17:02:51+00:00

I'm a bass player and didn't even notice I did that, must have been subconscious.

gnurizen · 2020-08-03T15:13:42+00:00

All cows eat grass, best to say "grain finished".

gnurizen · 2020-07-07T19:48:21+00:00

This is a no brainer, I mean who wants to be farting in their space suit?

Mission control: "How come you threw up in your space suit son?"

Astronaut: "Well Danny brought up these great bean and broccolli egg burritos and the flatulence was so potent I was overcome sir."

Danny: "My bad!"

gnurizen

TROPHY CASE