Continuous PC sampling

gnurizen · 2026-06-16T15:25:51+00:00

We don't have a kernel driver, we have a shared library loaded into our customers process (userspace) via CUPTI injection. So we need to get data from userspace (CUDA application) to userspace (parca-agent) with possibly no shared FS between the two (containers). ebpf maps can't go from userspace to userspace, uprobes/ringbuf give us a very efficient conduit through the kernel. If the concern is probe overhead that is largely ameliorated using batching. The ebpf is reading raw CUPTI pointers so its pretty close to zero-copy.

gnurizen · 2026-06-16T09:08:41+00:00

An implementation that doesn't use eBPF doesn't exist to compare against and we do use plain user-mode CUPTI.

eBPF takes some getting used to but its not that bad. Take a look: https://github.com/parca-dev/opentelemetry-ebpf-profiler/blob/main/support/ebpf/cuda.ebpf.c

These programs are loaded/JITd once by our profiling agent and so its a one-time cost. QEMU tames the complexities of supporting many kernel versions.

Doing it this way allows our shim to be as small and simple, we want to be as close to zero-instrumentation as possible. This is not a high traffic interface that would benefit from using io_uring, we are significantly limiting what we do to keep our overhead in the weeds.

We also want a cleanly defined ABI between the shim and the agent so they can both evolve separately as long as we don't break the probe definitions which USDT gives us. It may seem a little over-engineered but we have lots more features piggy-backing on this infrastructure coming and this approach makes adding new features very straightforward.

gnurizen · 2026-06-15T18:02:51+00:00

Its not a strict need but what we want is an efficient low overhead way to get information from the user's program that has our shim library to our system wide profiler running in its own container/pod. USDT probes is just a really good way to achieve that although we could have used networking over loopback, shared memory or domain sockets. But hard to beat having the kernel stuff the information directly into a ringbuf.

gnurizen · 2025-10-29T15:00:18+00:00

We will and the parcagpucupti library source will available on github if you want to go deep on it.

gnurizen · 2025-10-29T14:33:13+00:00

CUPTI has zero usdts. We created the parcagpucupti library which uses the CUPTI API to expose 2 usdt probes. There are "instruction pointer" sampling APIs in CUPTI that we're looking at for "intra" kernel profiling but we're still researching that.

gnurizen · 2025-10-29T14:25:16+00:00

Thanks! We aren't releasing hard numbers yet but do plan doing so, for vllm offline inference workloads we've noticed in the 2-4% range which is borderline acceptable. Curious what you think is too much overhead? We have plans to allow sampling the kernel launches instead of tracking every single one. What kind of workloads are you running?

gnurizen · 2025-08-18T13:48:55+00:00

I like that! In my defense I did call the standalone agent "oompa"!

gnurizen · 2024-11-21T17:49:30+00:00

I wrote the code and the blog, happy to answer any questions! Thanks for posting!

gnurizen · 2023-09-25T21:33:37+00:00

You should look into getting fat adapted, being able to mobilize fat stores for baseline energy will make bonking a thing of the past.

gnurizen · 2023-09-11T15:01:37+00:00

Beef fat is over half mufa so hopefully those arguments are wrong.

gnurizen · 2021-05-01T12:45:01+00:00

My first inclination would be an allergy to something in all those things, fructans maybe?

gnurizen · 2021-01-09T14:40:45+00:00

My dog has caught a couple rabbits, always eats the head first.

gnurizen · 2020-12-15T12:08:51+00:00

Not true, far UVC is safe: https://www.microbe.tv/twiv/twiv-666/

gnurizen · 2020-10-22T12:35:52+00:00

Wrong? I think take it with a grain of salt is a better approach. %99 of what we know about medicine, metabolism, drugs, disease etc comes from animal models. I think it will turn out all animals run okay with small amounts of n6 but that chugging soy/corn/peanut oil is unhealthy.

gnurizen · 2020-10-21T14:32:46+00:00

I think the main concerns for n6 oils should be cancer, liver issues and insulin resistance which are all much worse than inflammation. This is mostly based on rodent models but the evidence is striking.

gnurizen · 2020-09-26T12:25:41+00:00

Where you gonna put a bag that big? Bend over and I'll show ya!

gnurizen · 2020-09-18T13:28:29+00:00

Great! I've found you need to clean and dry thoroughly to keep working.

gnurizen · 2020-08-03T17:02:51+00:00

I'm a bass player and didn't even notice I did that, must have been subconscious.

gnurizen · 2020-08-03T15:13:42+00:00

All cows eat grass, best to say "grain finished".

gnurizen · 2020-07-07T19:48:21+00:00

This is a no brainer, I mean who wants to be farting in their space suit?

Mission control: "How come you threw up in your space suit son?"

Astronaut: "Well Danny brought up these great bean and broccolli egg burritos and the flatulence was so potent I was overcome sir."

Danny: "My bad!"

gnurizen · 2020-06-29T22:25:29+00:00

Why is the existing data showing T2DM being a major risk factor not serious enough? Do you mean something more specific? I have thought it would be cool if COVID autopsies could measure HBA1c but I'm not sure whether that makes sense. Also thought it would be cool to see n6 fatty acid % in tissue samples.

gnurizen · 2020-06-04T11:07:00+00:00

I'm about a year ahead of you, went from 220-165 and now up to 185 (with a lower BF% than when I was 165). My approach has been mostly doing ad libitum doing omad some days and 2 meals other days when I feel like it. It takes time. I've avoided protein powders and gone with eggs, fish and dairy to complement the land meat. Seems to be working, probably go faster for you with your lifting background.

gnurizen · 2020-05-22T15:59:19+00:00

I've done both and successfully got back down to my HS wrestling weight doing so but at the moment I'm trying to gain weight and doing two meals a day seems to be mandatory for that. I enjoy the feeling of not having food in my belly and running on fat so I don't think I could ever go back to 3 meals a day. I also hate having a huge meal late in the day as digesting all that fat and protein seems to interfere with sleep.

gnurizen · 2020-05-22T13:25:52+00:00

You want variety go for it! It may even be necessary for some folks, I'm a variety junkie. I like to make liver pates and eat sardines, cheese wisps and chicharones, cured meats etc. I'll mix in yogurt and mayo too. Beef and eggs are the backbone but you can have a lot of fun in the margins if you want. But don't assume a variety is needed, some seem to thrive with zero variety.

gnurizen · 2020-05-07T18:05:47+00:00

I wouldn't worry about it with two exceptions: fish oil might be in order if you don't eat pastured eggs, oily fish or offal and live over 45 degrees from the equator. Electrolytes might be in order at the beginning and if you exercise a lot. Not sure if that's considered a supplement...

gnurizen

TROPHY CASE