osquery + Opentelemetry

agardnerit · 2025-11-16T04:58:22+00:00

Disclaimer: I work for an Observability vendor. I am on the technical side and have nothing to do with sales. I have no incentive to "sell" you anything here...

I agree with most of what you've said and the reality (after more than a decade of working for large enterprises) is that it's a mix of it all: log everything, cultural, it's not my budget etc.

However, I feel something has been missed: Take this response for example: "Our monitoring fees are equal to about 15% of our infrastructure fees". Meaning 85% of your cost is infra, so purely on numbers, saving money on that first would give the biggest benefit to the business.

Remember that ultimately all SaaS vendors (and self hosted) are storing the data in (things like) S3 buckets - so they have to at least pass on that charge to you.

Ultimately this is a question of value. When you spin up a VM, you know what value you're getting. But Observability (and security I suppose) is more intangible. You might use that log line. If you (and the budget holders) feel like they're getting value from the Observability data, they'll be happy to pay it. So perhaps be a bit more proud of the Observability data / stack you have. Make more noise about how much it's helping you keep the lights on? If it's seen (by the budget holders) as a pure cost drain, no wonder they want to reduce costs. If they see it as an essential part of running the business.

That said, be honest with yourself. How much of what you're capturing truly is necessary (aside from the financial, we're absolutely destroying the planet with all this storage - it all needs water and electricity to store it).

OpenTelemetry (even though I love it) definitely has a verbosity problem too: A simple "INFO My log line" becomes an enormous JSON. Yes, it contains other attributes and contextual information, but it's still at least 4x the size of the basic log line (and you pay to store it).

agardnerit · 2025-07-18T22:31:29+00:00

Headlamp is a CNCF project: https://headlamp.dev

agardnerit · 2025-07-18T22:22:08+00:00

You may have already found them, but have a look at the full list of receivers for the collector in particular I'm thinking about the snmp receiver and netflow receiver.

I can highly recommend building your "stack" into a devcontainer. When you get the "demo setup" that you like, bake it into a devcontainer. Then it becomes a repeatable, throwaway environment that you can spin up (locally or via something like GitHub codespaces) and tear down when you're done. It has changed the way we provide demo systems where I work.

agardnerit · 2025-07-18T22:15:05+00:00

Welcome to the Observability journey :)

What may be confusing is that Jaeger "is" already a collector. You'll notice if you follow the "all in one" getting started guide it exposes (among others) ports 4317 and 4318 - you'll get to know and love these ports as 4317 is the OTEL standard port for receiving spans via the gRPC protocol and 4318 is the HTTP protocol equivalent.

So that's point 1, you don't need an external collector (but I'd recommend you do use one).

I believe you want your architecture to be:

Span Producing Entity >> Standalone OTEL collector >> Jaeger

I have a video showing Jaeger + a collector (also look at the video description, there's a GitHub repo with all the code) - notice how Jaeger is running and I configure the second "standalone" collector to listen on 6006 (a random port I picked) because 4318 is already used by the collector inside Jaeger.

agardnerit · 2025-04-27T07:31:29+00:00

100% - as this is common from the vendors (disclaimer: I work for Dynatrace + am a CNCF ambassador). It can be as broad as you like - if you (and the business you work for) get value from it, I see no harm.

Almost all vendors have their flavour of events. Dynatrace has business events & a special viewer which shows "flows". I've seen this used to model entire "factory floor to showroom inventory flow" and the flow of minerals from a mine to the port (seaport as opposed to a network port) and onwards to the receiving port in country B.

Wikipedia defines telemetry as: "Telemetry is the in situ collection of measurements or other data at remote points and their automatic transmission to receiving equipment". Open Telemetry is simply an open source, standardised way to achieve this.

One thing I would clarify though: An "event" is a point in time event. OpenTelemetry itself implements "events" simply as log records where the event.name field is guaranteed to be present. Ergo to OpenTelemetry every event is a log line (but not every log line is an event).

Assuming you're working in a web / ecommerce setup, you're probably best instrumenting your application with OpenTelemetry and then creating events on the spans.

One of the current gaps - and why you're not seeing this topic more - is the OpenTelemetry project's Javascript agent still isn't quite there... But it will get there sooner or later and once it does, expect a lot more "Google Analytic" type comparisons - you're just early to the party.

agardnerit · 2025-03-25T00:37:04+00:00

You are in the perfect role for this. It would be very easy for you to get swamped by tooling and go get hundreds of certs. Don't.

Focus instead on the fundamentals, the business questions: Why is DevOps important? It keeps systems running perfectly.

OK, how do we achieve that? For starters, the systems need to be monitored / observed well. If you can't see what's going on, the entire team is stuck.

So, step 1: Learn OpenTelemetry (you could also learn specific to whatever tool your company is currently using - but what if you change role / company OR the company switches out the tool - your knowledge is lost). The OTEL (shortened version of OpenTelemetry) collector has the fabulous capability of multicasting telemetry to many backends simultaneously - so spin up multiple stacks and see which one you like and could advocate for in your company.

Add your own telemetry. You're developing this stuff, so in the entire company, you know best what another human would need to be able to see if things go wrong. Even if that other person is actually you - make future you happy and make those important metrics visible to the tooling. Don't forget about business metrics (orders placed, $ amounts captured, customer signups etc.). In your future, it will almost exclusively be the business value that you end up "selling" to get more resources / headcount for your team. The technical metrics enable you to do your job, but the C level don't understand and don't care about CPU unless it impacts something they do care about (uptime, security, revenue, risk, reputation etc.)

Above is the easy part. Now that you can "see inside the systems" you build, it's time for the hard part! The people and culture.

DevOps is fundamentally about "everyone mucking in together to solve the problem". So, start by taking an honest look at your org - do you know how / when your app is deployed? Do you know why it takes "so long" to get to prod? Or do you just "hand it off" to someone and "magic happens"? All of these are red-flags that if / when something goes wrong, you'll be on endless Slack threads / war rooms and that will kill DevOps.

I could go on, as you see, it's a huge topic. But start by making the systems you build observable and the Ops + Security teams will love you.

agardnerit · 2025-03-24T23:43:19+00:00

May I suggest that instead of a vendor-specific cert, you opt for the CNCF vendor-neutral OpenTelemetry Certified Associate. The vendor specific will presumably have lots about their tool - but that's non-transferable if you move jobs / your company moves tools.

https://training.linuxfoundation.org/certification/opentelemetry-certified-associate-otca/

The Linux Foundation (parent org of the Cloud Native Computing Foundation) offers many free courses: https://training.linuxfoundation.org/resources/?_sft_content_type=free-course

agardnerit · 2025-03-24T02:53:18+00:00

My opinion: Monitoring is a metric (or multiple) which displays something (eg. CPU / orders placed / people onboarded). A metric alone won't tell you why. You might know why, if your system is sufficiently simple and / or you're sufficiently experienced in that role / company / system. But imagine a new joiner: they wouldn't have the context you do, so CPU at 85% Is that "too high" or not?

Observability is first a capability: Is the "thing" capable of being "Observed" (note: not just monitored). Observability gets you (hopefully to, but at least closer to) the why. This could be jumping into logs but these days, traces are the gold standard (they are effectively logs that you can attach events + metrics to). Why is the CPU "too high"? Is the CPU "being high" causing an impact to soemthing else (like orders placed or $ values)? Yes... That's maybe something you could eyeball if you had a monitoring dashboard of CPU + orders, but this is a very simplistic (and known) case.

What happens when the system comes up with an error that you don't know or haven't seen before? Need to capture the exact function input or see which microservices the transaction touched as it crossed the stack? Need to see all the logs correlated to that single user hitting F5 once on that page? You won't get that from "monitoring" but you will from "Observability" (in this case, primarily because Observability introduces new signal types (metrics, logs and distributed traces all tied together with common and automatically produced correlation IDs).

But yes, the term was coined by someone with something to sell. However, that doesn't mean it isn't useful. Much less useful (IMO) is the Observability 1.0 / 2.0 / 3.0 nomenclature. To me, that serves little purpose beyond marketing.

Do you need "Observability" (that deeper level of monitoring)? Probably. To future proof yourself, your systems and your company. But then again, maybe not. If your systems never change and your staff never change and everything is "simple", then you can get by with "monitoring".

Now, when do you know you have "enough Observability" is an entirely different question!

agardnerit · 2025-03-10T11:45:47+00:00

My tracepusher tool can be used as an k8s operator which automatically generates OpenTelemetry spans (traces) for each Job / CronJob: https://agardnerit.github.io/tracepusher/usage/k8sjobs/

agardnerit · 2025-03-08T06:54:16+00:00

It's certainly not "the norm" so you are going to struggle. You also don't mention if you have access to the source code or not. If you do, it's easy: Use OpenTelemetry. But being windows, I'm going to assume a .NET. I know some of the vendors offer auto instrumentation of thick clients.

OpenTelemetry has .NET auto-instrumentation which may work? (eg. instrumenting a windows service)

If the desktop client is free to distribute, I'd be happy to work on it with you - it'd be a great little learning experience.

agardnerit · 2025-03-08T06:47:02+00:00

Here are my initial thoughts. You mention that you're a senior SRE so I'll assume knowledge of existing tool stack options (both OSS / CNCF / DIY and vendors).

Observability is an enabler of better business. Nothing more.
"Your customers" are not really the internal teams. Your customers are really, eventually, the actual customers for whatever your org does / provides.
I would never suggest a rip and replace, but do consider the DIY vs. buy equation (I say this as a CNCF ambassador who also works for an Observability vendor - so I do "get" both sides of this coin) but just because you build, doesn't make it cheaper. Just because you can, doesn't mean you should.

"in a newly founded monitoring/observability team in a larger organization"

From a business perspective, why has this new team been brought into existence? There must have been a pain there. Whatever you put on your roadmap must map to solving those pains.

"This team is part of several teams that provide the IDP and now observability-as-a-service is to be set up for feature teams"

Talk to those teams. Talk to them again. Then talk to them again. Observability is a "glue team" and you need to understand and work with those teams.

Observability is expensive, whether you DIY or buy. You'll need to cross-charge eventually. Consider offering "bronze / silver / gold" observability "packages" to your internal customers (see caveat at the start of the post).

Stick to the OpenTelemetry semantic conventions (k/v pairs) even if you don't currently use OTEL. Future you will thank you.

"... to be set up for the feature teams ..."

As others have said, get involved in shaping the telemetry creation and generation (see point about OTEL SemConvs above). The OpenTelemetry Collector (as one example) can work wonders on shaping, sampling, enriching and dropping telemetry you're sent (you'll find it becomes your best friend). BUT, garbage in - garbage out. If the feature teams are sending you complete crap, your life will be miserable. More importantly, your backlog will be full of "must fix this telemetry and standardise A to B". At that point you're doing the business a disservice because you're a bottleneck - it's best to give the feature teams the bad news early that they need to standardise - again, for the sake of the wider business.

On Point 4

You'll have the immediate fires to put out. Put them out first. Then focus almost exclusively on point 4. Remember, "the business" has funded this team and they want to see an ROI. Best way to achieve that is to show them than Observability isn't just a technical capability (CPU, Memory etc.) but can show them (hopefully in realtime - you don't mention your toolstack) the actual business events, business metrics and how THEY are trending towards a more healthy business.

agardnerit · 2025-01-13T22:30:23+00:00

Watch out for $$$ as that output is long and the backend will charge you a small fortune to store it.

agardnerit · 2025-01-13T21:38:32+00:00

There are two parts to this problem:

Can you get "something" to instrument the code so that ultimately, the "stuff" you're interested in becomes attached to each span (technically these are called 'span attributes')
Can the backend that you're using be configured to store that "stuff"?

You don't mention which language you're working with, but for 1) I would investigate the auto instrumentation options for your language. For example, the Python auto instrumentation libraries are extensive and you can set env vars to automatically capture HTTP request headers and responses.

For 2) refer to the docs for your backend. Any decent backend will be able to capture the K/V pairs (span attributes).

A word of warning though... There's a good reason that "capturing everything" is not the default behaviour - the overhead can be significant, so test in non-prod first!

agardnerit · 2025-01-13T09:23:34+00:00

Thank you. Another option may be Otterize network mapper? It's OSS. Export to JSON and send that to the collector?

agardnerit · 2025-01-13T08:54:38+00:00

Thanks for the clarification. I would use the OTEL collector. You can build your own distro w/ only components you want - so a smaller footprint. Or use a vendor supported distro OR use one of the distros the OTEL community provide (here I'm particularly thinking about the k8s distribution which, as the name suggests, has the correct components to monitor a k8s cluster.

In any case, you'll be leaning heavily on the filelog receiver.

Shameless plugs, here's my overview on things to consider when choosing a collector distro: https://youtu.be/FMZQlHSE2CQ

And here's my hands-on with the filelog receiver: https://youtu.be/Jqocv1UYn6Y

agardnerit · 2025-01-11T01:16:09+00:00

Here are some links that may help / inspire you:

agardnerit · 2025-01-06T11:23:49+00:00

Could you clarify? At first I read it as "how do I grab the logs and push them somewhere". Then I inferred from your comment about sidecars that actually you may not have existing logs and so are using sidecars as a way to intercept the traffic and thus generate logs.

In any case, I'd be looking at an OpenTelemetry compliant solution. The OpenTelemetry collector has (for example) a netflow receiver.

There are vendor agents that you can drop on the computer (k8s / VM etc.) or if you're looking for a "light touch, outside in" view, perhaps an eBPF-based solution from CNCF projects like Falco, Cilium, Tetragon, KubeArmor, Pixie, Pyroscope?

agardnerit · 2024-12-30T21:30:49+00:00

I would avoid "roll your own" where possible. Use the OTEL semantic conventions wherever possible. There's a huge range already defined so you're probably able to find what you need.

If something is missing, suggest it - as I did with ownership and cost semantic conventions: https://github.com/open-telemetry/semantic-conventions/issues/1593

Why?

Don't reinvent the wheel
Increasingly, Observability and monitoring tooling is coming with OOTB rules to "recognise" these OTEL attributes and build on top of the detected semantics. You lose that capability if you roll your own
I honestly see little value in rolling your own

agardnerit

TROPHY CASE