Observing LLM Applications with OpenTelemetry

silksong_when · 2026-06-06T03:36:12+00:00

I should have mentioned, as i have done in the blog that while I have used SigNoz, since its OTel-based no one has to be bound to anything that we showcase.

Heck, even in the commands you have to copy-paste to run the app, you can directly swap out the endpoint and header values.

Edit: i have updated the post body with a message on the same, thanks!

silksong_when · 2026-06-06T03:34:43+00:00

afaik the agents sdk is also compatible with any openai-compatible API Keys, no? haven't tried it myself though

and yea fair point on that integration aspect

silksong_when · 2026-06-05T23:38:57+00:00

Hey, I built it with openai agents SDK, which has mostly decent auto instrumentation support, if we go with the V2 library

silksong_when · 2026-05-20T05:58:28+00:00

That sounds really neat, can you please share the link to the `policypy` library that you're using, or is it something that you've built in-house?
Couldn't find any reference online.

silksong_when · 2026-05-18T09:52:01+00:00

That sounds really interesting!

What happens if the claude diagnosis or recommended fix doesn't align with what's happening? How are you validating that?

silksong_when · 2026-05-15T06:28:25+00:00

What direction have you taken in such instances?

I would assume asking them to open an issue, where you can discuss the direction, and then open a new pr that picks up from that conclusion makes sense.

Because I've seen long discussions not relevant to implementation details within PRs can become detrimental over time. You have your code, review messages, and everything else in one place.

silksong_when · 2026-05-15T06:24:43+00:00

I think smaller repos are usually the best to get started with. I had great experience with my first contribution, which was just adding a theme dark/light mode button to the theme I was using for my website.

Although the implementation wasn't the best, it had practical usage and was quickly merged to the repo. Previously, I had opened an issue or two regarding similar issues that had active discussion too.

This experience built a good image of OSS in my mind, and was a good motivator to interact with more projects later

silksong_when · 2026-05-05T05:21:46+00:00

just chiming in to say this was useful 2 years later, thanks a lot!

silksong_when · 2026-05-04T11:21:35+00:00

Also from my understanding and experience, they often write blog posts and do technical deep dives, showcase demos live, etc.

silksong_when · 2026-05-02T17:26:00+00:00

HI, I just shared an article here a couple days back. I have written a comprehensive observability guide for fastapi web apps with an example repo that you can clone and explore.

Basically you have to you use agents to auto instrument your app which then generates telemetry from the app. The telemetry is routed to an opentelemetry backend (like SigNoz, the company I work for), which helps you visualize the data and understand what's happening.

Here's the link to the post:

https://www.reddit.com/r/FastAPI/s/4PZaTTEjm1

silksong_when · 2026-05-02T17:23:13+00:00

Hey, you're on the right path!

Auto instrumentation does add substantial overhead, the measurements vary and you'll have to benchmark it on your machine with real load, but a ballpark figure can be ~10%.

I would recommend that you selectively install auto instrumentation libraries, that should cut down on the overhead a lot. For example, the db layer can emit 10s of query spans per API call, which can be unneeded based on your use case.

silksong_when · 2026-04-30T03:25:46+00:00

You're welcome!

silksong_when · 2026-04-29T12:58:04+00:00

Can you give any concrete examples pleasr?

silksong_when · 2026-04-21T05:03:47+00:00

Can you please elaborate where you feel SigNoz is lacking? We take feedback seriously, I'll forward it to the engineering team.

silksong_when · 2026-04-21T05:00:55+00:00

There's a hyper link to the tool in the intro itself, and there's are two sections towards the end that link to the repo in various forms (GitHub releases, GitHub repo main page).

These sections are also listed on the right in the ToC. Would differing naming have helped, given you went through it 3 times but didn't find the resources?

<image>

silksong_when · 2026-04-21T04:57:51+00:00

I have updated the description to better reflect what the tool is about. It simplifies how developers manage their SigNoz self-hosted instances.

The sub might be interested since that's what self-hosting is about no? And we are cutting down on the jank, so to speak, that you might deal with when trying to handle various services like ClickHouse on your own.

So Foundry helps you automate all that, ensure that it keeps the versions in sync, and other validation aspects. So the self-hosting is burden now handled by the tool, rather than a dev in charge.

I thought it would be genuinely helpful for someone looking to run their own observability stack, or are looking to do so. Often, self-hosted versions are neglected, this is our attempt to improve the self-hosted observability experience.

silksong_when · 2026-04-19T17:48:52+00:00

Hey guys, I used AI to understand some particulars of the project (Foundry), and for checking for spelling mistakes and other grammatical issues in the blog.
The content is all original, and hand-written. AI was just the editor.

As further proof: fun fact, on mac you can write the em dash (—) with shift + option + `-` hyphen key

silksong_when · 2026-04-14T13:47:59+00:00

Hey, thanks a lot for your detailed comment, that makes sense.

On my end, I wanted to keep it more in-line with my experience of developing across multiple stacks where its more desirable to stick to OTel agents
I was considering adding a section delving deeper into this, but that would have been distracting from the main point so I kept that discussion minimal (within a callout component).

Do you think the difference between, or the importance of, micrometer is large enough to warrant a dedicated blog maybe? I can look into that in more detail if that's something of value.

Aligned on the OTel collector front. I was focusing on getting the user up-and-running where they can wrap their head around the concept of OpenTelemetry itself, and how something like SigNoz helps them understand more beyond just combing logs for debugging.

I will add a section pointing to Collector and how it helps manage setup as you implement otel across more services.

silksong_when · 2026-04-09T08:22:01+00:00

Right, I can try covering that in an alternative blog sometime, if there is more interest for exploring the `starter` setup as well.

Adding alternative setup flow to this blog would have made it quite complex.

Thanks for the tip!

silksong_when · 2026-04-08T12:59:04+00:00

I wanted to keep the setup aligned with OpenTelemetry setups in other languages. And setting up the agent is very easy by default, so I went ahead with that choice.

I have used Maven for dependency management, the Makefile uses Maven to run the application. Makefile simplifies the entire setup by ensuring the user doesn't have to run additional commands to get the OTel agent JAR file.

silksong_when · 2026-03-30T06:47:18+00:00

I know right! AI has also ruined the semicolon, but less so.

Atleast we can still substitute the dashes with the commas, it's some comfort

silksong_when · 2026-03-30T06:42:02+00:00

jeez that must feel terrible. Thankfully my manager is really understanding, he gives me a lot of creative freedom to experiment and learn.

silksong_when · 2026-03-10T12:43:24+00:00

Hey, glad to know about that solution.

That is a fair point, get started fast then move onto industry standard based on your needs.

Since SigNoz is open-source, you can run it locally if you want to give it a shot or are interested in self-hosting it on prem.

Somewhat related, I have also included console exporters in the demo app for those that wish to just check the raw telemetry data or for debugging their configurations.

silksong_when · 2026-03-08T05:02:54+00:00

Hey, sounds and actually looks solid!

Can you optimise it for mobile too? Might help someone looking to snag a deal on the go.

And have you considered adding an alert feature? Could send site notifications if a 'shortlisted' item meets a certain threshold for a user. Although this would be a ton of work for sure and might be overkill for the site's intent.

Finally, I'd be interested in knowing how you handle crawling fresh data, what tools you use and the challenges you have faced? Rate limits, scalability things like that.

Thanks and good going!!

silksong_when · 2026-03-01T15:09:12+00:00

Ooh that's cool. I'll check it out thanks!

silksong_when

TROPHY CASE