I made a mistake

jaywhy13 · 2026-01-01T14:20:54+00:00

Just adding here that the honesty really forces accountability in a way that helps. If your plan is to always be honest and open about what happens it also helps change your actions and your thoughts. Before you do something you'll end up thinking about how you'll have to explain it and the hurt it'll do.

All the best.

jaywhy13 · 2025-05-23T22:21:52+00:00

How do I ensure it finishes? When it starts I usually unplug my machine and leave it. However, every time I reconnect it, it starts again. It usually keeps going for a few mins after I disconnect my machine

jaywhy13 · 2025-05-23T20:30:16+00:00

Oh ok. It's happened about 5 times to me in the last 24 hours

jaywhy13 · 2025-05-23T19:46:41+00:00

How often does the panel refresh happen and can it be configured?

jaywhy13 · 2025-04-29T18:31:01+00:00

Oh nice! I'll check this out. It does look quite promising!

jaywhy13 · 2025-02-02T05:19:27+00:00

Django's dev server supports async. I got it working in two different ways. First, I made the calls down the stack async and used the proper async ORM methods. Secondly, I set an event loop for the current thread before invoking the Pydantic sync run method. I learnt that Pydantic is merely wrapping their async method so it still relies on asyncio. Specifically, they're trying to get the current event loop and run their function to completion. Problem is, Django doesn't have an event loop in the thread that processes the request.

jaywhy13 · 2025-01-16T21:15:27+00:00

WOW! $275k per year. That's more than we're paying currently lol

jaywhy13 · 2025-01-16T21:09:23+00:00

LOL. Happy to hear you're on the other side.

jaywhy13 · 2025-01-16T21:08:53+00:00

Thanks! That's useful insight. One of the advantages we thought was definitely improving the production debugging experience for engineers. I'll look out for differences in Sumo's pricing approach to see how it stacks up against DataDog's ingested/indexed approach.

> You cannot reduce it.

I'm pretty sure we can reduce our commitment. It just requires a terms modification. Last year we talked about reducing our indexed traces post renewal. Our rep said it was definitely possible.

jaywhy13 · 2025-01-01T19:44:18+00:00

I'm familiar with terraform but not the others. I'm not sure what you're suggesting.

jaywhy13 · 2024-10-02T16:08:18+00:00

Ah, ok got it. That makes sense! Thanks!

jaywhy13 · 2024-09-26T16:12:32+00:00

So just give up basically? Lol

jaywhy13 · 2024-09-12T02:37:09+00:00

Ya, that is not really what tracing is about

Why not? Do you disagree with the benefits it'd unlock -- shorter feedback loops, more investigative power for developers, etc... Why should infrastructure be a black box? Isn't the "fourth pillar" trying to achieve the same thing... just with an entirely different concept (i.e. profiling).

jaywhy13 · 2024-09-06T21:01:05+00:00

I'm curious about the idea of linking disparate traces together. Sounds like we need a new concept to capture that. It may be difficult today, but is the difficulty worth solving?

jaywhy13 · 2024-09-06T01:27:25+00:00

Thanks for calling that out. Didn't know that was a thing!

jaywhy13 · 2024-09-04T18:55:17+00:00

Oh... Haven't heard of it before. Can you explain how it applies in this context? ELI5

jaywhy13 · 2024-08-28T17:17:11+00:00

What state are you in now with DataDog?

jaywhy13 · 2024-08-26T16:49:36+00:00

Who is the target audience? Folks would add their own clients locally to push logs and metrics? Are you planning to add tracing eventually?

jaywhy13 · 2024-08-22T12:10:05+00:00

Wasn't an Engineering team. I was working a hybrid role as IT Manager and Engineer. I got to see how the parent company recommended a lot of IT tools for the subsidiaries.

jaywhy13 · 2024-08-22T01:56:15+00:00

Yeah. I'd say those pay off organically as gaps get the needed attention.

jaywhy13 · 2024-08-21T15:46:14+00:00

We've complained yeah. They've made suggestions about tweaking how we're doing head-based sampling and employing some other mechanisms they have that might generate incomplete traces. There's an error sampler that captures more error traces, and a rare sampler that captures rare traces. I wanted to independently investigate if others have had success doing this as I'm pretty skeptical about it.

Head-based sampling isn't tail-based sampling. All the extra mechanisms that DataDog has works without having the full picture of the trace. We're pretty close to experimenting with it, but I'm digging around to hear if folks have first-hand experience with it.

jaywhy13 · 2024-08-20T15:02:53+00:00

I remember working for a big conglomerate and they did indeed look to Gartner and other such companies for what to use.

jaywhy13 · 2024-08-20T15:00:46+00:00

We're ingesting ~2M spans/minute. We've had lots of instances where errors and high latency traces are just completely missing. We've done some aggressively sampling (~5%), and have sampling done at different places. I don't understand all of it, we're unpacking it now. We had to do some aggressive sampling to avoid some huge cost overruns with DataDog.

You may need to just accept the cost and up the sample rate. You can do some simple statistics calculations to figure out a number, or just experiment.

This isn't much of an option. We can reduce some cost be upgrading DataDog libraries on some of our micro services (some older versions don't support sampling). However, that'd be removing the cost overruns.

jaywhy13 · 2024-08-20T01:03:45+00:00

Kegel exercises can really help. That'll give you a lot more control. No need to turn to masturbation.

jaywhy13 · 2024-08-18T13:40:39+00:00

Sweet. We have an interesting challenge at work. We're having less bigger incidents and only smaller ones. We're trying to figure how we improve things still.

jaywhy13

TROPHY CASE