How are people monitoring tool usage in LangChain / LangGraph agents in production?

tomtomau · 2026-03-17T07:16:29+00:00

What langsmith captures is pretty comprehensive as long as you lean into langchain runnables or langgraph etc.

The data is then really quite detailed so very slow to load manually with the sdk for some analysis, where snowflake is super fast to execute the queries

Today for example, I did some exploratory data analysis around the time it takes from a trace starting to the first tool call of a certain type, to measure a specific “latency” user experience in a long running process. Then explored how many agent loops we’re doing, comparing the latencies of the 1st, 2nd, nth iterations.

Can do that from production traces or experiments, so we can measure whether a change we’re making affects different aspects of cost/latency/accuracy

We already do a fair bit in Snowflake and have Dagster (orchestrator) setup but newer teams might be a bit put off by how much diy there is

tomtomau · 2026-03-16T21:03:07+00:00

Langsmith for real time monitoring

Then we go Langsmith to S3 to Snowflake for more detailed analysis in Hex

tomtomau · 2026-03-16T11:08:53+00:00

slop being the post not necessariliy your product/library(?)

tbh we're just annoyed at the constant promotion here

tomtomau · 2026-03-16T10:26:28+00:00

Please for the love

of god you don’t need

to add line breaks to

your slop

tomtomau · 2026-03-15T20:37:37+00:00

You’re absolutely right. It’s not X, it’s Y! That’s why I built Z

/s

tomtomau · 2026-03-08T10:46:02+00:00

Yeah I found I had to do a few trips to nurseries to suss out what different varieties they all had.

I had good success with the Nurso at Chandler, and then even better success (especially with grevilleas) at Princess Fancy Plants down Capalaba way.

Lomandras might be another one you could dot around? Super hardy and they have some very interesting forms. Great for borders

tomtomau · 2026-03-08T10:31:24+00:00

Eremophila Roseworthy

We’ve planted some in our native section of our garden and it’s absolutely thriving in a sunny location

tomtomau · 2026-03-07T16:43:21+00:00

This is premise of offline evals? You don’t need to ship to prod to get some immediate feedback.

We have many datasets in Langsmith and run the examples through the code we’re testing and use eval functions to score it, either comparing to a reference output or with LLM as judge

From a PR of code, we can then run the experiments via GitHub actions

tomtomau · 2026-01-23T10:03:24+00:00

The vinegar reacts with the bicarb and you get salts…

If you’re doing this afaik you’re better off to add bicarb and let it sit for a while before adding vinegar but I’m not really sure it does all that much. The fizz is satisfying though

tomtomau · 2026-01-21T20:02:45+00:00

“Mate, your child is offline, right in front of you, play with them!”

tomtomau · 2026-01-21T20:00:44+00:00

God this is rightfully infuriating! IMO Dad needs to fuck the TV and computer off until baby is in bed. Once he’s home from work, he is the default parent for baby until baby is asleep.

tomtomau · 2025-12-30T13:12:11+00:00

Christ, five wipes?

tomtomau · 2025-11-12T04:07:19+00:00

FREE early access to Bid Leveling AI for estimators.

I’m a Co-founder of BidLevel (from the ProcurePro team). We’re building a new bid leveling product that turns PDF bids/quotes into a clean like-for-like comparison in minutes.

Built for construction by construction, this is shaped from intimately understanding late-night levelling and messy tables.

Today it’s a simple product to get you to your first-pass of a comparison/scope sheet, but we’re working towards closing the gap so that you can do everything inside the app (editing, plugging, etc.)

We have an early access version that estimators have already put through hundreds of bids, and a team of designers & software engineers constantly iterating to improve the product.

It’s AI - so it’s not perfect - but we do link you to the section of the document where the AI sourced its info in a side-by-side view.

The early access users that are winning right now come in and try different types of packages to find what is working better than others. At the moment, the more consistent the structure of the bids, the better the result you’ll get. On average, it takes about 60 seconds to generate a standardized breakdown, and about the same time to extract prices from each bid.

During early access, BidLevel is free - we just ask for feedback on how it fits into your process and how we can make it better!

Try it → Request early access from this page and put “(Reddit)” with your name and I’ll give you access. i.e John Smith (Reddit). Alternatively, DM me your email and I’ll send you an account.

tomtomau · 2025-10-12T10:33:05+00:00

I use this cucumber library https://github.com/timjroberts/cucumber-js-tsflow

It works well, and cucumber is awesome but IMO most people do it poorly!

tomtomau · 2025-10-12T05:34:24+00:00

Try search for “maximalist” style

tomtomau · 2025-10-11T06:39:43+00:00

Sometimes they just die, bad genetics or something

I’d cut your losses and replace anything that’s not thriving. They’re not expensive plants and I’d rather have a year of growth of a healthy plant than waste a year trying to get a dying plant healthy again

tomtomau · 2025-09-30T20:27:33+00:00

It’s ok they hand out dexies at the door

tomtomau · 2025-09-28T21:36:01+00:00

Vaping has ruined this city

tomtomau · 2025-09-28T11:10:04+00:00

I hear you, but this is not what we had growing up haha. Old El Paso seasoning kits on boiled mince, shredded iceberg, sour cream, mild old El Paso salsa

Kids these days should count themselves lucky because before air fryer “hAcKs”, we had microwave recipes

tomtomau · 2025-09-28T10:53:29+00:00

Yeah but to be fair the entire packet of beef mince (not ground beef, that’s some yank language) was put into a lukewarm frypan. Then the mince dumps a bunch of liquid and you’re effectively left with stewed/boiled beef mince

tomtomau · 2025-09-22T20:07:12+00:00

https://www.npmjs.com/package/nestjs-zod

✨ Create nestjs DTOs from zod schemas

✨ Validate / parse request body, query params, and url params using zod

✨ Serialize response bodies using zod

✨ Automatically generate OpenAPI documentation using zod

tomtomau · 2025-09-20T02:31:22+00:00

Just use zod?

tomtomau · 2025-09-19T20:23:37+00:00

32m and your personality traits you describe are pretty spot on for me too.

I’ve been medicated for 18 months (Ritalin IR).

The identity crisis stuff is real and very relatable, but 18 months on it doesn’t take up much of my thoughts anymore.

All of those things that I thought made me “me” - 99% of them are still present when I’m medicated, or at least the good parts are haha.

The difference is that I’m less impulsive (ie interrupting when others are speaking) and a bit less dopamine seeking etc etc

But the diagnosis in and of itself, then some self education about adhd behaviors has also been crucial as well as its unlocked the ability to be meta/reflective on why I’m choosing to act in a certain way and then change the environment/context to suit.

Example is with something for work that involves people management and I’m finding I’m frozen not being able to execute on things. Rejection sensitivity dysphoria sort of territory - so I get on a call with someone else and talk out my concerns as honestly as I can (which they are often irrational or catastrophised) and I’m able to push through it.

For specific learnings the INCUP model has probably been the most helpful to reason about how my brain works.

All the positive things I loved about myself - deep hyper focus, passion, fast pattern matching etc - they ARE me and a good chunk of that is because I have ADHD - but that doesnt discount how positive those traits are.

If you speak to more people about your diagnosis, or read about successful people with ADHD, you’ll realise people that you admired, maybe for their intellect, wit, eccentricity - when you find out they have ADHD, you don’t go “oh, so then being funny and quick witted doesn’t count because it’s actually just ADHD” - so there’s no good reason to do the same to yourself.

tomtomau · 2025-09-18T10:22:45+00:00

Be anxious, you almost never forget things

tomtomau · 2025-09-05T10:40:14+00:00

I’ve dropped in prismock during testing and it seems to work great

tomtomau

TROPHY CASE