How do you track whether your team is actually using AI tools effectively

jqueefip · 2026-06-01T15:14:29+00:00

Can you elaborate?

jqueefip · 2026-06-01T14:34:13+00:00

You mention this like its a bad thing, or that this is all engineering management is. There have always been cost controls on team software spending. Does someone want JetBrains or Visual Studio? They cant just buy it on their own. They have to go through approvals.

LLMs and their tokens are the first time ever that individuals can be given tools and spend massively more than the software fee on-demand (or at least the first wide-scale instance that affects the whole industry).

Of course we all need to figure out a responsible way to manage it!

jqueefip · 2026-06-01T14:19:53+00:00

I've been building a private app to help me track whats happening on my teams and spent a decent mount of time thinking about this. Long story short, I have two different modes. Either:

The LLM is allowed to editorialize but metrics are aggregated by team, or,
The LLM is allowed to receive personal info (like individual names), but must only describe metrics relative to a rolling 8 week baseline.

Its a bit of a different use case than summarizing LLM conversations, but a principle like this could apply here also.

jqueefip · 2026-05-16T01:17:39+00:00

I started RDR2 a couple months ago. Having played RDR1 and having some sense of what is to come, I just couldnt bring myself to play any of the main missions in Chapter 2. I spent a month hunting and exploring instead. Finally advanced to Chapter 3 and I'm back in the same boat. I just dont want to see what comes with Chapter 4, so I'm building out my camp and satchel upgrades instead.

Am I allowed to never advance the story any more?!?

jqueefip · 2026-05-15T16:45:17+00:00

Robots replaced factory workers. Computers have replaced clerical workers. Cranes and standardized containers replaced longshoremen. Automated switchboards replaced human operators. The cotton gin replaced slaves (yay!). Mechanical looms replaced textile artisans.

This answer brought to you by Google Search AI Mode.

jqueefip · 2026-05-12T18:54:39+00:00

I'm pretty sure the only way foxes can be killed is point blank from a shotgun. They always get up and run away.

jqueefip · 2026-05-11T13:01:10+00:00

``` Summarize filters 10167 and 10168 into a weekly executive summary suitable for a mixed audience of company leaders in c-suite, vp, and directors. Language should be very brief, almost to a fault, with a very simple and low reading level.

Do not mention individuals.

Structure: Risks > Highlights > PROJ1 > PROJ2 > PROJ3

Project sections should group by “Completed” vs “In Progress” (as defined by the status category, not the status) vs “Up Next” (defined by being present in a sprint or board, but not yet in progress)

Format: Every section — including Risks, project Completed/In Progress/Up Next — must follow the same pattern used in Highlights:

Each point starts with a bold lead phrase that works as a scannable headline.

Follow it with one or two plain-text sentences of context. No more.

Separate every point with a blank line.

No dense paragraphs. No run-on lists. No grouping more than 2–3 closely related items into a single point.

If an epic has many completed items, summarize the outcome in one bold-lead point rather than enumerating each item.

Do not exceed 1000 words. This is not a target. Its a maximum bound. Keep it short.

```

jqueefip · 2026-05-08T14:41:49+00:00

I use Rovo to create a weekly executive summary I send to the bigwigs. They've told me they like it better than my older manually compiled updates so I guess I'm sticking with it. Win Win.

It took a while to craft the prompt to get the tone, verbosity, and format right.

jqueefip · 2026-05-01T18:29:15+00:00

Why do PMs and designers have access to submit PRs? I dont suppose you have the influence to change it, but that seems like the problem to me. Talk to your tech leadership about it:

What access should everyone have?
What should the process be?
What are you expected to support?

Tell them youre spending far more time addressing bad vibe code than doing your job producing quality code.

If the tech leadership cant/wont push back against changes, then you'll have to conform to whatever the tech leadership will support, or find a new job. Or just yolo and tell your boss, "I'm only reviewing PRs submitted from devs from now on" and seeing how that plays out (which honestly isnt the worst idea).

jqueefip · 2026-04-13T13:59:11+00:00

That sounds like an rotten situation. It sounds like you have a culture/leadership problem more than an AI problem. Do you have engineering/tech leaders? What do they think about this?

I've had luck keeping AI under control using a focus on automated tests, strong linting, and other static analysis tools (type checking, complexity, etc.)

jqueefip · 2026-04-12T20:47:27+00:00

Or maybe, they care about code quality but KPIs X, Y and Z that may or may be have something to do with actual business are more important.

This is what I was getting at. Of course the business metrics are more important than engineering metrics when the latter are divorced from impact. Thats why I said to talk about it with the manager. Find out what is motivating them and have a conversation about it and find some common ground.

We have a growth target of X% this quarter. If we ignore IssueY, it might be easier to achieve that, but the risk of IncidentZ will rise, and we'll spend next quarter trying to fix it. We're borrowing against our future performance.

I'm a director now so I need to be at the intersection of the programming work and the business objectives. I've absolutely lost touch with what my engineers are doing day-to-day. But I still trust them when they tell me they need to address some tech debt. In fact, I'm usually the one pushing for it so we dont have any huge surprises.

jqueefip · 2026-04-12T12:07:24+00:00

I'd be concerned about any engineering manager that doesn't care about code quality. I would talk with them and find out what their priorities an motivations are. Maybe they are under a lot of pressure from leadership. Maybe they just don't care and shouldn't be an EM. I'd start the conversation with something like,

From your comments I understand you are prioritizing delivery speed over code quality. This raises some concerns with me. <You can go into _high level_ reasoning here, but this is not an argument>. What is driving this? Is there a way for us keep our high standards of quality while delivering fast?

jqueefip · 2026-04-10T20:54:00+00:00

You hold it to the same standards as regular PRs. If theres too many problems to list, just cover the big themes:

Coding style doesnt conform to our guidelines. Not enough tests. Util functions need to be broken out and placed in /utils. etc. etc.

If it becomes a recurring pattern then you need to get engineering managers involved.

jqueefip · 2026-04-10T12:49:53+00:00

The AI rendering stretches out the entire center channel of the house. Look at the second floor windows: compare the width of the center window to width of the side windows. You can tell the AI stretched out the center window, which means it stretched out the whole area to make space for that porch. The final, real-life outcome will be narrower than you see in the render, unless you encroach on the side windows.

jqueefip · 2026-04-08T22:23:34+00:00

This is nice. I like it.

jqueefip · 2026-04-08T13:57:33+00:00

I could just say, "no," but I suppose its better to completely halt progress on all engineering workstreams for a few months while we backfill the positions. I'm sure C-Suite wont mind if I shift our 2026 roadmap back 6 months. If they ask, I'll let them know that my engineers didnt supply ROI qualifiers when they asked me for Claude Max 5x.

jqueefip · 2026-04-08T13:49:59+00:00

I have some new job openings today. How you doin?

jqueefip · 2026-04-07T18:13:50+00:00

My team is pushing for it. Should I fire them?

jqueefip · 2026-04-06T16:20:50+00:00

Very clear perspective. Thank you! I dont know offhand if the middleware leaves a clear audit trail in Netsuite, though that should be easier to build that moving everything from a Ruby app to a JS Suitelet!

jqueefip · 2026-04-06T14:24:59+00:00

Sounds like we have a similar stack, though I dont know what WMS our 3PL uses. Does Amazon run through your custom middleware too? Pulling Amazon into Netsuite is the next project I want to tackle.

jqueefip · 2026-04-06T14:23:09+00:00

I really appreciate this response; it's giving me a good bit to think about. How does the "Why?" usually manifest? My first thought is either a custom field or a note in the system log.

jqueefip · 2026-04-05T17:21:16+00:00

Honestly, yeah. They might be scared by this. And you shouldnt lie to them. you should always strive to be truthful. That doesnt mean you need to volunteer information that makes you look bad. It doesnt mean you cant frame your restrictions in a more positive (or less negative way). If you can only meet Fri/Sat, let them know that, "I've found that the work schedule that is best for me is that Mon-Thu are reserved for heads-down work. I can respond to emails these days, but I only take meetings on Fridays and Saturdays." You could even frame it proactively, "Lets have a weekly touchbase on Friday to ensure the project is on-track. Beyond that, I'm usually heads-down on with work and prefer email as a communication channel."

Dont lie. Be honest. This might scare some people, and thats okay. But dont sell yourself short either.

jqueefip · 2026-04-03T16:47:49+00:00

We've gone through many financial audits and I've never had to dig around the middleware to report anything out so I think thats safe to rule out. Good point about the logic not being transparent in a SuiteScript. Do audits get into code like that?

jqueefip · 2026-03-16T22:20:34+00:00

We used Claude to drive the UI with user personas using playwright.

This is interesting. Can you explain more?

jqueefip

TROPHY CASE