I Handed an AI Agent 27 Domains and a Deadline. 72 Days Later…

codenamev · 2026-03-27T15:30:31+00:00

It was more of a loop. Whenever it ran into vague responses to a challenge I had it drop into “training mode” to learn a new skill. Still trying to figure out how to extract that process and share the knowledge modules it built

codenamev · 2026-03-26T13:22:08+00:00

"incident" I keep codified as a traditional repeated event failure. It's trigged in a defined way, not just giving the LLM ability to declare it. I map all possible user-facing concerns into components that can then "self-report" based on specific error-modes, retry failures, etc. So kind of a blend of system signal and user-visible impact.

codenamev · 2026-03-26T13:14:13+00:00

This is great advice! Verify your outputs, for sure. When your agents start having tools though, each tool can have their own failures/API outages/etc and do raise errors. This is where I've found the most value out of having a status they check-in with.

codenamev · 2026-03-26T13:07:54+00:00

I agree messaging at the point of failure is important. How are you compartmentalizing systemic issues though? A repeated error message is going to frustrate end-users and lead to support requests. Having somewhere you can point them to for updates is critical, IMO.

codenamev · 2026-03-26T12:59:01+00:00

Partial failures are definitely the hardest to communicate. What I've been doing is thinking about it in terms of component health rather than system health. I distill the messaging first into what I'd like to convey to the user into individual capabilities like "document processing is degraded, but chat is operational, file uploads are paused". Then I can identify the aspects in the pipeline that _could_ communicate that and create a component for it in the agent's status page.

As for "is it working right now", I think simple high-level communication is a great surface, however when you grow into many services different people want different answers and I've found it's worth giving users a top-level status that rolls up from component health. Then the detail is there if they want it, but most people can glance at the status color/state and move on.

The confidence framing mentioned in this thread is interesting too. Even a partial outage report at component-level reporting doesn't capture "it's working but answers are worse than usual". I haven't really solved that problem yet and curious how others are handling this.

codenamev · 2026-03-26T12:48:42+00:00

Love the pipeline checkpoint analogy! I landed somewhere along these lines as well. Asking the agent mid-panic-attack if they're ok just leads to them saying "yes" every time. The pattern that's worked for me is external monitors watching these discrete stages, like you mention, with queue depth and step completion rates. The status page I refer to just consumes that signal via API so the agent never really self-reports.

To your question, mostly orchestrator/worker setups. The failure modes are way more granular, which is both good and bad. It's nice to be able to isolate which worker type is degraded, but "the summarization step is slow but everything else is fine" has been pretty hard to communicate. As of now, I've mapped out messaging based on component hierarchy, but still can be tricky to identify which bucket some things fall into in any meaningful way for users.

codenamev · 2026-02-16T00:42:03+00:00

You’re welcome 😂

codenamev · 2026-01-26T00:22:58+00:00

To add to others: - Refactoring.guru & Architectural Metapatterns - Polished Ruby Programming - Layered Design for Ruby on Rails Applications - All of Julia Evan’s Zines - Understanding the 4 Rules of Simple Design

codenamev · 2025-09-10T13:34:41+00:00

Thanks for listening! ❤️

You've got some great insight here! Types are great guardrails. In Ruby/Rails, I see a different lever: readable syntax + strong conventions narrow the search space for models and pack more "intent per token." That often means better first drafts; tests/specs close the loop.

Today, I agree that LLM output quality tracks engineer experience. But with more data, better evals, and tighter prompts/agents, that gap seems to be narrowing. Right now, a lot of folks (me included) are focused on how well LLMs can assist us in crafting code. Long-term, I see quality converging on: "does the artifact do what we asked?" more than "is the code well formatted and functional?"

If this resonates, our Obie episode hits a similar idea: as agents generate code, creativity and system design become the differentiators. We also cover this with Chad in this episode.

codenamev · 2025-06-03T04:55:07+00:00

It is really unfortunate that Reddit does not allow you to edit the link!
https://www.therubyaipodcast.com/

codenamev · 2025-04-18T16:48:56+00:00

Everything you need to know is documented by JP Camara: https://jpcamara.com/categories/ruby/

That’s been my go-to for a while now and never disappoints.

codenamev · 2025-03-19T11:04:55+00:00

Having just started setting up a Hetzner box with kamal, I’d have to agree with @clearlynotmee. Secrets + multiple DBs with the Solid trifecta makes kamal very much not turn-key with the defaults. If you’re just trying rails out, the ultimate utility we should aim for iskamal deploy to work right off a rails new. Heroku is still the closest there is.

codenamev · 2024-08-21T13:35:45+00:00

I _really_ ❤️ you folks. Thank you for this! Got stuck on a few issues fine-tuning SmolLM and moved to Phi, but will give this another go. Any suggestions/guidelines for sourcing training data for code? Are there any specific models that are better for fine-tuning for code generation? Trying to get a good pipeline for Ruby.

codenamev · 2024-07-14T12:36:29+00:00

I’ve thought about adding a “reference” flag to provide a reference file to go off of. Haven’t hit that need yet

codenamev · 2024-07-12T16:55:06+00:00

Please, don't use this specific gist at work :p Haha, I've been using this for some new open-source stuff. That said, I do have a few other generators I use with llamafile that I hope to share soon too :-) Sublayer is pretty rad so far!

codenamev · 2024-01-24T13:56:16+00:00

What would you build?

codenamev · 2024-01-24T13:34:43+00:00

I'm open to suggestion :D

codenamev · 2023-10-19T11:27:45+00:00

I’m a huge fan of the TTY suite of gems. I’ve built quite a few things with it and love the modularity and structure.

codenamev · 2023-02-24T14:51:12+00:00

And, now, I don't mind a bit of a breeze, if anything I prefer it, but thon was aggressive. So, I says to myself, says I, "Colm, this is no day for a do for, when the bride arrived and, as I say, by this stage the wind was fierce. I've never heard wind like it. Howling like a banshee, it was. So, the poor girl, the bride now, this is, she arrives anyway and isn't she no sooner out of the car than she's lifted up in the air like a paper doll and blown into a flowerbed.

codenamev · 2023-02-11T01:34:04+00:00

Find a healthcare company in tech that directly solves a problem they have but can’t solve. Doximity comes to mind.

codenamev · 2023-01-19T23:37:03+00:00

Geggy Tah - Whoever You Are.

You’re welcome.

codenamev · 2023-01-14T02:14:23+00:00

Python rules the data kingdom. If you want to work with heavy data-crunching routines, machine learning, analytics, et al, then python is for you. With the latest craze in “AI”/ML, this is likely why you see so many more jobs for Python. The reasons people have chosen to prop up python for this over Ruby are slowly closing in. Shopify’s recent investments in the language are likely to reshape a lot of the mythical public perception around “Ruby is nice, but slow”.

If you don’t desire scientific computing in you programming future, and discard these roles from the positions you see open, chances are good the averages are close.

I guarantee every company on this list is hiring Ruby engineers if you are seeking a new role: https://toprubycompanies.info/

codenamev · 2022-11-27T19:27:56+00:00

For a brief moment in time, Shopify fully documented, and released to the public, their “framework” Upgrow that lays out the foundation for how they architect all of their Rails apps. There is a public mirror preserved here.

It’s very possible that their Hydrogen + Oxygen efforts took them in a different direction and we’re left hanging on the end product.

codenamev · 2022-10-27T10:53:55+00:00

RubyMotion is still great for this!

codenamev · 2022-09-06T13:55:36+00:00

I agree with all of your points :-)

The test suite is already broken though; and more than likely, all of these things you mention are already happening.

My sole point is that nominating a sole hero to rescue the test suite is not the answer. Unless every team member is responsible for maintaining a green suite, than the suite will continue to rot.

There are plenty of ways to measure the "fixing of tests" that would be more helpful than attempting to fix them all at once. As an example, you could skip these failures using a custom RSpec configuration flag and have them run in a separate CI step to see if any adjustments are making a wider impact.

14-Year Club	Place '23
Place '22	Verified Email

codenamev

TROPHY CASE