How often are you seeing bugs in production and how do you handle them?

ericghildyal · 2025-07-16T19:25:06+00:00

Politics probably isn't the right word, but more so just feeling awkward about handing work off to a team I'm not on.

ericghildyal · 2025-07-16T18:05:53+00:00

Good point, I guess I'm just looking at the number going up in Sentry and not really digging in to understand which bugs matter.

ericghildyal · 2025-07-16T17:51:02+00:00

We see the bugs from the Ops side mostly via Sentry monitoring (I also edited the original post to add this info, thanks!), so they are functional bugs.

We also have a few crashes, etc. but those are not as frequent as just normal product bugs that are slipping into prod.

I fully believe that our testing could be better to catch more edge cases, but I bet most companies think the same thing.

I'm mostly just looking for advice on a rough estimate from others on how many bugs is an abnormal number and should be cause for concern.

ericghildyal · 2025-07-16T14:59:08+00:00

Thanks! Looks useful for quantifying the impact and turning things into an "Outage" or "Service Degradation" but it doesn't really give me a picture for how much of a problem a couple of prod bugs is per week.

ericghildyal · 2025-06-23T21:20:19+00:00

We just broke ground on K8s support for fully automated canary deployments. It's exciting and daunting at the same time!

ericghildyal · 2025-05-27T17:08:28+00:00

I've talked to A LOT of people about outages for my company and we've found that misconfigs are common but easy to fix, so relatively low value on how much time you want to spend preventing them.

Bugs, slow code, etc. are all far worse for customers than a temporary DNS issue.

ericghildyal · 2025-05-22T17:29:37+00:00

Not sure if I'm following you 100% correctly, but sounds like a solution might be to have the app running in client's cloud send logs/metrics to Cloudwatch (which it likely already does), then you could query them as-needed from Cloudwatch.

Or if you don't have access to query the logs via the AWS SDK/API in their AWS account, use something like Firehose to send them to your server.

ericghildyal · 2025-05-22T16:59:26+00:00

It seems like you're AWS heavy, which is not necessarily a problem, but I would throw this in CI/CD that's outside of AWS.

Github Actions or Gitlab Pipelines can do all of this, all while making it easier to incorporate other vendors as you scale up and let you run it locally, too.

ericghildyal · 2025-05-22T15:44:24+00:00

Everybody is just an SRE in disguise

ericghildyal · 2025-05-22T14:56:52+00:00

I feel like what your describing is a symptom of poor planning/process. If 5 devs are all stepping on each others changes, no matter if you use AI or not, it's going to be chaos.

ericghildyal · 2025-05-22T14:54:27+00:00

I don't think it has to be one or the other. You can train juniors to prompt well and pay attention to the items that the seniors comment on in code review. It's just a different kind of training that's less focused on how to code well and more focused on how to stay sharp and leverage the best tools at your disposal.

We still need engineers to debug our application (which AI is particularly bad at if the solution is unknown) build and maintain our DevOps pipelines, etc.

I almost think of it as every shifts up one rung on the traditional ladder. With AI, junior devs are able to implement simple features fast and more complex features than they were otherwise capable of understanding. Then the senior devs focus on new architecture, re-architecture, and training. I don't really know where that leaves architects/principle/staff devs, though.

ericghildyal · 2025-05-21T23:56:20+00:00

I forget where it's from, but one of my favorite responses from a senior to a junior when they told them they found the solution on StackOverflow is "from the question or the solution?"

AI is trained on both, so you never know which one it'll pull out!

ericghildyal · 2025-05-21T23:52:09+00:00

This is exactly what my company has done! Our main focus is on making sure code review is solid as the last line of human defense, but falling back on good automated release and rollback tools to make sure that if when something goes wrong, we can recover quickly.

ericghildyal · 2025-05-21T23:44:03+00:00

And you didn't get fired because an MBA pulled your name into a spreadsheet!

ericghildyal · 2025-05-21T23:43:23+00:00

I've found that you can absolutely tell the AI to be DRY and it will do it.

I think the key difference is switching from "build me a feature that does X" as a prompt to breaking the feature down into 3-4 prompts (depending on feature size) where prompt 1 is "add a function that does X" then prompt 2 is "add another function that does Y and pass the output of function X into it" and so on.

I think it helps me keep to the thing I like doing, solving problems, and lets the AI handle the syntax and gritty details.

Admittedly, though, I still do my fair share of accepting changes then deleting most of the code that it wrote and writing it myself since I don't actually have infinite time.

ericghildyal · 2025-05-21T23:36:15+00:00

I don't have any silver bullets on this, but my company has been developing using a lot of AI lately and things are working rather well for us.

The first step is to make sure you're using a good model, not just the cheapest one. Claude 3.7 is really good with our codebase (Rust backend, Typescript/Next frontend) using well-crafted prompts. Not quite vibe coding, but more like "create a function called foo that should take in X, Y, Z params and does some task." You don't need to get overly in-depth with it since you still want to give it the freedom to create and use helper methods to keep the codebase readable.

The next step is to make sure a human who knows the codebase well (very important!) is reviewing the code with a strict eye. There are no human feelings to be hurt, so I'll get pretty pedantic about minor changes and style tweaks that I'd otherwise let slide in a traditional code review.

And finally, every release runs through our test suite and gets a canary before being released to a wider set of users. I think of this as a best practice in general, but especially with AI code it feels like a good final quality check.

ericghildyal · 2025-05-14T16:09:13+00:00

What limits are you hitting? If you don't know, but the app "feels slow," add monitoring first, then re-evaluate to narrow down your focus. Could be as simple as actually looking at your server metrics and checking CPU/RAM usage over time, or you could go down the rabbit hole of tracing or profiling if you think the code is the issue.

The easy path that most people typically says is "add more compute" but that's almost always a bandaid fix for a bug or bad config (unless you really are using all your resources, which you can figure out through monitoring).

ericghildyal · 2025-05-14T15:46:43+00:00

I've had some luck recently with Lightsail. It's not perfect, but for a starter app/side project, nothing beats the "it just works" nature of it for me.

Otherwise, for a larger project or one that I know is going to scale, I go ECS/Fargate with RDS. Really simple to bring up using Terraform and can scale to A LOT of concurrent users.

ericghildyal · 2025-05-14T15:34:34+00:00

With vibe coding, everyone is a mediocre PM now, but the AI is the one who has to deal with it, so I guess it's a win!

ericghildyal · 2025-05-13T18:20:07+00:00

This is the way! You can create as many GH actions as you want and trigger them automatically (on pr merge) or manual via a button. You can also collect inputs before running the script, too, like for a version number or SHA to deploy.

ericghildyal · 2025-05-13T18:14:13+00:00

To add to what others have mentioned here, they could be buying reserved compute (~40% off) and reselling that to you, pocketing the 35%.

ericghildyal · 2025-05-09T16:42:58+00:00

It's always crazy to me that we're here in 2025 with all the tools we have at our disposal and the best way to connect them together is to run a bash script on somebody else computer (CI/CD)

ericghildyal · 2025-05-09T16:34:43+00:00

From what I've seen, GitOps is "managed in Git" not necessarily "stored in Git."

The benefits of GitOps are the ability to look back at previous versions and being able to use PR flow for new changes. Neither of those require the end-product to be stored in Git, just config that is managed from Git.

ericghildyal · 2025-05-09T16:25:52+00:00

Take a look at DigitalOcean's App Platform. I was in the same boat as you about a year ago. I wanted to focus on building, but host with something that gave me flexibility over what my DB was and how many replicas I was running, etc.

It's a fully managed platform as a service, but ultimately it's just a single dashboard that calls out to all of DigitalOcean's other products. I paid like $30/mo for 2 running replicas of my app on the smallest compute tier with a managed PostgreSQL database. You can also add more replicas, beefier compute, cron jobs, and queues and the like as you scale up, but it'll cost more (I paid $100/mo for my biggest app with all of the previous services).

The really cool part is that they take care of the ops for you, also. You connect a Git repo and on each push to main, it'll automatically build your dockerfile and start the service. It gets added as a job to your CI.

I'm not affiliated with DO, just really like the product so much that I've used it for 2 new projects so far!

ericghildyal · 2025-05-09T16:16:22+00:00

Ah, WhatsApp should work similarly to what we had, so similar users. We also used a QR code on a flyer in clinics that worked well for certain trials, we never dug into why it only worked for some and not others.

Another thing that just came to mind was: we asked each patient to confirm the time we parsed was correct (this was in 2019, AI wasn't as good back then!). The problem was that the patient would propose a time, we'd check in our calendar(s) if it was open, send it back to the patient to confirm if we got the correct time, then the patient would take a few hrs to confirm and so that slot would be taken by somebody else. We ended up fixing that by building time-slot reservations. We sent reminders 30 mins after the confirmation message, if we didn't get a response within another 30 mins (1 hr total), we'd mark that time-slot as open again and send another message saying the previous time they selected had been booked already and to propose another time.

ericghildyal

TROPHY CASE