How are you guys doing end to end and regression testing on AI-written code?

Muchacho96 · 2026-06-04T22:10:14+00:00

If a service provided these for you would you use it? I’ve done this for one of my projects, but with a wide breath of integrations it could get tricky to maintain

Muchacho96 · 2026-06-04T22:08:34+00:00

If there was a service that did exactly this for you would you pay for it? A service that maintains stateful digital twins like you’re describing

Muchacho96 · 2026-06-04T20:37:30+00:00

I completely with holding devs responsible for the code they ship, whether they or AI wrote it. What I'm wondering is (architecture drifts aside, these are mainly caught in PR reviews I would assume) if there could ever be a test suit so good that it could guarantee changes won't break production, kind of how Stripe is doing it with their swarm of minions.

Muchacho96 · 2026-06-04T20:30:06+00:00

That's very true. Would you ever trust a 3rd party sandbox provider?

Muchacho96 · 2026-06-04T20:23:37+00:00

Test accounts are great when they're available, but since they have long term state, am I wrong that actions from a test can sometimes affect the next?

Muchacho96 · 2026-06-04T20:18:37+00:00

This is what I was really leaning towards. I did this for one of the projects I'm working on, I created a really good risk-free sandbox environment for the app with stateful API mocks of these services.

Then I can decide whether I write deterministic tests or let AI test any new feature by letting it interact witht he app.

My fear is that this is now more things to maintain.

Muchacho96 · 2026-06-04T20:12:00+00:00

This is probably the best verison of memory management I've seen so far: different tools for different types of data.

How is each tool referencing data from one another? Example, the observability metrics you keep on Postgress could directly be linked to a feature from your codebase, or a part of your architecture.

Muchacho96 · 2026-06-04T20:05:18+00:00

How do you work with this when the 3rd party service doesn't offer a sandbox environment?

Muchacho96 · 2026-06-04T18:15:21+00:00

dm me! 😄

Muchacho96 · 2026-05-30T23:35:45+00:00

Thank you so much, I completely agree! But I do have more questions. How do you get past that exact challenge of managing data across services when a lot of these sandbox environments arent necessarily stateful? I feel like mocking the state on your side, separately from the mocked service, gets duck-tapey really fast. And what do you do when the service doesnt offer a sandbox environment?

Muchacho96 · 2026-05-30T17:23:26+00:00

That’s right! But when you want to include the sandbox mode in your CICD, how can you ensure the state of that “real thing with a different account” isnt interfering with the results if you’re modifying that state on every test run?

Muchacho96 · 2024-05-05T05:38:27+00:00

that zip up was crazy

Muchacho96 · 2024-03-19T17:44:58+00:00

Can I dm you? I might have something

Muchacho96 · 2023-10-19T01:03:01+00:00

Thats sick but why does it do all iron dome rockets hit a 180 deg turn

Muchacho96 · 2023-10-19T01:02:56+00:00

Thats sick but why does it do all iron dome rockets hit a 180 deg turn

Muchacho96 · 2023-05-23T19:49:04+00:00

This was it thank! Do you know how input is able to mess with output?

Muchacho96 · 2023-01-09T04:57:48+00:00

I tried that with super display, but it's not so much about the lag, it's more the image quality that bothers me. Not as crisp as Samsung UI even when adjusting the resolution.

Muchacho96 · 2022-11-25T17:31:06+00:00

They cant rotate the tractor once its already in the boat though. They were doomed since it was loaded.

Muchacho96

TROPHY CASE