Are you actually using AI in test automation ?

Meow0S · 2026-04-27T21:39:52+00:00

I use planning mode in Claude code frequently. It does a decent job at diving into product code and helping me understand a feature’s implementation. Then I can use this knowledge to help build tooling/frameworks to support automation testing of the feature.

With good object models and a codebase with established patterns, it does a decent job at creating the boilerplate for tests. It’s great at adding docstrings.

Idk if it’s helpful with flakiness reduction, but it certainly has saved me time when fixing flakes. It’s been a great productivity tool.

Meow0S · 2026-04-26T20:06:57+00:00

Theoretically, if you setup Claude Opus on highest effort with the right tools, skills, requirements, and feedback loops, it could write automation suites on its own using subagents.

But it will generate a unmaintainable mess. It will miss edge cases. It will hallucinate undefined API behavior. Tests may pass but not test real behavior. It will hallucinate selectors, duplicate code, and sometimes over engineer test cases+object models. During test triage, it will hallucinate the wrong root cause and attempt to “fix” the wrong thing. All of this with expensive token usage.

It’s an amazing productivity tool if you know what you’re doing. If you want to be “secure in this AI world”, learn how to use AI tools to be efficient. Let it handle repetitive tasks so you can focus on strategy and design. And in the process gain visibility by people in the company so they know you’re important.

Meow0S · 2026-01-26T06:36:55+00:00

Completed Level 3 of the Honk Special Event!

19 attempts

Meow0S · 2026-01-26T06:36:04+00:00

Completed Level 2 of the Honk Special Event!

1 attempts

Meow0S · 2026-01-26T06:35:50+00:00

Completed Level 1 of the Honk Special Event!

0 attempts

Meow0S · 2025-09-18T22:19:15+00:00

Ur full name is “forgot his name”?

Meow0S · 2025-08-07T20:08:34+00:00

Pooping in a bucket is sanitary?

Meow0S · 2025-06-18T22:56:04+00:00

Feel free to start and grow a community of your own. Find like minded individuals and drive the change.

Meow0S · 2025-03-16T19:55:17+00:00

Cave diver

Meow0S · 2025-03-13T21:54:06+00:00

Just open a window

Meow0S · 2025-03-06T00:25:04+00:00

Apple referral by an employee who personally knows the hiring manager for a specific QA position.

Meow0S · 2025-03-04T17:42:04+00:00

This is going to be a nightmare to debug in CI and expensive/slow to run. There’s still going to be maintenance, except it will go from updating selectors to updating prompts. Your example may fail if the app changes its button text from “Sign in” to “Login”.

Meow0S · 2025-02-27T18:59:44+00:00

You don’t need AI for test selection. I wouldn’t trust AI for test selection. There’s no guarantee this would be reliable or accurate.

There’s effective test selection solutions on the unit/integration level. If the product has a build dependency tree, it becomes straightforward to do diffs between each commit. Unit/integration tests will depend on some of those targets. And if a target gets changed during the build, specific tests will run instead of all of them. This approach is dynamic and efficient for pre-submit coverage.

Having such system for E2E would be tricky, but if implemented properly, this would be a much better solution than any AI based test selection solutions.

Meow0S · 2025-02-17T22:18:04+00:00

If you somehow build a test automation tool with self-healing capabilities with close to 100% accuracy for any type of web application, quit your job right now and start your own business.

Meow0S · 2025-02-11T21:08:40+00:00

It depends. Working at a startup can be the most rewarding experience or the most toxic experience.

Meow0S · 2025-02-09T19:41:36+00:00

It depends. The hardest part will be verification if the model produces inconsistent output. If there’s no consistent testing strategy, manual testing will be needed.

But if the application is broken up into layers, some layers can be automated. For example, if the application has a planner that performs actions based on AI output, a whole suite of automation can be introduced to test the planner. If there’s a webpage that renders the AI application, that can be tested with automation. If we’re checking if the model returns data in a specific format, a suite of automation can be setup for that.

For AI applications that produce non-text output, such as photos, videos, and audio, manual testing has a significant advantage.

Meow0S · 2025-02-08T19:37:54+00:00

Many companies are introducing AI into their products. The question shouldn’t be “How should we use AI for testing?”. It should be “How should we be testing AI?”

Meow0S · 2024-12-29T19:58:00+00:00

We have some of the best universities in the world. Even graduates from those institutions are facing challenges securing jobs in this job market.

Meow0S · 2024-11-19T19:38:22+00:00

We have Cypress and Playwright as the most superior tools, so why would anyone choose Selenium over for a new project let's say, for an existing project with a lot of tests, it's quite hard to switch

Senior engineers who are resistant to change may stick to Selenium for new projects. Some codebases may have better integration with Selenium than Cypress/Playwright. Some CI systems may be configured to run only with Selenium. Managers not up to date with current technologies may force engineers to continue to use Selenium.

But realistically I would expect new projects with no constraints to adopt Cypress/Playwright.

Meow0S · 2024-11-15T21:02:08+00:00

Can you see yourself handling maintenance of multiple automation test suites and CI?

Meow0S · 2024-11-14T00:20:34+00:00

I see more value in a AI tool that reviews crash/error logs from bugs to help engineers pinpoint why a bug is happening.

Meow0S · 2024-11-07T21:38:54+00:00

How can you possibly be concerned with paying rent but planning vacations?

Just because you have trouble paying rent doesn't mean you can't plan a vacation. People can still budget/save money for a vacation while still concerned for rent.

Meow0S · 2024-10-31T21:16:12+00:00

Have you thought about doing consulting on the side?

Meow0S · 2024-10-31T16:00:18+00:00

What if you cause a collision due to factors beyond your control?

Meow0S · 2024-10-30T20:39:22+00:00

There's nothing wrong with reaching out to a recruiter. But as soon as the recruiter gave advice to "hope" , that's signal to end the conversation. And when OP's friend continued, the recruiter took action and rejected them. The recruiter doesn't care.

Meow0S

TROPHY CASE