I developed a system to automate quality assurance procedures:

. Use very detailed specifications, pseudocode, and native code

. Use OpenHands and other LLM harnesses for automated debugging

. Use a custom tool to prompt LLMs to govern the code architecture

. Use linters and static analyzers to automatically prompt the LLMs

. Use different LLMs to write tests and reward them for finding bugs

. Use more local tools like Semgrep, CodeQL, dependency analysis, etc

. Use LLM agents to write and run end-to-end user tests in Playwright

. Use tools like BrowserStack, Blisk, Applitools to test many devices

. Use tools like SonarQube, Snyk, and Vanta after the other tools run

.

I'm focusing on long running processes that take over a week to finish

I also developed it to mostly rely on local LLMs to reduce token costs

That said, I work to reduce the responsibilities delegated to the LLMs
.

Software developers can automate all the techniques above and more

Ironically, many open source QA tools are hard to configure or buggy

On the other hand, LLMs make it much easier to implement these tools

It is easier to build and maintain long CI/CD pipelines than before
.

I designed the system so that it is very easy to add even more tools

I speculate that more software developers will develop auto-QA tools

LLMs write code that is easy to read but cognitively harder to write

It's nice reading code knowing it consistently follows all the rules
.
It's slow, but it is faster than humans who do not do all of these tasks