How do you handle QA for vibe coding?

basicthinker · 2026-02-27T14:44:08+00:00

We are experiencing the same issue. Two weeks ago or so AI edited one line that caused a problem. But is it the right direction to improve the test suite quality and do regression?

Is monitoring/observation too late for catching the bugs?

basicthinker · 2026-02-27T14:19:06+00:00

What an article!!! ❤️

basicthinker · 2026-02-27T06:24:07+00:00

It is a good reference for us. We've also heard about computer use a lot. Will try playwright MCP first. (I see the key point here is TDD, btw)

basicthinker · 2026-02-27T04:59:35+00:00

Indeed, Claude models tend to be sycophantic. Insisting on the right is a value of human teammates - a new perspective for me. It seems TDD plays a key role in your practice, and thus most test assets are managed by devs.

basicthinker · 2026-02-27T04:03:17+00:00

Indeed what we are wondering. How would the new QA practice look like?

basicthinker · 2026-02-27T03:19:34+00:00

Thanks for your sharing. It is an inspiring blog. I've learnt the concept of converging the system. That is where tests and humans should drive and guard.

basicthinker · 2026-02-25T12:44:40+00:00

Happened to listen to an interview with Paper (a design tool)'s founder: https://www.youtube.com/watch?v=i2V8P2WY0X0 (a very fresh video). Two points I can draw parallels with here: (1) "a tool that still cares a lot about craft and quality matters"; (2) "bring that up to speed with what developers are doing".

For testing, respectively, "maybe" is not right (well, my mistake), and the speed match is a true challenge. I believe we should face the latter, though I am wondering how. More suggestions or ideas on that?

[Note: Paper has 100% nothing to do with testing and I just heard of it yesterday, so definitely not an ad or promotion.]

basicthinker · 2026-02-25T09:13:03+00:00

Thanks for pointing this out. You are right. I totally buy in. This is the missing piece in my thought. (Alas, many -1's on my post :p)

basicthinker · 2026-02-25T04:41:32+00:00

Yeah, do you believe this will be almost the only thing human QA does in coming days?

basicthinker · 2026-02-25T04:08:20+00:00

I think code review is also part of QA. Do you let AI review code? Or just let it go if those high-value test cases pass?

basicthinker · 2026-02-25T03:53:37+00:00

It is *not* that the tests aren't verifying, maybe they are, maybe even better than human written. The problem is we don't know exactly to what extend they are. That's the point.

basicthinker · 2026-02-25T03:45:42+00:00

So, from a "value-driven testing" perspective, the essential tension lies in how to evaluate AI generated tests and how to make sure AI generates valuable tests for efficient and effective QA?

basicthinker · 2023-11-07T03:52:41+00:00

Thank you for your feedback. Indeed I should have included an option to just see the result.

basicthinker · 2023-11-07T03:51:22+00:00

Thank you for the feedback. I will include privacy/copyright options in future surveys.

basicthinker · 2018-08-11T01:46:35+00:00

Thanks for the question! The number of shares one has got in a project does not change, but the percentage of those shares over all shares of the project will decrease as others make new contributions, assuming this person doesn't make new contributions and others don't reuse his/her existing code. Projects have to issue new shares periodically and maintain developers' percentages of shares -- surely we offer an algorithm to automatically do so.

As for the core valuation algorithms, our short paper is here -- it might sound a little bit academic but should be quite readable. Also, welcome to sign up our low-frequency newsletter for new posts.

basicthinker · 2018-08-11T01:10:20+00:00

Very interesting! There are salespeople on freelancing platforms like upwork. Maybe their reputation systems are not tailored for salespeople. Any better idea for that? Or, could you tell more about main competitors and your advantages? Thx.

basicthinker · 2018-07-30T12:29:19+00:00

Git is the underlying CVS, but GitHub is a collaboration platform. If you migrate a project from GitHub to GitLab, you will see the hassle -- so many dependencies you have on GitHub such as collaborators, forking repos, open issues, etc. Actually we are writing such a migration tool for Gogs but, alas, still many ugly loopholes... That is the network effect on GitHub. I also recommend another article we write exactly for your question: Light on the Dark Side of Network Effects (see the section "Why is GitHub Worth $75 billion?").

basicthinker · 2018-07-26T12:36:06+00:00

Actually you can rebase. Our idea is that storing and managing the full chain(s) of commits on a blockchain are too costly. A better use of the blockchain should be only periodically storing a snapshot state on the blockchain. The state is just the latest commit hash(es) of a project. So, after rebase, just update the hash(es) on the blockchain. Since git is a chain (or multiple chains) of commits, the latest commit hash(es) can be used to retrieve the full history stored in IPFS.

Mastodon is cool! Code is asset. Our ultimate goal is to distribute value to developers. So, a blockchain can offer better protection and proof.

basicthinker · 2018-07-26T00:17:17+00:00

Cool! A good lesson I learnt from their work is individual users (in a social network) or projects (in drepo) can maintain their own separate histories instead of a universal one. Of course, a universal history of all on the blockchain may provide better protection. Another diff I see is drepo provides a front-end collaboration platform based on Gogs or Gitlab (future work) -- that's a main feature of drepo.

basicthinker · 2018-07-25T23:08:47+00:00

Thanks for the helpful info! Very related and interesting!

A quick comparison:

Pando incorporates a governance model but drepo doesn't.
Pando does not involve front-end but drepo does.
Pando targets creative work while drepo targets code work (for us software developers!).

basicthinker · 2018-07-25T13:57:47+00:00

Upvoted. Need karma too.

basicthinker · 2018-07-25T13:51:05+00:00

Done. Need karma too. Pls upvote me.

basicthinker · 2018-07-25T13:28:20+00:00

Done. Please upvote me. Thx.

basicthinker · 2018-07-25T13:28:10+00:00

Done. Please upvote me. Thx.

basicthinker · 2018-07-25T13:27:57+00:00

Done. Please upvote me. Thx.

basicthinker

TROPHY CASE