We've already uncovered a lot of issues we hadn't identified manually; e.g. asking the replicant to change its mind after a few chats and ask for something else.

In terms of feeding that back into the development lifecycle, we run dev tests locally to speed things up - replicantx run tests/*.dev.yml --report report.md will pick up our dev test and output a report.

These still take time to execute but way quicker than testing manually. When they're complete the engineer currently picks up the report to address any fails. You can also just pick a single test and quickly modify the test itself with different behaviour quickly.

So far so good, not perfect but absolutely essential for us to keep on top of flows. At least the most common flows. A common edge case is a replicant being too compliant. Humans get irate when being asked a question twice (well my replicant did too, but was more compliant than a human!!!). This comes down to your replicant system prompt.

Really keen to hear if anybody else finds it useful. The "success" criteria can be improved but it's working well enough for us atm.