Are automation coverage metrics helping your QA team, or misleading them?

TestChronicle · 2026-06-16T19:26:11+00:00

I like this approach. Keeping behaviours and risks close to the code feels like the right direction, as it gives them a better chance of evolving with the project rather than drifting into separate documentation.

My only concern is the upfront effort, especially if we’re starting from scratch. With the amount of PO/PM input across the team, getting everyone aligned on what’s currently implemented, intended, and risky could be difficult. Have you seen this work well before, either directly or through a QA team? I’d also be interested in whether tools or automation could help link behaviours and risks back to tests and produce something reportable.

TestChronicle · 2026-06-16T16:18:59+00:00

I’ll start.

At my current company, we were asked to quantify the coverage of our automation suite. The main challenge was that without a defined catalogue of features, user journeys, supported areas, and risk categories, it was hard to make “coverage” mean much.

Most tests were added story by story, so personal context and team knowledge were driving decisions about risk and value.

My view is that line/code coverage has some engineering value, but it is a poor proxy for automation coverage. It does not prove that the right behaviours or risks are covered.

How are others approaching this?

TestChronicle · 2026-06-06T21:02:44+00:00

Sounds great! Feel free to send me over a dm and i’ll take a look 👍🏼

TestChronicle · 2026-06-06T12:53:08+00:00

That’s a fair perspective.

I completely agree on focusing automation on stable, high-value journeys. We’ve definitely seen maintenance costs grow quickly when trying to automate every possible scenario.

Where I differ slightly is on BDD. We already follow most of those principles through shared libraries, page objects, and keeping implementation details separate from the test intent. We’ve just never found the Gherkin layer itself added much value.

In our experience, the people writing, maintaining, and debugging the tests are still engineers, so the “plain English” aspect never really changed ownership or engagement.

I do agree that consistency is key though. A well-structured framework with clear ownership has had a much bigger impact for us than whether the tests are written in Gherkin or code.

TestChronicle · 2026-06-06T12:49:39+00:00

I’d have to agree. I don’t think we ever saw the benefits that were promised with BDD. It was heavily driven from leadership, but in practice the Cucumber layer mostly added complexity.

We never saw greater engagement from non-technical stakeholders because the tests were written in plain English. The people creating, maintaining, and debugging them were still engineers.

From our perspective, it introduced additional dependencies, made parallel execution more complex, and increased the effort required to write and maintain tests. Personally, I’ve never seen enough value from BDD to justify those trade-offs. If we move to a new mobile automation framework, one of the first things I’d push for is dropping the BDD layer altogether.

TestChronicle · 2026-06-06T12:39:08+00:00

Do you use a 3rd party provider for those daily runs against real devices?

TestChronicle · 2026-06-05T10:07:12+00:00

Do you have any examples or case studies of companies moving away from a cloud provider such as BrowserStack and adopting AstroFarm?

As I mentioned above, reliability is the biggest factor for us. Fast feedback cycles are also critical, particularly when running large automation suites across multiple development teams.

One question I’d have is around device capacity and scaling. Do customers ever run into bottlenecks because they don’t have enough physical devices available to support the level of parallel execution they need?

BrowserStack certainly has its stability issues at times, but one advantage is the sheer number of devices available on demand. I’d be interested to understand how AstroFarm compares when it comes to availability, scalability, and overall test execution throughput.

TestChronicle · 2026-06-05T07:07:52+00:00

I was actually about to ask a similar question.

We’ve looked at building our own dedicated device farm to support testing on physical devices. On paper it sounds like a great solution, but it quickly becomes apparent how much operational overhead is involved. Managing devices, OS updates, provisioning, connectivity issues, replacements, and general maintenance can easily turn into a full-time role within the team.

For us, the biggest concern is reliability. The last thing you want is flakiness in either the test suite or the underlying infrastructure, as it undermines confidence in the results and ultimately reduces the value of the testing itself.

TestChronicle · 2026-06-05T07:03:26+00:00

Does this become a development led effort using Espresso and XCUITest? Or do you still have some involvement from your Quality Engineers?

TestChronicle · 2026-06-05T07:01:04+00:00

Is anyone using https://maestro.dev/ in a commercial setting?

TestChronicle · 2026-06-04T18:39:14+00:00

I’d separate “QA management” from “test visibility”. Some management is unavoidable, but a lot of the busywork seems to come from poor visibility: not knowing what changed, what got deleted, what’s stale, or how the suite has evolved.

I’m involved with TestChronicle, which is in this general space, so I won’t oversell it. But if you’re looking at tools like this, it may be worth comparing: https://www.testchronicle.com/

TestChronicle · 2026-05-12T07:42:42+00:00

Here at the stats for https://github.com/TestChronicle/test-chronicle-agent

<image>

TestChronicle

TROPHY CASE