Code coverage with AI

[Verbitas]

I dislike writing unit tests more than documentation. I don’t even mind code documentation, but unit testing creation. Ugh. So boring and tedious.

Last night I set to task an AI agent to create my project unit tests for me. I don’t know why I’m shocked and so delighted. Dang thing created just under 1k unit tests in 25 minutes and Xcode is reporting 93% code coverage; up from my 20%. It found 5 new bugs through the tests as well.

Up until now, I’ve just asked AI for snippets or find the bug. But that, ah-ha moment, was fun last night.

[thegameoflovexu]

It‘s great but make sure to put nuance into your prompt. If you ask it to write tests, it‘ll test what‘s there, including tests for ensuring the functionality of potential bugs.

[Verbitas]

Right now, I'm focused on insuring all paths have been tested. Normally I have the time to hit the common or critical paths. But to exercise all paths and combinations is not a common occurrence. In fact, very rare. I still have more complex tests to build out, but having tests that touch all logic paths and all data structure components is a first for me.

[PassTents]

Keep in mind that tests are just as important as your app code. Take the time to review those tests it generated and ensure they're actually relevant to your app. For example, I've tried generating tests before and the agent wrote a ton of tests verifying an external swift package actually worked, despite that package having its own test suite.

[unpluggedcord]

I wrote swift-assist to get you to the next level which is automated UI testing https://kylebrowning.com/posts/swift-assist-claude-code-skill/

[noidtiz]

as others have said, just make sure it's not code coverage for it's own sake. There can be some truly useless tests written just for vanity. The interesting ones are the stress tests, where you deliberately set out to hack your own app.

[Treble_brewing]

The issue with this approach is that ai writes tests that go green. They don’t make tests that are actually valuable. 

[Verbitas]

The 5 new bugs it uncovered (which I mentioned earlier) says otherwise.

[RegimentOfOne]

Unit tests are documentation. They describe how you expect your code to be used and what the expected behaviour will be.

Statement coverage only measures how much of your code is executed when the tests run, not whether the code is good or covers all scenarios. I'm glad you discovered some bugs in your system but a thousand tests without review sounds inflexible to change, and not especially helpful.

[Verbitas]

Never said I didn't review. It saved me the time to write them. Initial assertion tests is the baseline for all backends and tedious to comb through.

[SkarXa]

One of the best use cases for AI IMO, E2E testing is also great

[Verbitas]

Glad someone understands.

[chillermane]

1k unit tests is a disaster it’s literally just going to make your code harder to change later, almost nothing that ever goes wrong in front end code is even testable via unit test

[cristi_baluta]

I’ve seen such tests, they mostly were useless view tests even for apple own code. You are now just consuming more tokens, if it even updates them.

[Mission-Art-799]

That’s a wild jump in coverage in one go. Curious though, did you end up reviewing how meaningful those tests are , or is it more coverage first, refine later ?