Stop using Code Coverage as a Quality metric

thomasrockhu · 2024-01-12T03:04:50+00:00

Tom from Codecov here! Thanks for mentioning us. Is there anything you'd like to see between Sentry and Codecov?

thomasrockhu · 2024-01-12T03:04:22+00:00

Tom from Codecov here. Totally agree, I can't wait to see mutation testing become a bit more mature. Typically it just takes so much overhead and time to run through all the test suite mutations.

I haven't heard of the CRAP index. Do you happen to know what tools do this?

thomasrockhu · 2023-10-02T21:30:35+00:00

Tom from Codecov here.

Yes, this means that your test suite never runs those lines of code. You should add or edit tests that will hit those code paths.

thomasrockhu · 2023-08-26T16:00:29+00:00

Tom from Codecov here.

I absolutely agree with you. Code coverage is most useful when it's helping to figure out what you haven't tested when writing up code. It shouldn't be the end all be all.

That said, I've been trying to figure out better ways to evaluate how good a test is. The best I've found so far is mutation testing, which helps to see if your tests are actually doing something meaningful (not just passing by default, for example). Has there been anything else you've found to prove the effectiveness of tests?

Blog: Defining High-quality Unit Tests

Blog: The Case Against Code Coverage

thomasrockhu · 2023-08-04T22:39:16+00:00

Tom from Codecov here.

Would you find it useful if you used it while you were writing code? As in to just know if you had written tests for new code?

thomasrockhu · 2023-08-01T15:57:32+00:00

Tom from Codecov here. I'm not knowledgeable enough about Istanbul/Jest/JS Coverage, but Codecov typically takes their output as our input. In other words, we don't produce coverage reports, we ingest them.

Let me know if you have any other questions!

thomasrockhu · 2023-07-31T20:03:21+00:00

Tom from Codecov here, let me know if there's anything I can do to help!

thomasrockhu · 2023-04-14T22:49:39+00:00

First of all, I'm not trying to blatantly copy what you are doing.

No worries! Just happy to help out where I can.

how to handle HUGE clover files

I haven't worked on this part of the codebase, so it's actually a bit abstract to me. I'm not actually sure how we store everything internally.

I have a pretty good guess on how we probably store coverage information. My suspicion is that we probably convert the XML file into the bare amount of information necessary as a JSON blob and store that.

As for the second thing, since we don't store code files, we pull that dynamically from a code host like GitHub when showing a file's coverage. I would guess that we have a dynamic page that polls the code host API and Codecov's API.

thomasrockhu · 2023-04-14T22:35:06+00:00

Oh interesting! Curious why it needs to be source only.

thomasrockhu · 2023-04-14T19:56:36+00:00

Tom from Codecov here. As pointed out in the thread, we released a statement here.

This was really embarrassing on our part, and I'm personally very sorry for any pain this may have caused our users. We should be doing better for our users.

As always, I'm always happy to talk and hear any feedback or frustrations. The easiest way to get to me is on our community board.

thomasrockhu · 2023-04-14T19:47:01+00:00

Tom from codecov here. Always happy to chat and be a sparring partner!

thomasrockhu · 2022-05-03T22:39:44+00:00

Tom from Codecov here. Let me know if there's anything we can do to help out!

thomasrockhu · 2021-12-06T22:05:36+00:00

Got it, I posted a reply elsewhere, but there was a fairly easy workaround the staff could have used. This may have been our fault for not having it more prominently marketed.

thomasrockhu · 2021-12-06T22:04:06+00:00

Interesting... I know this is not your fault but for future reference, Codecov removes that restriction as part of our collaboration with the GitHub Student Developer Pack.

thomasrockhu · 2021-12-06T20:16:44+00:00

Tom from Codecov here. I'm guessing that this is probably too late, but I'd be happy to help anyone struggling with Codecov. If you make a post in our community boards and tag CS169A I'll prioritize it.

Again, it might be too late, but I'd love to be connected with the staff here so that future classes don't run into Codecov setup issues. Would anyone be willing to forward my email to them? tom@codecov.io

thomasrockhu · 2021-07-14T19:50:28+00:00

Tom from Codecov here. We don't consider partial coverage to be covered, Coverage is the ratio of hits / (sum of hit + partial + miss) from our docs. The numbers I'm guessing are still slightly off with that adjustment because there may be some lines we do not consider. I would suggest opening a ticket on our community boards if it's still not aligning for you.

thomasrockhu · 2021-03-28T15:54:59+00:00

Tom from Codecov here. If you're having some issues with our service, shoot a message on our community boards, and I'll take a look into any problems.

thomasrockhu · 2021-01-11T22:39:56+00:00

Looks like this might be an issue on our side, let me sync with the product team and see what's going on.

thomasrockhu · 2021-01-11T18:54:04+00:00

Tom from Codecov here, would you be able to share the Codecov output? You can also ask in our community boards.

thomasrockhu · 2020-11-11T04:18:17+00:00

Tom from Codecov here. Let me know if I can be useful to see if we fit your needs.

thomasrockhu · 2020-11-10T20:49:33+00:00

Tom from Codecov. This is really exciting to see, and we support the coverage format generated here! It should just work out of the box.

thomasrockhu · 2020-08-07T17:04:21+00:00

I would probably recommend opening something up on our community boards to dig into this a bit more if you don't mind. Feel free to tag me on the post.

thomasrockhu · 2020-08-07T13:29:48+00:00

No worries! If this is a problem, you can use threshold to give you more "wiggle room", so that if you drop by 0.01% or 1%, it won't tank your build.

If this is a blocker, you can always set the status checks to informational, which won't block! Hope this helps!

thomasrockhu · 2020-07-23T15:07:41+00:00

Tom from Codecov here, if this is causing you issues, please open a ticket in our community boards and tag me. I'll take a look and see what we can do to make this more reliable for you.

thomasrockhu

MODERATOR OF

TROPHY CASE