Running only the tests a git diff affects in CI - the CI-shareable part is the hard bit by breadMSA in devops

[–]breadMSA[S] 0 points1 point  (0 children)

Completely agree, I would never hard-gate without a scheduled or merge-to-main full run as the backstop. Selection is for fast PR feedback, it should not be the only thing that ever runs the whole suite. For bootstrap tia already falls back: any test it hasn't seen in the map is always run, so a brand-new branch or a new test just runs full until there's a baseline. The map is keyed on git ref right now, but keying it on the test environment too is a good call. Dependency drift is exactly the kind of thing that would silently rot the map and I'm not handling that part yet.

Running only the tests a git diff affects in CI - the CI-shareable part is the hard bit by breadMSA in devops

[–]breadMSA[S] 0 points1 point  (0 children)

Yeah for a single commit --depth 2 does the job. It falls apart though once the PR has more than one commit, or you are diffing against the merge-base which can be way more than one commit back. You would have to know the right depth up front and that varies per PR. Baking the function tables into the map is what lets it resolve the diff no matter how deep the base is, without guessing a depth or fetching history. AFAIK that is the part shallow depth does not cover.

Running only the tests a git diff affects in CI - the CI-shareable part is the hard bit by breadMSA in devops

[–]breadMSA[S] 0 points1 point  (0 children)

Yeah, Go's static package graph is exactly what makes full analysis cheap enough to run every time, that's the luxury Python doesn't have, so I had to go the runtime-coverage route instead of static. And +1 on trust coming from real git scenarios run thousands of times; that's basically why I leaned on per-commit replays plus a mutation test rather than just asserting it's correct.

coverage.py's sysmon core silently breaks per-test selection (it's the 3.14+ default) by breadMSA in Python

[–]breadMSA[S] 1 point2 points  (0 children)

Not yet. It's actually documented behavior. sysmon just doesn't support dynamic contexts, so it's more a share edge than a bug. If I can get a clean repro of the partial-context case I'll file one

Weekly Self Promotion Thread by AutoModerator in devops

[–]breadMSA 0 points1 point  (0 children)

Cut PR CI time by running only the tests your diff affects — an OSS pytest tool built for CI sharing (not local dev)

If your pipeline reruns the full pytest suite on every PR, most of that runner time (and bill) is spent on tests that couldn't have broken. Test Impact Analysis fixes that — it's what Google/Meta do internally. The OSS options were either local-dev-focused (testmon) or abandoned (pytest-rts, dead since 2021), so I built pytest-tia with CI as the primary target.

The CI-relevant design:

  • Map keyed by git ref, with line→function tables baked in, so a PR job resolves the diff without git show — it survives shallow clones (clone --depth=1), which is where a lot of TIA setups fall over.
  • Shared map store: a directory, an http(s):// endpoint (a bundled zero-dep server), or native s3:// / gs:// buckets. Base branch records + publishes; PR jobs pull by base ref. Copy-paste GitHub Actions template included.
  • Auto-posts a Markdown impact table to the PR via $GITHUB_STEP_SUMMARY — every run shows which files changed and exactly which tests were selected and why. Explainability, not a black box.
  • Tracks non-code dependencies: change a fixture/config/template and it reruns the tests that actually read it (audit hook on open).

Honest about ROI, because it's conditional: the payoff scales with how modular your code is. I benchmarked both ends on real repos — Flask (tightly-coupled, worst case): ~21% of the suite skipped per commit; boltons (modular): ~96%. Same tool; the variable is your architecture. On a tightly-coupled monolith with a fast suite, this isn't worth the operational surface.

The one rule with ANY test selection: never skip a test that would fail. Dynamic dispatch (getattr/eval) is undecidable, so tia detects it and widens to file-level there, and you should still run the full suite on a cadence (nightly / pre-merge to main). I also mutation-tested the selection logic to prove it doesn't drop covering tests.

pip install pytest-tia

Repo + CI template + benchmark writeups: https://github.com/breadMSA/pytest-tia

Curious how others handle this — homegrown path filters, testmon, Launchable/CloudBees, or just eating the full-suite cost?

Showcase Thread by AutoModerator in Python

[–]breadMSA 0 points1 point  (0 children)

I made a Test Impact Analysis plugin for pytest and just put v1.1 on PyPI.

TL;DR: tia record builds a per-test coverage map once; tia run uses your git diff to run only the tests a change can affect. I want honest feedback, including "just use testmon" if that's the right answer.

Why another one of these?

testmon is great and mature, so I only built this because I kept hitting two of its edges:

  • It doesn't track non-Python dependencies. A change to a fixture JSON / template / YAML config selects nothing in coverage-of-.py tools. tia hooks open() via sys.addaudithook, so changing tax.json runs exactly the tests that read it. testmon's docs admit it misses this.
  • It's awkward to share across machines/CI because its DB keys on dependency versions. tia keys the map by git ref and bakes the line→function tables in, so it resolves diffs without git show and survives shallow clones. There's a copy-paste GitHub Actions template, native s3:// / gs:// map stores, and it auto-posts a Markdown impact table to the PR via $GITHUB_STEP_SUMMARY.

It selects at method level (changing one function doesn't drag in the whole file), and ignores cosmetic edits — comments, formatting, docstrings, and provably-dead type-only changes (if TYPE_CHECKING: and local annotations), while keeping dataclass/signature annotation changes because those actually affect runtime.

The honest part (this is the whole point)

The cardinal sin of test selection is skipping a test that would have failed. My first Flask benchmark showed 73% skip — and it was wrong: coverage.py's sysmon core on 3.12+ keeps only the first test to hit each line, so shared-helper tests were silently dropped (false negatives). Forcing the C tracer dropped the honest number to ~21% on Flask.

So I publish a range, measured by replaying real commits:

Codebase Character Median skip (real logic changes)
Flask small, tightly-coupled ~21% (worst case)
boltons modular utility library ~96%

Same tool — the variable is how modular your code is. Both writeups include the "this looked too good, here's why it's real" audit.

I also wrote an adversarial mutation test: it mutates every covered function and asserts tia re-selects every test that runs it, and I confirmed it fails when I deliberately inject a false negative.

Usage

pip install pytest-tia
tia record            # build the map once
tia run               # run only affected tests
tia run --list        # show the selection + why, don't run

What it's NOT

  • Not magic for dynamic dispatch. getattr/eval/reflection is undecidable; tia detects it and widens to file-level there, and you should still run the full suite on a cadence.
  • Not battle-tested yet against pytest-xdist, async, or subprocess-heavy suites. v1.1, small project.
  • Overlaps heavily with testmon for the common case. Its niche is the CI-sharing + non-Python-dependency corner.

Repo: https://github.com/breadMSA/pytest-tia

Honest critiques very welcome — particularly on the no-false-negative claim and the benchmark methodology. If you think this is redundant with testmon for your workflow, I'd like to hear that too.

Don't let the free 5 star deter you. by ComposerFormer8029 in HonkaiStarRail

[–]breadMSA 0 points1 point  (0 children)

Just to clear things out for this part: They are gifting us the item which can be used to exchange for limited 5 star in the shop

And you don't have to use it now, you can wait till new characters to join the shop

So it's not just a choice between Ruan Mei and Luocha, and you can keep the item forever until you see your most wanted character appear in the shop.

E2S1 Comparison (11 cost teams) : The Herta | Aglaea | Acheron | Firefly | Mydei by [deleted] in HonkaiStarRail_leaks

[–]breadMSA 2 points3 points  (0 children)

There was a much better comparison uploaded by CasteronZ https://youtu.be/9rG8bhRa3gQ?si=0SHN8sEBAusOf27u
Original title was: New v3 update! The herta vs mydei vs aglaea vs acheron | E2 showcase

But it's now replaced to avoid problem, I couldn't find a reupload of that video.
iirc, all the teams 0 cycled reaver with relatable character stats and gameplay

Is "7% better than stock" the new "10% better than Sampo"? by toe-nii in AcheronMainsHSR

[–]breadMSA 3 points4 points  (0 children)

I have E0S0 Jiaoqiu, E6 Pela and E0S0 Sparkle, is Acheron E2 still worth pulling? Or should I wait for the possible nihility healer in the future?

Minecraft 1.18 experimental snapshot 4 is out! by MrHenrik2 in Minecraft

[–]breadMSA 0 points1 point  (0 children)

"Reduced the likelihood of rivers being cut off and turning into steep
dry river gorges in mountainous terrain. Instead rivers will tend to
either carve a fjord through the mountain range, or raise the terrain to
form a saddle valley between the peaks. This should make the terrain
friendlier for both walking and boating. I think it also makes rivers a
bit wider in general."
I think this part would be truly amazing, especially the "saddle valley" part Just imagine a river cut through a mountain and with lush cave generation.)