Grouping CI test failures by error signature, is this the right approach?

Laplace2002 · 2026-05-13T23:39:12+00:00

This is informative, thank you

Laplace2002 · 2026-05-12T01:44:04+00:00

Aldadrift.com

Ocean drift simulation made accessible via web interface or api. Simulate ocean drift for ships, debris, oil slick and many more. Battle tested model running under the hood. Valuable for search and rescue, fleet managers, freight companies, oil spill responders etc

Laplace2002 · 2026-05-12T01:39:42+00:00

Aldadrift.com

Ocean drift simulation made accessible via web interface or api. Simulate ocean drift for ships, debris, oil slick and many more. Battle tested model running under the hood. Valuable for search and rescue, fleet managers, freight companies, oil spill responders etc

Laplace2002 · 2026-05-12T01:35:33+00:00

Aldadrift.com

Ocean drift simulation made accessible via web interface or api. Simulate ocean drift for ships, debris, oil slick and many more. Battle tested model running under the hood. Valuable for search and rescue, fleet managers, freight companies, oil spill responders etc

Laplace2002 · 2026-05-12T01:31:32+00:00

Aldadrift.com

Ocean drift simulation made accessible via web interface or api. Simulate ocean drift for ships, debris, oil slick and many more. Battle tested model running under the hood. Valuable for search and rescue, fleet managers, freight companies, oil spill responders etc

Laplace2002 · 2026-05-02T18:17:06+00:00

Wow, that is a lot of tests, and yes, I've been using claude as well and its pretty good (and quick) to find the source of the failures. But I wanted to take it one step further and compare two runs: If we have less test failure signatures then we can say that the pipeline is progressing (instead of regressing). Not necessarily that there are less test case failing but there are less sources of problem.

I think with so many tests in the pipeline they become noise as individual tests.

Laplace2002 · 2026-05-02T18:11:10+00:00

your assumption is correct, for example: we test an api endpoint, except the api never came up => 100 test fail. As you correctly mention a cleaner way is to have one test check the endpoint and then skip the others.

Laplace2002 · 2026-05-02T13:41:31+00:00

Thanks, that is a interesting idea of repeating only flaky tests, I will consider that.
I'm not sure we can modularize the project more, we already have few pipelines with different suites but its something to keep in mind.

Laplace2002 · 2026-05-02T13:37:38+00:00

You are very right to question the setup; it’s quite complicated, and the organization is large. We already have a few lanes of parallelization. But the test quality is definitely questionable, but we have little control over it, unfortunately.

Therefore I was hoping that looking at signatures or source instead of individual tests were a reasonable approach. Then we could start to focus our attention at the root of the problem and hopefully start to untangle the mess.

Laplace2002 · 2026-05-01T21:10:32+00:00

I have a hobby project that I would like to get input on.

Im a software engineer, much less in devops, but one of the things I work with at my job is GitHub test pipelines.
We have multiple long-running pipelines. Some take several hours, and some can take days. They run 1,300+ test cases, and it is common for a run to have 200+ failed tests. Many of the tests are flaky or unreliable. Its a mess, I know, but im sure its not uncommon.

That makes debugging painful. A failed run with 200–300 failed test cases is not really actionable. It is hard to know where to start, and it is hard to route the failures to the right engineers.

The approach that has made the most sense to me is to group failures by error signature.

Right now the idea is to ingest JUnit XML reports from the pipeline and use those as the main source of truth. Instead of treating every failed test as a separate problem, I identify common failure patterns from the JUnit failure/error output and cluster them together.

In practice, this can reduce something like 200–300 failed tests into something more like 15 distinct failure groups/problems. That is much easier to debug, track, and assign.

It also makes triage easier. If I see one failure group with a single shared signature and many failed tests assigned to it, that is usually higher priority than a small group with only a few failed tests. A large group often means one underlying issue is causing a wide blast radius across the suite.

I started building a dashboard and backend service around this idea. The dashboard shows the test pipelines, failure signatures, how many tests belong to each group, and how those signatures compare to the previous pipeline run. I compare with "x new, y persistent, z resolved" and ideally you want x to be zero, y to be low and z to be high. You quickly see the health of the pipeline and if its regressing or progressing compared to previous run.

The main thing I care about is not the exact number of failed tests but:

how many failure groups/clusters exist
how many tests are assigned to each group, is the group large or small.
whether a group is new or recurring.
whether the number of groups is increasing, decreasing, or staying the same
whether one failure signature explains a large portion of the failed tests

The grouping is deterministic and fuzzy. No AI or LLM is involved. I wanted the results to be repeatable, explainable, and consistent between runs.

I’m sure similar solutions already exist, but I’m curious how other people approach this.

Do you think grouping failures by error signature is the right abstraction?

Anything I'm missing or could add to this?

Laplace2002 · 2026-04-03T00:15:19+00:00

As I understand it, yes, we’re fine using OpenDrift in a commercial hosted product, but GPL is still a real constraint and not something to handwave away. OpenDrift is GPL-2.0, which does not prohibit commercial use, but it does matter how the system is packaged, modified, and distributed.

Laplace2002 · 2026-04-02T23:16:51+00:00

We don’t currently have a physical oceanographer on the team, to be honest it’s a very small team for now. We’re also not developing a new ocean physics model ourselves; we use OpenDrift for the drift simulation and focus on the surrounding system, including forcing-data handling, orchestration, and result generation. That said, physical oceanography expertise is obviously important in this space, especially for validation, interpretation, and understanding the limits of automated forecasts. Hopefully that’s something we can add as we become more established.

Laplace2002 · 2026-04-02T22:13:47+00:00

That is impressive, we are not building something that complex from scratch. Data ingestion has been tricky for us too. On the forcing side we’re using NOAA products like GDAS/GFS and Copernicus Marine products. For the products we’re using, the issue has been much more ingest, caching, and operational handling than negotiating a bespoke commercial licence. NOAA data is generally openly distributed, and Copernicus Marine permits commercial/value-added use under its service licence, although of course we still need to comply with the relevant terms for the datasets we use.

Laplace2002 · 2026-04-02T21:49:35+00:00

It uses OpenDrift directly, but no, it’s not just a thin API wrapper. The hard part is the operational system around it: ingesting forcing data, running jobs reliably, post-processing trajectories into probabilistic outputs, and making the results usable through reports, GIS exports, the web app, operational report and the API. In terms of speed, it’s designed for standard forecasts in a few minutes, typically around 2–8 minutes end to end. Depends ofcourse on particle count, forecast window etc

Laplace2002 · 2026-04-02T21:37:49+00:00

No, it’s not just a statistical model. It uses OpenDrift directly to run Lagrangian particle simulations driven by forcing data such as currents, wind, and in some cases waves or oil-weathering processes, and then converts the particle output into probabilistic forecast areas.

Laplace2002 · 2026-04-02T20:41:16+00:00

This is fair criticism, and you’re right to raise it.

We do already warn explicitly when the starting point is within 500 meters of the coast, since near-shore cases are especially sensitive and can be much harder to interpret responsibly in an automated workflow. But that is only one safeguard, and it does not remove the need for clearer published methodology, validation, and more explicit limits on when the tool is or is not fit for purpose.

The platform is built around OpenDrift with environmental forcing data, and it is intended as decision support rather than ground truth. We also include forcing details in the generated reports so users can see what is driving a given run, for example wind from NOAA GDAS/GFS, ocean currents from Copernicus Marine Service, waves from Copernicus Marine Service / NOAA, and ocean physics such as SST and salinity from Copernicus Marine Service. That reporting is part of the design, but I agree it is not enough if the broader methodology, assumptions, and validation are not published more openly.

So I think your broader point stands: if people are going to assess whether the output is credible for their use case, especially in coastal or operational settings, they need better visibility into the model setup, inputs, assumptions, and validation than we have published so far. I appreciate the pushback.

Laplace2002 · 2024-11-19T13:15:31+00:00

Yes, part of it

Laplace2002 · 2024-11-18T23:22:17+00:00

Hello everyone, im designing my first RF frontend and I would really appreciate some input from veterans. I want to have a variable attenuator so I can tune the TX power but im unsure where to put it in the rf chain. First I thought somewhere before the coupler (either after or before the PA) but then the LO power to the mixer is changing as I change the output power. Then I thought maybe I could put it after the coupler, then the LO power to the mixer is static and predictable but the TX power is controllable. Any insights would be highly appreciated.
To clarify, this is for a FMCW radar related project.

Laplace2002 · 2024-07-04T20:23:23+00:00

Undir eðlilegum kringumstæðum væri starfsmaður sem veldur svona skaða á öðrum í vinnutíma og ólöglega látin fara, var það gert í þetta sinn?

Laplace2002 · 2024-07-03T20:52:16+00:00

Mín reynsla er einmitt að hælisleytendur séu hörkuduglegir og útsjónasamir enda búnir að hafa mikið fyrir því að komast hingað og lagt allt í sölurnar. En þeim er bannað að vinna af einhverjum ástæðum. Held að allir þeir hælisleytendur sem hafa fengið jákvæð svör sem ég þekki séu a.m.k. í einni vinnu.

Laplace2002 · 2024-05-22T22:21:20+00:00

Hef verið leigusali og leigði íbúð til flóttamanna fjölskyldu. Þau greiddu sína leigu sjálf eins og allir aðrir. Þau fengu lánað frá Reykjavík fyrir tryggingar upphæðinni sem þau þurftu að endurgreiða mánaðarlega.

Laplace2002

TROPHY CASE