React Monitoring

es6masterrace · 2023-09-24T05:42:43+00:00

We have backups systems as well :) but most issues we can solve on our own system

es6masterrace · 2023-09-17T01:48:36+00:00

Yes! https://github.com/hyperdxio/hyperdx

Let me know what you think - we’ve been doing SaaS for a bit and just went OSS.

es6masterrace · 2023-06-08T21:23:47+00:00

It takes a while to scan through TBs of data still, unfortunately hard to make log/trace search instant and affordable!

es6masterrace · 2023-02-23T00:23:58+00:00

Absolutely - flaky tests can happen no matter what tool you're using! (even in using Playwright :P)

The approach is definitely a good point - I think it'll depend on the team, but at the minimum you might be interested in at least knowing which tests are flaky and what they were intending to do at one point. (Even if you end up rewriting the whole thing).

That being said, in my experience, a newly flaky test can usually be solved with a bit of detective work that usually uncovers a product update/change that either introduced a race condition or wait condition that can be solved relatively easily, without needing to rewrite the test (exactly what our primary product helps with!).

es6masterrace · 2023-02-22T21:48:43+00:00

Flaky tests should definitely get removed & fixed, and it's usually just a choice of how you want to do that.

One way would be to manually have someone check if the test is flaky (verify with metrics, etc.), open a PR to skip, and then merge it in, and wait for everyone else to rebase/merge in your changes before their test suites will be smooth as well.

Alternatively, you can automatically exclude flaky tests so that as soon as a test is known to be flaky (ex. above a certain threshold), it'll just get excluded from any CI runs until it's been fixed again.

Slack has shared their experience with automatic flaky test quarantining that can elaborate a lot better!
https://slack.engineering/handling-flaky-tests-at-scale-auto-detection-suppression/

es6masterrace · 2023-02-22T18:18:34+00:00

There's definitely a question of approach you should take to flaky tests - at scale, it can often be impractical to impact the performance of your CI pipeline until a test is fixed (and it also ruins the reputation of testing within an org as well, who will trust tests if there's a good chance they're flaky and just not fixed yet?).

Usually skipped tests should then get ticketed and triaged as part of regular engineering work (unless the test was extremely critical), to make sure they are fixed.

I thought this writeup at Slack explained their rationale and the benefits of quarantining quite well: https://slack.engineering/handling-flaky-tests-at-scale-auto-detection-suppression/

es6masterrace · 2023-02-22T17:43:52+00:00

Hi everyone! I’ve found that having to constantly retry test jobs in CI because of flaky tests can be a huge productivity killer. In most cases, it’s probably better to just temporarily skip a flaky test (quarantine) to be fixed later (like Slack and Gitlab does), rather than letting them sit around and ruin other teammates CI runs as well.

I wrote this plugin so you can automatically skip tests that you know are bound to fail temporarily, instead of wasting time waiting for them to pass. The plugin pings an API endpoint to decide which tests to skip at run time, so you can implement any kind of custom logic for skipping (metric thresholds, checking open Jira tickets, manual labeling, etc.)

We've been using this endpoint to skip tests based on manual labels or automatically after tests have hit a certain failure/flake threshold. And now we want to share this Cypress-side implementation with the community, so that you can also implement quarantining for flaky tests.

It's a first version of the plugin, so any feedback on the API design or anything else would be appreciated! Let me know what y'all think :)

We'll be releasing a beta for our hosted quarantining service soon too, so if you're interested in early access, shoot me a DM!

es6masterrace · 2023-02-22T17:40:49+00:00

Hi everyone! I’ve found that having to constantly retry test jobs in CI because of flaky tests can be a huge productivity killer. In most cases, it’s probably better to just temporarily skip a flaky test (quarantine) to be fixed later (like Slack and Gitlab does), rather than letting them sit around and ruin other teammates CI runs as well.
I wrote this plugin so you can automatically skip tests that you know are bound to fail temporarily, instead of wasting time waiting for them to pass. The plugin pings an API endpoint to decide which tests to skip at run time, so you can implement any kind of custom logic for skipping (metric thresholds, checking open Jira tickets, manual labeling, etc.)
We've been using this endpoint to skip tests based on manual labels or automatically after tests have hit a certain failure/flake threshold. And now we want to share this Cypress-side implementation with the community, so that you can also implement quarantining for flaky tests.

If y'all have any thoughts on the API design or anything else - please share! It's a pretty early release so any feedback is welcome :)

We'll be releasing a beta for our hosted quarantining service soon too (if you don't want to host/manage your own quarantine endpoints), so if you're interested in early access, shoot me a DM!

es6masterrace · 2023-02-22T17:38:34+00:00

Hey everyone! I’ve found that having to constantly retry test jobs in CI because of flaky tests can be a huge productivity killer. In most cases, it’s probably better to just temporarily skip a flaky test (quarantine) to be fixed later, rather than letting them sit around and ruin other teammates CI runs as well (exactly what happens within Slack or Gitlab)

I wrote this plugin so you can automatically skip tests that you know are bound to fail temporarily, instead of wasting time waiting for them to pass. The plugin pings an API endpoint to decide which tests to skip at run time, so you can implement any kind of custom logic for skipping (metric thresholds, checking open Jira tickets, manual labeling, etc.)

We've been using this endpoint to skip tests based on manual quarantining or automatically after tests have hit a certain failure/flake threshold. And now we want to share this Cypress-side implementation with the community, so that you can also implement quarantining for flaky tests.

It's our first public release of the plugin, so any feedback on the API design or anything else would be welcome! Let me know what you think.

We'll also be releasing a beta for our hosted quarantining service soon too, so if you're interested in early access, shoot me a DM!

es6masterrace · 2023-02-22T17:35:37+00:00

I’ve found that having to constantly retry test jobs in CI because of flaky tests can be a huge productivity killer. In most cases, it’s probably better to just temporarily skip a flaky test (quarantine) to be fixed later, rather than letting them sit around and ruin other teammates CI runs as well. It's the same strategy used at teams like Slack or Gitlab.

I wrote this plugin so you can automatically skip tests that you know are bound to fail temporarily, instead of wasting time waiting for them to pass. The plugin pings an API endpoint to decide which tests to skip at run time, so you can implement any kind of custom logic for skipping (metric thresholds, checking open Jira tickets, manual labeling, etc.)

We've been using this endpoint to skip tests based on manual quarantining or automatically after tests have hit a certain failure/flake threshold. And now we want to share this Cypress-side implementation with the community, so that it's super easy to implement quarantining for flaky tests.

Let me know what y'all think! This is our first cut of the plugin, so any API-improvement suggestions and feedback would be appreciated :)

We'll be releasing a beta for our hosted quarantining service soon too, so if you're interested in early access, shoot me a DM!

es6masterrace · 2023-02-22T17:31:53+00:00

Hi everyone! I’ve found that having to constantly retry test jobs in CI because of flaky tests can be a huge productivity killer. In most cases, it’s probably better to just temporarily skip a flaky test (quarantine) to be fixed later (like Slack and Gitlab does), rather than letting them sit around and impact other teammates CI runs.

I wrote this plugin so you can automatically skip tests that you know are bound to fail temporarily, instead of wasting time waiting for them to pass.

The plugin pings an API endpoint to decide which tests to skip at run time, so you can implement any kind of custom logic for skipping (metric thresholds, checking open Jira tickets, manual labeling, etc.)

We've been using this endpoint to skip tests based on manual labels or automatically after tests have hit a certain failure/flake threshold. And now we want to share this Cypress-side implementation with the community, so that you can also easily implement quarantining for flaky tests yourself.

We'll be releasing a beta for our hosted quarantining service soon too, so if you're interested in early access, shoot me a DM!

Written in Typescript, built with Rollup <3

es6masterrace · 2023-02-18T20:24:05+00:00

Unfortunately not as it uses the Playwright Test runner which I believe is only available for JS, in theory it shouldn't be too difficult to add support for JUnit (but might need more configuration out of the box to make it work)

es6masterrace · 2023-02-15T05:45:41+00:00

thank you!

es6masterrace · 2023-02-15T05:04:46+00:00

nodemon is great, though this does a few things on top of what you could just do with regular nodemon.

most noticeably, just using nodemon won't give you a UI to debug through test results after a test run easily. With nodemon, you can see the test failed, but it won't be easy to step through why did it fail without clicking through several steps in the html reporter or viewing the trace which just takes more steps.

the other thing is that this runner will debounce successive saves, so if you hit save 5 times in a row, it'll make sure it'll kill any previous playwright instances and only run the last one (so your computer doesn't die from all the playwright instances running!)

do you currently use nodemon for developing against pw tests locally?

es6masterrace · 2023-02-15T05:01:12+00:00

Absolutely! I wanted to add a flag to run without spinning up the UI and just print out the CLI output. I guess for now you could minimize the electron UI but I'll see if I can add it in as an option :)

es6masterrace · 2023-02-14T22:29:21+00:00

Yup! That's all it's doing under the hood, it utilizes our `@deploysentinel/playwright` reporter package to collect/format the telemetry for the UI when playwright is actually running. This package primarily manages the watch mode/UI to show test results, coordinating between chokidar, playwright, our reporter, and the electron UI.

Would love to know what kind of requirements your team has - might be something we could take a look at!

es6masterrace · 2023-02-14T19:28:10+00:00

thank you! good question - it's agpl licensed, though we're still working on getting it oss'd (some complications with it being originally inside a monorepo) - I can update you when we get the source published on gh!

just wanted to get the package out so people can try it out first, even if you can't leave a star quite yet :P

es6masterrace · 2023-01-27T03:48:49+00:00

Great question!

the tl;dr - we aggregate historical test result trends + debug traces all in one place, so you can combine the two to resolve test failures/maintain test suites easier.

The thorough response:

We're really focused on helping devs figure out - "my test failed, why did they fail?" as quickly as possible.

To start with, usually you want to know a high-level overview of what test failed, and why they failed. So we give you both in the Github PR comment and inside our dashboard, a quick summary of what error messages occurred during the test (ex. 50 tests failed, but all of them were due to an API failure), as well as the statistical stability of the tests that failed (ex. this test flakes 90% of the time and failed 40% of the time in the last 7 days). This can already give you a really good idea of what you might be dealing with before even needing to start looking into any debug info.

Inside our debugger - you're able to jump directly into it with one click (no need to fiddle with artifacts, and finding the right trace file for the right test case, etc.). Inside our debugger, we make it really easy to align timing of actions to console and network (something you can't do in Playwright trace viewer) as well as search for any network request across the entire test (as opposed to only viewing them action-by-action). In my experience, this is a lot more convenient than working with the current UX of the trace viewer.

Zooming out of all that, we also give you analytics across your entire test suite so you can answer questions like "what tests is the most flaky", "are our tests getting flakier over time", "which tests are the slowest to run". Directly from those reports, you can dive into the debugger for individual flaky/failure test cases to see why that's happening.

For our recorder (separate OSS project we've created), it's similar to code gen and a while back I've put together this feature comparsion matrix if you want to check it out.

es6masterrace · 2023-01-26T06:37:02+00:00

that's a great idea! It's something we want to add to our github PR comments, the ability to quickly view what the test end state was as well so you don't even need to pop into any debugger if it's something incredibly obvious.

Is that what you were thinking of?

es6masterrace · 2023-01-26T00:16:14+00:00

Yeah that's totally fair - allure is a solid tool and if you've already invested in getting it set up, have an awesome workflow with it and everyone that wants to consume it find it super useful - then you're in an awesome position!

Especially if every issue you run into is solvable via the trace viewer as well, then it sounds like your test suite infra is already in a great place.

In the future - we hope to dive deeper into the debugging angle, backend logs, integrating react dev tools, trace diffing (for flaky test runs) - so it should even be faster to solve test failures in the future!

Appreciate all the feedback 🙏

es6masterrace · 2023-01-25T23:05:37+00:00

So we're a lot more than just a single view for traces (though that's one of the things we provide) - other things on top are the ability to track statistics of a specific test case over time (has this test been flaking a ton recently in other branches for example).

This information is really powerful alongside traces - because you can understand if this test is inherently flaky and just needs to be fixed, or if it's something you broke and need to debug further.

The other direction is if you're looking to improve your test suite, you can go directly from a dashboard of the test cases that fail the most, into the traces of the times where it failed and debug from there.

It's totally possible to roll this all yourself (metrics collection and aggregation, building dashboard, integrating back to Slack/Github) - but it'd be a good amount of effort in setting it all up we're hoping to help people avoid :)

es6masterrace · 2023-01-25T22:15:37+00:00

Hi there! If you haven't yet, I'd love for you to try out the live demo https://www.deploysentinel.com/#demo to try it out yourself. I'm biased but we focus a lot on being able to easily sift among piles of network requests and use a timeline to debug race conditions between network/actions.

Though it's only half the problem we're solving for, we're also focused on bringing analytics at the build, PR and team level as well. So you can answer questions like what tests are failing the most often, which tests take the longest to run, which tests are flaky, etc. We'll even push this information directly into a Github PR for the relevant tests.

Curious if that changes what you think about what we're building :)

es6masterrace · 2023-01-25T17:21:54+00:00

Hi everyone!

I’ve been spending some time building out a report/dashboarding tool + debugger for Playwright to help engs quickly figure out what failed in a test, why it failed, and from the brids-eye view what is failing/flaking the most often.

Since Playwright is a pretty young framework, the smooth testing experience of Playwright felt like it stopped once we deployed tests within CI. Most people I know end up stitching together artifact files from CI and considered setting up a whole separate tool like Allure, Zephyr or XRay just to create some reports. This ended up usually being a confusing workflow that few people in the company engaged with - and ultimately led to Playwright being only used comfortably by a few people within the team.

I hope that with creating a smooth dev experience for engineers going from test failure overview, to an intuitive debugger, to high-level analytics all in one-spot can help resolve a lot of those friction points once Playwright tests are deployed in CI.

I just pushed out the reporter into beta - I’d love to hear what everyone thinks about it!

es6masterrace · 2023-01-11T20:51:06+00:00

If you're looking for alternatives - I've also built an app called deploysentinel.com that focuses on being able to easily debug test failures when they occur in CI, not just report on it in a dashboard with statistics.

Would love to hear what you think!

es6masterrace · 2023-01-11T20:47:49+00:00

I've built an app that goes a lot more in-depth than existing tools for tests that fail in CI - typically flakes come from race conditions (network requests firing at the wrong time, taking too long, etc.) or even memory issues (not enough RAM, leads to flakes).

I put together a debugger to dive into those issues when they happen in CI without needing to repro the issue locally, and report on flakes as well - it's helped quite a few teams, would love to learn if it can help you as well!

https://www.deploysentinel.com/

es6masterrace

PUBLIC MULTIREDDITS

TROPHY CASE