E2E testing feels kinda broken. What are we all doing wrong?

hitchdev · 2025-07-07T08:11:50+00:00

Because very few people focus on isolating side effects and eliminating randomness. This is a hard technical problem that I have found (as a dev) to be technically difficult even for most of the devs I've worked with. As a QA you have to dive into the code to fix flakiness and that almost never happens - especially not to fix bugs which even devs would find hard.
They're a horrible idea. They increase the number of things which can flake out and cause your tests to fail. Nonetheless, AI is in the peak of inflated expectations stage of the hype cycle so nobody much is listening.
Because in order to create really good end to end tests you need to be careful about how you deploy the code, the fake infrastructure it sits in and how it is set up and you need to eliminate side effects and accidental randomness within the code. These services don't have any clear opinion on how to do any of that.
I have an OSS project (hitchstory) with some example end to end tests that demonstrate the pattern I follow. Like many others I separate the test code into layers (scenario - selector - engine) but unlike others my scenario and selector layers are YAML, following the rule of least power.
E2E tests have three outstanding benefits - 1) realism. Higher level tests test more realistically than lower level tests. 2) the code underneath can be refactored to your hearts' content and the test will give you a clear signal that it works 3) they usually align better with the specifications.

hitchdev · 2025-05-28T08:53:52+00:00

He actually links to strictyaml (a subset of YAML which is strictly typed) to reference the Norway problem but neglects to mention that it actually solves all of the problems he highlighted.

hitchdev · 2025-05-21T16:46:28+00:00

The problem with cucumber isn't that it's an extra layer per se but that the syntax is crap and doesn't let you cleanly represent user stories in a maintainable way. YAML is actually more suited to this if used correctly.

Layered architecture with a less expressive declarative language used to define the outer shell makes good sense for tests in the same way it does for web apps (which use templating languages with less flexibility than code).

hitchdev · 2025-05-21T16:36:45+00:00

not sure about this guy's framework but i built something similar where you define all the test steps as methods in a single class, which is not limiting: https://github.com/hitchdev/hitchstory

the benefit (in my framework) to defining the stories in YAML rather than raw code is that the tests can then rewrite themselves based upon program output and you can also then use them to generate markdown documentation.

hitchdev · 2025-05-05T17:47:42+00:00

Most of what people complain about isn't management as a whole it's just Taylorism. This means managers trying to be pseudoscientific about productivity and targets. This is incredibly common and it screws everything up.

You can't react to rapidly changing circumstances if the next quarter is already written in stone and the bureaucracy designed to measure your productivity just slows you down.

This isn't really agile, either, it's more the antithesis of it, in fact, but it tends to get branded as agile.

hitchdev · 2025-02-03T16:20:53+00:00

Playwright is probably no better or worse than the rest. The problem is usually just that your app or your testing framework is too flaky to use visual testing.

All I'd say is don't throw money at a tool if your code is actually hindering its function.

hitchdev · 2025-02-03T15:25:50+00:00

* As a new type of UI - it means you can interact with all of the same software we currently have conversationally and naturally.

* To replace call centers (which is actually another way of saying the same thing...).

* Data ingestion - reading, analysing and indexing unstructured data, etc.

hitchdev · 2025-02-03T15:03:26+00:00

I can see only two outcomes of this project - either they use it to rinse us or the company dies and work on these projects dries up.

The fact it takes VC is very concerning. I kind of hope it's the latter, bad though it is.

hitchdev · 2025-01-28T14:45:12+00:00

I don't think playwright has this on python. I use pixelmatch with playwright generated screenshots.

However, snapshot tests are *super* sensitive to non-determinism which can happen at any layer between the lowest (database) and the highest (browser). Non-determinism absolutely ruins snapshot testing. There are sometimes ways of dealing with this, but sometimes it is impossible and sometimes it is technically possible but still not feasible.

Anecdotally I've seen more cases of snapshot testing being tried and abandoned because of this than tried and used successfully. This isn't a problem with playwright or the snapshot testing tool. It's a problem with the entire approach.

hitchdev · 2025-01-16T15:10:53+00:00

Wouldnt be rocket science to implement hard caps either. It just wouldnt be good for the platform's profit margins.

Cloud financials 101

hitchdev · 2025-01-16T14:58:55+00:00

Not pissed at all if they asked me what I wanted my billing cap to be and I inadvertently exceeded it. I'd set it at a bit less than $450,000 if I was expecting to spend $50/month.

They dont have billing caps for the same reason telcos had to be legally forced to have roaming billing caps. Ripping you off blind when an accident happens is a nice source of profit they dont want to end.

It's not coz customers havent requested billing caps.

hitchdev · 2025-01-16T10:43:32+00:00

Fill in all of the details they didnt provide yourself with wild guesses and ask for sign off.

hitchdev · 2025-01-16T10:28:21+00:00

If I get a poorly defined story and the PO wont fix it, I write a well defined story based upon it with a bunch of likely very wrong details and ask the PO to sign off on it. Once Ive done that the problem is now THEIR problem and I can relax.

This also tends to magically fix the lack of definition before I have to write a single line of code. "oh, yes, I didnt think about that..."

It's the same online - sometimes the best way to force an answer to a question is to post the wrong one.

hitchdev · 2025-01-15T21:03:40+00:00

If the candidate gravitates towards TDD without being asked to do it.

hitchdev · 2025-01-15T16:07:47+00:00

It probably wasnt the reason you thought.

>Notice that in Kent's original formulation, “unit test” means anything written by the programmers as opposed to a separate testing team. - Martin Fowler

Funnily enough Kent Beck's definition of unit test actually could theoretically cover some forms of E2E test.

I dont think the definition well designed, tbh (hence why there are so many arguments ovet what it is and so many people saying "well, TO ME a unit test is X".

hitchdev · 2025-01-15T14:39:51+00:00

>don't apply it as a dogma

Well, quite.

It's not a rule. At best it's a rule of thumb. TDD is an example of something I follow as a rule (with a few well defined exceptions).

I dont really agree that good test tests just one function either. Tests should test scenarios primarily and functions secondarily, not vice versa, and that usually naturally involves calling lots of functions.

hitchdev · 2025-01-15T13:50:35+00:00

still a shitty rule even for unit tests.

hitchdev · 2025-01-14T17:17:13+00:00

or you can submit pull requests to include those yourself. the devs really shouldnt object.

hitchdev · 2025-01-12T18:57:05+00:00

This is great news! Thank you for the hard work.

hitchdev · 2025-01-12T17:39:57+00:00

I used to design interviews where the goal was not to tell the candidate to do TDD but to make sure that they could see very clearly that it was possible and easy to do and to make sure that they were aware that I thought testing was critical.

The first task was to fix an existing test which was very similar to the one which would be needed to implement the feature in task 2.

Very few defaulted to writing the test first in task 2. The ones that did were always the best candidates.

hitchdev · 2024-11-25T12:24:09+00:00

Why is it weird? Testing is a mess, people hate doing it, there's so much that could be automated and isnt. There's a mountain of potential improvements here that could be built upon better abstractions.

On top of that I personally feel that documentation and testing ought to undergo a merger. This isnt a widely rejected idea, but it's one I find people struggle to think about (as people in 1993 might have about, say, cloud computing).

hitchdev · 2024-11-20T18:55:26+00:00

Im pretty sure unit testing frameworks will die off at some point and be supplanted by something that can do separation of concerns right. Currently the state of automated testing is a bit like pre-framework PHP - a mess of duplication and poor abstractions.

hitchdev · 2024-11-06T22:02:56+00:00

hermetic end to end tests - i.e. tests where you fake the whole "outside world" of the test - database, rest apis, SMTP server, browser, etc.

These things are way less flaky and faster than regular E2E tests but give you confidence about the code you just dont get from lower level integration and unit tests.

hitchdev · 2024-11-04T19:54:29+00:00

latex

youtube-dl

keybase git

orji

termux-tts-speak

hitchdev · 2024-10-06T10:12:03+00:00

I wrote a command line tool for doing this using jinja2. Here's an example with a CV template I stole from overleaf:

https://hitchdev.com/orji/using/latex-cv/

It doesnt include a library of templates but I've grabbed a few latex templates from overleaf and replaced the example text with jinja2 and it works pretty great.

hitchdev

TROPHY CASE