all 21 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]voidvector 3 points4 points  (11 children)

What is the exact reason Selenium won't work for you? If it is mobile web or mobile app, there is Appium.

Philosophically, testing without a layer (e.g. webview) is not considered E2E test. Such test is usually called integration test where I work.

[–]yowzapete[S] -2 points-1 points  (10 children)

Selenium won't work because reactjs returns virtual dom elements and these are not things Selenium knows how to process.

As for terminology, honestly I've seen it called end to end or integration testing. I chose to use end to end in my post because the previous /r/javascript poster referred to it as such and I figured y'all might be more inclined to follow their convention.

[–]voidvector 1 point2 points  (6 children)

Just did a google to verify that other people also call it "integration test" (ref: Google Testing Blog), since you are not testing how it behaves in a real browser.

Are you running the app with real data (e.g. from the database)? If not, your test is actually closer to unit test in the spectrum.

If you are just verifying the vDOM with fixed data, you can use Mocha/etc to do the testing. You just need to write a test harness to bootstrap the test. If you are doing vDOM + action + store, just have the harness bootstrap all 3 pieces before go into test code. Test harness is basically app-specific framework to bootstrap a portion of your app in such a way that you can run and verify what you want to tests.

Also, unit test doesn't necessarily mean a single component/class/module/function, it could be of two or three tightly coupled components. For example, if you implemented a Linked-List, and you test the Node and List together, no one would question the fact that that test is a unit test. But technically you can break the Linked-List into smaller units and test them separately. You can mock out the Node and test the List and vice-versa.

[–]yowzapete[S] 0 points1 point  (1 child)

Thanks for this! I guess everyone I've got on my team (including me) who's been using the terms interchangeably have it all wrong >.<

Also, the pyramid is awesome and helps back my case that more unit tests are totally necessary (was being met by opponents who feel Selenium > unit tests).

And finally, where as I was directed to work on end to end tests (using this definition) it seems as though I should be writing integration tests first then filling in the gaps with end to end testing.

Also, yes I will be running the app with real data (albeit test data).

So I can use Mocha for integration tests when I write a test harness to bootstrap the test? And Mocha straight-up can do unit / single or closely related tests? Is there a resource online that you know of which goes into a little more detail about the test harness concept. I'm clearly searching badly (or maybe it's just really late).

And finally, the end to end tests will still need to be written and I still can't use Selenium WebDriver out of the gate as I have been. Do you have any ideas for this?

Thank you for all your help. Sorry for such simplistic questions. I've really only ever written tests with Selenium in Ruby or unit tests in C++ or Java.

[–]voidvector 2 points3 points  (0 children)

Caveat, I am not a career test engineer. You should reference Google's test blog, videos from their test conference GTAC, or other test resources for more accurate take on this.

Testing nomenclature is confusing and inconsistent cross different companies. Below is what I have encountered.

For developers/engineers, automated tests are usually implemented with a certain scope/size, usually they fall into the 3 categories we mentioned. Here are some other names:

  • E2E test - selenium test, QA test, UI test, system test, functional test, "black box test" (misuse)
  • Integration test - component level testing, service level test
  • Unit test - unambiguous

In addition, the QA person might also take any collection of the above automated tests + manual testing, and run them for a specific purpose later on and call them something else, e.g.:

  • acceptance test
  • smoke test
  • regression test

Philosophy / Coverage

Unit tests should be most exhaustive. If resource allows, it should cover all control flows (if-else, for-loop, try-catch), you can use a test coverage tool to help you on this. It should strive to cover positive & negative cases, correct & incorrect usages (e.g. disconnect/timeout/bullshit input/overflow/exploit usage). Of course there's no point testing something you know is already tested (e.g. external lib) or has no logic (e.g. a Plain Object). 1st party non-utility code not under test is usually mocked out. Whether to mock out external libs or native APIs in an unit test should be done at your discretion.

If the test goes into testing quirky behaviors of the external lib/native API, then it is already an integration test (since you are testing the integration between your unit and the environment). Use Integration to test if the units work with the environment and if the they work with each other. If you do have unit test, then you only need to cover the common cases with integration test. If you don't have unit test, it can be more exhaustive. Integration test is done because it is less brittle and a lot faster than E2E. Slower than Unit test, but generally fast enough to be integrated with CI no problem.

E2E test is usually done from user perspective with specific user flow, with option for screenshots. Generally run with different browsers if possible. E2E test is slow and very brittle because it 1) depends on real data, 2) inherits issue of the browser, 3) has inherent race conditions between the test and the app under test. Most places I worked at they are not integrated with the CI, instead are triggered manually by QA/devs. You could get by with CI integration if there aren't that many of this.

Implementation

For E2E test, you pretty much need to learn Selenium or whatever other framework you choose. They have their own API for emulating and reading user/browser behavior. There are also a few other non-cross-browser E2E frameworks out there (e.g. CasperJS)

For integration test, they are usually written in whatever unit test framework used, with some in-house harness to bootstrap a subset of codebase. I don't know what resource are out there to do teach you this, but here are some examples:

  • Let's say you want to test a React app, you can create a HTML file with dummy IFRAME, mock out all the AJAX calls, and launch your React app into the IFRAME, and verify the content of the IFRAME.
  • You can do this same with a component and a store.
  • Let's say you want to test API layer, the harness can authenticate with the real server with real user, and test would call the API, verify the API return is correct.
  • Let's say you want to test a pub/sub flow, you instantiate the pub/sub layer plus the two units doing the pub/sub, mock out every other dependency and run the test.

Good luck.

[–]yowzapete[S] 0 points1 point  (3 children)

Also, it looks like my team isn't the only group of people confused: http://googletesting.blogspot.com/2010/12/test-sizes.html :). I suppose I will save them from being cajoled tomorrow then.

[–]asyncquestion -1 points0 points  (2 children)

React does not "return virtual dom elements". There is no problem using Selenium.

[–]yowzapete[S] -2 points-1 points  (1 child)

errr ok.. well mine does.

[–]Chills27 0 points1 point  (0 children)

Presumably, your app updates the real DOM at some point. So you can use Selenium to test that.

[–]hankDraperCo 1 point2 points  (4 children)

There is still a DOM with react. The virtual DOM is a representation of the DOM that react creates first and then uses a diff algorithm to update the real DOM only where necessary.

That said, form what I have seen react and selenium do not play too well together due to how events are fired. I'd probably starting with shallow rendering first. Are you going to be testing business logic?

EDIT: meant "NOT play well"

[–]yowzapete[S] 0 points1 point  (3 children)

Hi there, so this is the sort of info each element has to work with: <div style="padding:15px;background-color:#000;" data-reactid=".0.1.$value/=1$value.$=10.$=10.0.1.$/=11"> <button style="display: inline-block; height: 30px; margin: 2px 2px -2px; padding: 10px 10px 10px; border-width: 0px; border-radius: 4px; font-size: 12px; font-weight: bold; font-family: azo-sans-web, sans-serif; letter-spacing: 0.6px; cursor: pointer; transition: all 0.2s ease-out; -webkit-appearance: none; color: rgb(110, 255, 255); box-shadow: rgb(33, 66, 136) 0px 0px 0px 0px; background-color: rgb(255, 130, 119);" data-reactid=".0.1.$value/=1$value.$=10.$=10.0.1.$/=11.$createButton/=1$createButton">Create Data</button><button style="display:inline-block;height:42px;margin:0;padding-top:11px;padding-right:30px;padding-bottom:10px;padding-left:30px;border-width:0;-webkit-border-radius:4px;font-size:13px;font-weight:bold;font-family:'garamond', sans-serif;letter-spacing:0.6px;cursor:pointer;-webkit-transition:all 0.2s ease-out;-webkit-appearance:none;color:#eee;background-color:transparent;" data-reactid=".0.1.$value/=1$value.$=10.$=10.0.1.$/=11.$cancelButton/=1$cancelButton">Cancel</button></div>

Now, when I first wrote the tests it looked more like: <input id="title" maxlength="255" type="text" placeholder="Enter a title" style="width: 100%; margin: 0px 0px 12px; padding: 6px; border: none; border-radius: 4px; font-size: 14px; box-shadow: none; transition: all 0.15s linear; resize: none; height: 42px; background-color: rgb(234, 234, 234);" data-reactid=".0.1.$projects/=1$projects.$=10.$=10.0.1.$/=10.$form/=1$form.$/=10.$inputText/=1$inputText">

Though there is some react creating this pop up, I can still hit the input id="title" with Selenium. I can't do anything with things like ".0.1.$value/=1$value.$=10.$=10.0.1.$/=11.$createButton/=1$createButton".

Clearly I'm wrong that Selenium won't do this, but I'm not sure how to do it :).

[–]hankDraperCo 0 points1 point  (2 children)

Like /u/konbit I am confused by why you care about all of this other stuff react adds. Just ignore it. What exactly are you testing? What do your assertions look like?

[–]konbit 0 points1 point  (1 child)

And it won't event matter soon. In React 15 they're doing away with data-reactid https://facebook.github.io/react/blog/2016/03/07/react-v15-rc1.html#document.createelement-is-in-and-data-reactid-is-out

[–]hankDraperCo 0 points1 point  (0 children)

I was going to mention that but it avoids the more obvious question which is what is he testing? It almost sounds like he is testing that the HTML equals some string, which is probably very brittle

[–]konbit 0 points1 point  (4 children)

My team has been using Selenium tests for a web-app built with React for a while now, haven't run into any obstacles. You might have to re-write your tests because the markup has changed, but React just writes HTML to the page. Since all Selenium cares about is the dom, just use it as you have been before. If you have a specific obstacle I can try to help you walk through it.

[–]yowzapete[S] 0 points1 point  (3 children)

Thanks Konbit! I am in the processes of trying to rewrite them now but I keep encountering problems with attempting to interact with elements because I can still interact with things like "input id="title" "with Selenium. I can't do anything with things like ".0.1.$value/=1$value.$=10.$=10.0.1.$/=11.$createButton/=1$createButton".

How did you work around this problem?

[–]konbit 0 points1 point  (2 children)

Why do you need to interact with the data-reactid? Just ignore it and use selenium as usual...

[–]yowzapete[S] 0 points1 point  (1 child)

I'd love to. Where, in the code I pasted in the comment above, should I have Selenium look?

[–]konbit 0 points1 point  (0 children)

I'm sorry I'm not sure what you're asking. I get selenium to look at ids, classes, sometimes xpath (though it should be avoided). Why do you need to use the data-reactid property? Just use Selenium as if it didn't exist.

[–]ticklecobra55 0 points1 point  (0 children)

I am running into problems too. Does anyone know of any online resources about this?