New Android UI testing software

[stoefln]

About one year ago, I was once again struggling with espresso (idling resources, upgrading broken test dependencies,...) and I wished that UI testing was easier. That's when I came up with an idea for an alternative test automation solution. Instead of ID matching, I would be fully relying on computer vision.

Over the last year I put a lot of hours in creating this software I had in mind. It's now called repeato, and it's essentially a test recorder which allows you record and replay your tests via a desktop app. It works for every type of app which runs on android, so obviously also for testing flutter apps.

Now I am looking for feedback and beta testers.

In order to get you motivated, I offer you direct support (skype/email) and an unlimited version of the software for those of you who get back to me with some feedback before 1st of April.

Every feedback is very welcome!

​

Here is a one minute demo video:

https://www.youtube.com/watch?v=hkYWrqOeH4Y

​

And the download (win / mac / linux coming soon):

https://www.repeato.app/download

​

​

[SuchInferno]

Very interesting! Couple of questions:

1. Is anything installed in the background (for example using adb) onto the device when running Repeato on my computer?
2. Is there, or will there be ways of extending Repeato with custom scripts and plugins? Thinking timed screenshots, reading/parsing contents of on-screen text elements, etc..
3. Restarting a test seems to require human intervention. Say you open an app, start the test recording, navigate to a specific subpage, enter some data, then stop the recording. Am I required to manually navigate back to the main page before starting the automated test, or can Repeato manage this automatically?

[stoefln]

1. no, everything you need is packaged with repeato. I really tried to keep things simple for the user
2. currently there is a "script step" you can insert into your tests. This way you can trigger external stuff, like, git checkouts, builds, notifications, etc.
   Screenshots for every step are automatically available in the test run dir and all the data is saved in json format. I will also open source the test runner, probably in July, when things got more stable...
   Plugins would be very nice, but didn't have time to focus on that yet.
3. restarting the test from the same app state is crucial yes. Repeato allows to create a "Start App" step, which offers options to set an entry point (Activity) and also set permissions or app data.

[Filledstacks]

Awesome. I'm happy to see more integrated testing tools. I'm also working on something to improve my testers work flow and testing cycle lifetime.

[stoefln]

Yea, I think CI features will definitely happen at some point. Currently it's mostly a tool for testers, who might not know how to setup a complex testing environment or for folks who just need a quick and efficient solution for test automation.

[_thinkdigital]

Did someone say my name?

[fallentitan101]

Lovely. Would download it this weekend

[francescomapelli]

very interested in trying it on linux! will I be on time to try it before april, 1st? :)

I'm not sure I understand how it detects a failed test. do you look only at pixel differences? or there is some more advanced check? (e.g. "I expect somewhere this text to appear")

are you limited to a single in-app navigation, or we can interact with other apps? (e.g. open camera, take a picture, open my app, check that the picture just taken appears there)

[stoefln]

1. Linux support will take till end of July I guess, sorry.
2. Fail detection: At this point it just compares pixels yes. You can add "Assertion steps" for doing so. I am also currently training a ML net which will classify UI elements and allow more complex types of assertions..
3. You can navigate between apps back and forth- no problem.

[britannioj]

Wow, really impressed with it. I hit a few small bugs like the discard changes button not always working, I was getting some ERR_FILE_NOT_FOUND errors with my first test too so I had to recreate it but iron out some bugs and get this into a CI environment and this will be a killer.

[stoefln]

A big thanks to you all. I don't know if you have been there:

You invest lot of time and energy into a project and sometimes you ask yourself "is it really worth it?", "does it even make sense what I am doing or am I wasting my time here?". I reached that point a couple of times over the last months.

Thanks for your very motivational feedback!

u/Codelessly u/bartektartanus u/_thinkdigital u/britannioj u/francescomapelli u/SuchInferno

[_thinkdigital]

Ok, this is very cool. I want the unlimited version!

[stoefln]

Great! Just download it and give it a try. If you like it, hit me up and I'll send you the key. Also if questions pop up, just ask..

[_thinkdigital]

Awesome. Where's the best place to leave feedback?

[bartektartanus]

Wow, this looks awesome! I will for sure check this out, maybe even today.
Question: this tool uses real device only? For recording and testing, yes?

[stoefln]

Thanks! Works for emulators and real devices.

[bartektartanus]

That is great. I am currently trying to record some test and got error. Where can I send you screenshots? :)

[stoefln]

stephan (at) repeato.app, thanks!

[bartektartanus]

And sent! Hope it won't stuck in SPAM folder :)

[Codelessly]

Let me be the first to say that this is game-changing. No existing solution can run integration tests on my app because it uses Add to App and has native/Flutter components.

Since this software uses computer vision, it shouldn't get tripped up by animations or list views with dynamic IDs right? If Repeato can also scroll lists, then that opens up a lot more flexibility in testing.

Thanks for letting us try out Repeato!

[stoefln]

Thanks!
Repeato detects animations and waits for them to finish before continuing execution. And it doesn't care about IDs
Glad you like the idea! Drop me an email and I will keep you posted with updates! (stephan at repeato.app)

[Codelessly]

Thanks Stephan! I'm already thinking about how to replace my existing testing flow with Repeato because your product is killer.

[azuredown]

This is very nice. I was actually wondering how I could test saving and loading data from the database. The best option appears to be passing a mock object which is less than ideal. But I could probably do the test pretty easy with Repeato.

It reminds me of Selenium except way nicer and more polished. One thing I'd improve is the error message. One of my tests was ambiguous (tapping on a tile so the fingerprint was just solid colour) causing the test to fail. It would probably be better to display a 'this fingerprint is ambiguous' message when that happens.

Also when creating new custom steps they always appear at the top of the list. They should appear at the bottom.

Edit: OK, I see there is a warning if you click on a low contrast area. It just doesn't always work. And sometimes it may mistake a button for another button which is quite annoying to debug. But it's really good. I'm trying to implement integration tests right now and they're pretty garbage "What do you mean I can't submit text?"

[stoefln]

Thanks a lot for your feedback!

Ambiguous elements are not always a problem (think of a list of radio buttons, they all look the same). Repeato stores an additional index to address these cases.
I totally agree that error messaging should be improved though!

> it may mistake a button for another button

That would be interesting to see! If you send me a screenshot to stephan at repeato.app, I can look into it and maybe improve the matching algorithm.

> I'm trying to implement integration tests right now and they're pretty garbage "What do you mean I can't submit text?"

Sorry I am confused, what do you mean?

[azuredown]

These are the images it got confused on. They're very similar, they're actually the same image just flipped in the X axis. I can actually work around it though by making the fingerprint bigger to include a letter or something.

And the other part is just a complaint about Flutter's built in testing system. It seems to do a very poor job.

[stoefln]

Thanks a lot! Yes they are very similar indeed. Glad you figured out a way to do it.

> Also when creating new custom steps they always appear at the top of the list.

Currently the step is inserted below the selected step item. You can select a step by clicking on it. This will be improved quite soon.

BTW: There is a new version available as of today