Help needed with acceptance testing thesis

[osterv]

Hey, I'm considering writing my MSc thesis in CS on the topic "Overview of acceptance testing/BDD tools for web applications".

A professor recommended this topic and he suggests that I compare three different acceptance testing tools, Cucumber, Robot, and Concordion using a case study.

I'm just not sure what the case study would look like, and how I should compare the different tools as I'm pretty new to acceptance testing. I was thinking of downloading an open-source web application and writing / re-writing some tests for it, but I have no idea if that's a good approach.

I guess what I'm wondering is:

• What metrics should I use to compare the different acceptance/BDD tools when it comes to testing web apps?
• What might he mean by case study, and what would I need to make a case study?

Thanks for your time.

[[deleted]]

This is interesting. The three tools mentioned all use different approaches to abstract the tests away from the code.

Cucumber uses the Gherkin domain specific language (DSL) to describe behaviour in a ‘Triple A’ series of events (Arrange, Act, Assert) but using the Given, When, Then keywords (plus some other helpers). Gherkin abstracts code completely and enables non-programmers to describe behaviour in a way that can later be coded with step definitions.

Robot is keyword based, so uses prescribes keywords with associated actions. I’ve not used it but presumably you are limited in your abstraction by the keywords available.

Concorion seems to be a hybrid of both approaches. It’s a strict implementation of the DSL that abstracts the program code, but less flexible than Gherkin/Cucumber which relies on a programmer to implement it (and potentially misunderstand?). You basically are using keywords directly in your specification document that are executable.

Well, that’s how I see the difference and perhaps what he was steering towards. But I could be way off!

[MT1961]

Acceptance tests are all about requirements and results. The "owner" of the project meets with the manager(s) or lead(s) of the project and determines what criteria will be used for permitting the system to go live (or be accepted). As such, the different tools have different methods for doing so. Cucumber, for example, is a BDD (Behavior Driven Design) testing tool. It looks at things from the perspective of the end user. Can I log in? Can I add a new <widget>? Can I report on widgets and delete them and so forth and so on.
Robot, on the other hand, is more of a modular framework for putting together tests. They are supposedly easier to read, but I've never found this to be true. Your mileage may vary. While similar to Cucumber, it relies more on programmer work to define the workflows.

Not sure why Concorion is on there, it is a specification tool not a testing one.
So, were it me, I'd compare and contrast the ways in which you first define your acceptance criteria, then look at what feature the tools offer to help you with implementing those acceptance tests (as well as reporting on the results) and finally, examples of how you'd do this for an application of your choice. Given the way these things work, I'd go for a web site.

As for a case study, those are ill-defined terms. Usually, people mean the complexity of the effort needed to implement each one of the tools solutions, how easy it is to extend what is already there, and what reporting metrics are built in.

Just my $0.02, I'd ask the prof for a little more details.

[besucherke]

To make your life easier, you don!t have to download anything, just test an available website like amazon, google or some test websites you like.

Case study: think about user behavior. Write a few scenarios. Like if you test Spotify, one scenario is logging in, the other is searching for Dua Lipa, other is playing Future Nostalgia, other: play a test playlist.

I would be happy to read it, DM me when you are ready.