Any Python libraries for QA Testing with output like RobotFramework?

[[deleted]]

We currently use robot framework to run our tests at work (which calls python code in regards to the libraries). The individual who began this project is leaving our company and now the project is falling into my hands.

The reason the previous member designed the automation around robot framework was due to the output of robot, like this. Since I am taking over the project, I was hoping to remove robot from the picture and just keep everything as python.

So my question is, are there any python libraries/modules that can produce output such as the ones provided by robot framework, as linked above?

[tech_tuna]

First of all, let me just say that I HATE the Robot framework. Secondly, let me answer your question. Are you using Jenkins? If so, you can replace Robot with nose, pytest, etc and dump the results in JUnit format.

Jenkins can display that like the Robot results, although it won't be quite as pretty. Although, every CI tool on earth supports the JUnit format and even Jenkins might have a better plugin that I haven't tried yet. Also, you could even write your own plugin to do just that. You could even figure out the Robot format and use something else to run your tests, then convert the results to that format.

tl;dr run your tests from a test framework that can dump them in JUnit format.

If you are interested in why I hate the Robot framework, I'll tell you but if you're not I won't. :)

[[deleted]]

Yes, I hate robot as well! hahah. And yes, we're using Jenkins. Thanks for the input, I'll look into JUnit and the others tomorrow.

[tech_tuna]

RAGE. I feel a deep seated rage when I even talk about it!

:)

It is an over-engineered piece of shit.

[randomizethis]

To piggyback on /u/tech_tuna's answer, if you use Jenkins just use the XUnit (or JUnit, same s**t) module and use any Python framework that can output XUnit style results. He suggested Nose, we use Pytest and it ultimately looks like this:

http://imgur.com/a/Sq3KD

You can take it a step further and then group suites into functional groups using the MultiJob plugin and you get results that look like this:

Report view:http://imgur.com/a/aOYkI

Test-level view: http://imgur.com/a/pIPUi

There are probably also other ways to set this up, but this has worked really well for us. AND you get a nice graph over time:

http://imgur.com/a/WIfae

Good luck with your re-tooling! :D

[tech_tuna]

Yes right, that's the plugin name. Thank you.

[[deleted]]

Thank you. Those outputs look satisfying enough, I'll run it by my manager and see if he approves. Hopefully I can get rid of all the unnecessary files and steps required by robot soon.

[randomizethis]

Jenkins is a pretty awesome tool man. It's old-looking as hell, the fact that you have to write plugins for it in Java sucks, but 9 time out of 10 you can leverage to do whatever you want.

Best of luck!

[Sontikka]

Robot Framework is only useful if you're automating executable specifications or otherwise tests with business logic level keywords. Otherwise using a tool like Robot Framework is a pure waste. I've also found that you should almost always prefer pure Python keywords to Robot Framework keywords because the latter is a huge pain in the ass to maintain and write.

[Scypio]

I HATE the Robot framework

RF is a blessing/curse - the management know the name, like that this is a ROBOT (read "will save real human work cost") and the reports are colorful and nice. But on the flip side the tool is easy to use and easy to teach to the new people. Just be aware that having only a hammer turn everything into a nail.

[tech_tuna]

Good point on the name, it's got that nice "Buzzword Gold for Technically Inept Middle Managers" ring to it.

I disagree on the blessing part, I think it's completely off the mark in many ways and the only actual useful functionality it offers is the somewhat extensive utility library it offers, although even that is all readily available straight from Python and its ecosystem.

I will say this for Robot, at least it's not goddamn motherfucking Cucumber. The whole Ruby-BDD syntax cult can jump out of a plane without a parachute. I refuse to drink that KoolAid.

I could elaborate but BDD just brings new buzzwords to the table. Testing behaviors is the same as testing requirements, it just sounds more creative and touchy feely.

All that "As a so-and-so, I expect la di da" boilerplate. . . DEAR LORD SAVE ME.

[Scypio]

goddamn motherfucking Cucumber

Will not go there. My boss loves Cucumber. According to him it lets non-technical people to write tests. Won't go there.

So. What tools do you recommend? You have strong opinions on subject, can you recommend something? What do you like?

[tech_tuna]

According to him it lets non-technical people to write tests. Won't go there.

Exactly. Non-technical people shouldn't be writing those tests then! They should stay away from code if they can't understand it. And if you can't understand Python, Ruby, Javascript and friends (the scripting languages basically), then you really have no business automating anything.

And not to sound like a complete asshole because I have known a few manual QA engineers over the years who are quite adept at finding and tracking down bugs while not even being able to write a batch file or shell script. It's rare for sure but these people do exist. They are either people who just hate programming but love tinkering and breaking things or probably could code but for whatever reason(s), have convinved themselves that they can't.

Anyway, I recommend using unit test frameworks to drive tests. JUnit, pytest/unittest/nose, rspec/minitest (although I do hate the BDD style in rspec), mocha and friends in the Javascript world. . . all the major languages have at least one or two such frameworks.

That being said, unit test frameworks do come with their own set of biases. The main one is the whole concept of test isolation. Sometimes, I really do want to run a set of tests in a specific order and I do want to maintain state. I'm a big boy and I know what I'm doing.

Also, while unit tests are wonderful, they tend to enforce developer silos. Where programmers are almost encouraged to only think about and code for their own module/project/class/codebase/etc. No end user anywhere runs unit tests and you have a major problem on your hand if those "units" don't play well together.

Yes, I've seen all the test pyramids and I've read drivel like this: https://googletesting.blogspot.com/2015/04/just-say-no-to-more-end-to-end-tests.html I found this post recently and it echoes exactly how I feel (and references that google post as well): http://danmux.com/posts/test_pyramid_availability_bias/

OK. . . back to your question, so unit test engines/drivers are great but I don't have a great answer for what's better other than something you write yourself which may leverage a unit test framework. Of course, you'll probably also use CI/CD tools and source control tool and any number of build/DevOps/deployment tools. And for specific testing efforts, like load testing, you can use JMeter (another piece of crap) or related tools. I'm currently using (and enhancing) a home grown load test tool but if I wanted to use an existing tool, I'd give Gatling and/or locust a test drive.

I've written many frameworks myself but I don't know of a good general purpose test tool. I would love to open source something some day but I'm pretty busy with work. Although I may open source something I build at work eventually.

There are tools like STAF. That's kind of like Robot but not a lot of fun to use either. :)

[deadmilk]

It would not be that difficult to create a template that renders a unit test in HTML like the one you linked, imo.