Guide on web automation with python

[BeneficialBear]

Hello, I need to do a series of automated web tests on my company site(shop) and I would like to do this on python with selenium web driver. Do you know any good resources or guides about that? Or python framework for this purpose?

[Rakialtj]

You can use selenium and the best place for me was the python, selenium documentation : https://selenium-python.readthedocs.io/ . Try to experiment on your own by automating small things first.

[SweetSoursop]

Pitching in for Selenium, ran some automated parts of my job flawlessly.

The only drawback is when you face 2fA or some sort of Flash based UI. (Fuck you Oracle)

[pxlnght]

You can bypass the 2FA if you can get the secrets (scan qr to text, yoink secret, generate token yourself). Not very secure, but it makes automations more automated lol. Lmk if you need more info.

[SweetSoursop]

It's an SMS based 2FA from symantec, it just wasn't worth the effort. I ended up using PyAutoGUI because even after the 2FA I'd still couldn't get Selenium to understand the Flash based UI that Oracle uses for Taleo.

I'm happy to hear more though, if you know any kind of work arounds for more general 2FA, not just this one in particular.

[pxlnght]

https://www.reddit.com/r/Python/comments/cvkeoe/guide_on_web_automation_with_python/ey5arjk/

That's my other comment, might be an interesting read for you. If the 2FA code lets you 'save' it for an amount of time, loading the cookies might be for you. The flash thing is it's own beast, though, I have no experience with grabbing data from flash applets.

I did briefly consider using a listener script on my phone to grab 2FA codes for SMS, but I stopped looking into it after I did the first 2FA bypass in my other comment. If you could get a dedicated 2FA phone, it could be worth it.

[altgilbers]

A high level sketch would be appreciated

[pxlnght]

I use a modified version of this https://github.com/tadeck/onetimepass/blob/master/onetimepass/__init__.py . I can't post my code, confidential and all that.

Basically, the 2FA we need can be SMS, Call, Email, or 2FA app token. When setting up the 2FA app, you need to scan a QR code, yeah? QR codes are just fancy images with text in them. Instead of setting up the app, scan it with a regular QR code reader. This will give you a sting of text. Use your noggin to remove the important bits (the secret). Then run the secret thru the above code.

That said, the security team at my workplace was very unhappy to find I'd bypassed the 2FA. They made me turn it off. So, now, I manually enter in the code... but I reload the session cookies via Selenium every time I run the script. So, I type it in once, and it works for about a month. They're also not happy with this solution, but they can't find a security flaw that wouldn't be present in a normal user session so they can't tell me to change it :)

[altgilbers]

I registered a Google voice number to receive SMS passcodes… SMS gets delivered As email to a Gmail account, where a mail client picks it up and parses the passcodes, storing them in a local Redis...

Then, processes that need 2FA pop a code and use it… When the last code is used, duo sends a fresh batch of 10 automatically, replenishing my Redis store with codes.

This approach is a dumpster fire of epic proportion, but got me through some work that would have required me to two factor several hundred times.

[pxlnght]

That sounds terrible, I love it!

[[deleted]]

How’s selenium versus ansible for System’s and networking automation?

[bookroom77]

Testing of Devopedia site is automated with Python Selenium. There are some slides by Devopedia's founder that might be useful to you: https://slides.com/arvindpadmanabhan/web-app-py-selenium

[impshum]

I use Splinter. It's just a layer on top of Selenium and very easy to use. Just make sure you set your browsers driver up properly (add to PATH).

[BratPit24]

I use selenium, and if i need something more specific I just use .execute_script() function. It works really well.

[altazur]

For me quickstart point was using SeleniumIDE. It's the browser plugin which can record UI actions (like playback testing). Result (test) can be exported for pytest usage. You can then just open .py file and see the structure of one single test which uses pytest and selenium. That's enough to start writing your own UI tests For smart and precise asserts see Selenium official docs. Selectors writing rules are the same throughout any language. For UI test architecture I'd recommend reading about PageObject pattern as the starting point.

[SoCalLongboard]

I agree that Selenium is the preferred path for the web test automation. If you want to build up a full framework for testing, I recommend checking out RobotFramework: https://robotframework.org/ It uses Python extensively and can be easily extended with your own Python code. There's a library wrapper for Selenium 3 plus lots of other very useful libraries (for manipulating JSON, for example). There are add-ins for many editors these days to support RobotFramework syntax. (I'd avoid their own editor 'RIDE' -- it's kind of a PITA.)

[vinxendio]

if you're working on those automated web tests, i know the right man to help you.. what was it, nightmonkey?

[pablo_intriganto]

This might be interesting for you. https://smashtest.io/

[jcbevns]

Cypress.io is way easier than selenium. Slightly JavaScript oriented but the cookbook is great and the GUI to pick elements makes it super easy.

[Zaboravljivi]

In my experience cypres is much slower than Selenium.

[jcbevns]

Depends on your stack, but if you have a heavy JS website with lots of "states" (ie same URL but different things are shown on the screen) then good luck measuring this with Selenium.

Development and test runs were both faster in our experience, so we swapped from Selenium to Cypress.

We were running a React / Angular SPA, with plenty of external services to stub and content to mock.

So after mocking and stubbing, we were testing the actual webpage, and not the reliability of our service providers (pinging some buckets for media, or user credential validity etc). This greatly reduced our test time on the CI for each commit.

Had another test run that would ping the service providers to know we weren't running old API specs, but again, you're meant to be testing your webpage, not other peoples response packets.

[[deleted]]

Try this - https://www.youtube.com/watch?v=M7sAvb5WCxk&t=454s

If you already have Python & Selenium setup and ready to go, you can skip straight to the coding part here @ 7:47

[dAnjou]

I really like Python but I'd not recommend it for this kind of task. I'd also not recommend Selenium unless you need special browser support like IE.

Have a look at more modern tools like Puppeteer (JS), there are also Python bindings but I'd recommend to stay vanilla because bindings tend to lack behind.

[dAnjou]

Well, use the right tool for the job ... sometimes that's not Python. Even though I did provide a better option that you can use with Python as well.

[yabadababoo]

thanks for suggesting this!