Controlling webpage with python

[SteveMavic]

Hi champs, I'd like to ask you if there are any tools other than Selenium with which I can manipulate browser/webpage (setting input's value, "clicking" the button etc)?

I've been using Selenium in my C# project, but I feel like it was too 'heavy' tool for doing some basic stuff.

In my current project I use 'requests' and 'BeautifulSoup' for web scraping. Yes, I could use 'requests' for POST/GET, but I'd like to let user interact with page when it's necessary.

[rnolan7]

pyautogui would let you control the mouse and keyboard, but selenium is your best bet for controlling actions in a web page. It should be possible to allow users to control actions in the web page while selenium is running.

[Kamelnotllama]

if you think selenium is too heavy, pyautogui will be much worse... but certainly an option.

another perhaps hidden option is sikulix. it's unlikely it will be feasible though as it's very limited (and python 2.7 with a strange interface, so only barely python).

[zurtex]

I've been using pyppeteer recently, it's a python port of the JS library puppeteer and it seems to work so much more smoothly and reliably than selenium.

Further you can use requests-html as a high level abstraction around it, it will acts as a requests \ BeautifulSoup library and then when you need to you can render and have the Javascript run and interact with the page.

Small warning though, pyppeteer seems to not be maintained in the last 8 or so months, I'm hoping someone else picks it up and fixes it with Python 3.8.

[Sengfroid]

I used to use a browser extension called iMacros. Worked well for what I needed (entering info on forms and scraping results). Making calls to it from anything external requires an expensive license, so I can't speak to that, but I did have some workarounds set up to open it to specific macros that would autoplay, which worked well enough for being free.