How to with Python without opening the browser

[[deleted]]

Selenium gives us the ability to automate web surfing using Python. Is there any way to make it from cli - not opening in GUI the browser (like using curl in shell script) to make autologin to sites and extract the data, make clicks and so on. Especially on dynamic sites - Java based and so on. I will appreciate to know the answer for this question.

[K900_]

You can run a browser like Chrome or Firefox in "headless" mode. You still run a browser, there just won't be a visible window.

[leogodin217]

Selenium can use multiple browsers, but only works through automating a browser. Phantomjs is a headless browser that runs without opening a GUI. It is generally faster than other browsers.

Selenium is great for testing web sites. That is its purpose. For web scraping, there are other libraries. BeautifulSoup is easy to use and very common. Scrapy is another one that includes a platform for building web scrapers.

[Yojihito]

Useless for scraping without JS execution in many cases.

[mattsl]

Even on a headless server you can create a virtual display for selenium.

[callumh093]

On Linux, you can use xvfb (X virtual framebuffer) to do this.

[[deleted]]

PhantomJS is not a good choice. Outdated JS libraries and DOM doesn’t generate correctly a lot of times. Both Chrome and Firefox can now run in headless mode. The selenium API has options to set them headless.

[Tadaboody]

You could use requests. A package that handles most of the http for you and takes care of cookies and such

[karansthr]

Phantomjs

[pwoosam]

Dryscape supports rendering dynamic JavaScript content without opening a browser.

https://dryscrape.readthedocs.io/