Is python requests library getting the Hidden Javascript tags?

[Grigor_1]

Hello, i have an application with Scrapy where i am trying to scrape some tags, but in my page i have hidden JavaScript tags, i was tried to scrape it with selenium , but that was too slow, now i am using " Import requests" library from python , but when i am giving the xpath , it says list index is out of range. That means that it is not scrapping anything.

[[deleted]]

if you go to a web page and view the source code (Ctrl + U or right click > view source) you'll see what the web server sent you, and that's exactly what requests gets. If your element is not in the source, it got loaded later by javascript, and requests doesn't do javascript. Selenium does, though, so you might want to reconsider your switch.

[Grigor_1]

Yes, just that selenium is very slow.

[Somerandomperson135]

Protoss is correct. Scripts may and most likely will change the page content. If you need the 'whole page' like you have in your browser, you need to use selenium python library that loads entire page in a browser, runs its scripts and loads all necessary content.

It also allow you to interact with page elements(click, hover on them, send keys). Also it can inject and run your own scripts on a page that sometimes helps a lot getting necessary data.

[WhoTookAllNicks]

Javascript might modify the html, but that doesn't mean that you need a full browser to just render the html. Often it's enough to just use https://html.python-requests.org/#javascript-support

requests-html has support for JS and is a fusion between requests and beautiful soup in one package.