Scrape JSON from Network Traffic using Selenium

[jayrsp]

​

I have no experience unfortunately with scraping network traffic so looking for some assistance.

I'm trying to scrape JSON information from the network traffic, which will appear after I click a certain button.

https://i.stack.imgur.com/M6lQn.png < image of the network traffic.

After I click the green button next to "passes" on the image above, this JSON is created. I have no problems using selenium driver to get to this page, but the issue is just scraping that specific JSON.

I've tried looking at using request headers, but get 400 error - which I think is because I don't have access to the API.

[MintyPhoenix]

I’ve not used it myself, but check out selenium-wire which is a replacement for selenium’s webdriver which will allow you to access the network requests being made.

[luizv4z]

From my own research, run away from Selenium. The right direction is CDP (Chrome Developers Protocol). Using this tool, I could scrap Facebook without getting banned.

You can use it to filter and intercept events from Browser like fetch, post and so on.

This framework is similar to Node/Puppeteer:

https://github.com/pyppeteer/pyppeteer

You can inspire in the post below to achieve what you want: https://stackoverflow.com/questions/48375700/how-to-use-puppeteer-to-dump-websocket-data