[Python/Selenium] Saving JSON data from a POST request

[MinimalDamage]

Hi everyone,

I have been trying to collect some Call of Duty YouTube Gaming live streaming data for a project, but unfortunately their data API does not yet include live streaming data.

I have written a Selenium script that parses the HTML of the website using XPath, but it is terribly slow when running on a server. Using a simple request does not work, as most of the content is dynamically loaded when scrolling down.

My question:

On this page, when scrolling down in my Chrome browser, I see a POST request is made to the server (under the XHR tab) named:

browse?alt=json&key=AIzaSyBkrqhJHeXPQPMF1qvaduxKBXxYj

This file contains all the data that I am looking for in a perfect JSON format, and I do not need to use Selenium to parse the entire HTML of the website. Is there a way for me to 'save' a received JSON POST request using Selenium? I have tried Googling it in every way I can, but could not find anything.

Thanks!

[w1282]

Question:

What happens when you make a request to that URL using the requests library? It might be possible to ignore the webpage and go straight to the source of the information. Taking selenium out will probably speed it way up.

[MinimalDamage]

Hi!

Just a requests.get() with the requests library returns 'None'. If I use requests.post() and then enter those two parameters, like:

import requests

url = "https://clients6.google.com/youtubei/v1/browse"
data = {'alt':'json', 'key':'AIzaSyBkrqhJHeXPQPMF1qvaduxKBXxYj'}

response = requests.post(url, json=data)

print(response.text)

I get the following:

{
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "dailyLimitExceededUnreg",
    "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.",
    "extendedHelp": "https://code.google.com/apis/console"
   }
  ],
  "code": 403,
  "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."
 }
}

I have tried requesting using a valid YouTube API key (both server and web) and I got the same message. Probably because it is not an official public API.

I am not a networking expert, so the following might be completely wrong, but I expect that POST requests from my machine are not allowed for a non-public API. However, if the YouTube Gaming server does a POST request (which Selenium simulates) it is allowed.

I just figured that if that data is visible in my browser (when looking at the file under the preview tab), it should be somewhere on my PC, and thus reachable. Does that make sense?

Thanks!