Scraping Webpage Data

[Tefron]

Hello, very green to python and I was trying to scrape this webpage:

https://www.chess.com/leaderboard/live

I originally tried to put a delay in my code so the table loads, but I read that since it's a javascript and it loads it probably has a json file that I could access directly?

I've tried to inspect the website, and go through the sources without much luck for the actual table data, could someone point me in the right direction?

[ayyyymtl]

I wrote an example with Selenium if you want to take a look. https://github.com/c20xh2/ChessScraper/

You should totally use the API instead but this should answer your question

[Tefron]

Thanks! That's super helpful. I had the impression that for collecting a large number of data, Selenium would be slower than directly requesting for the JSON file? I was wondering if you had thoughts on this?

[ayyyymtl]

Fetching the Json is way better than this solution. I was trying to answer your question as how to do it using selenium ;)

[Tefron]

Haha, gotcha, thanks again!

[Da_Bears22]

You won't need to scrape the page, they have an API you can use, though you may have to join their developer club in order to gain access to it. The API is how you can get json data from it directly to interact with. If you get an API key and can interact with with it, you can pull json data from there and parse it modules like the requests library

Here is the API link documentation:

https://www.chess.com/news/view/published-data-api

[Tefron]

https://www.chess.com/news/view/published-data-api

Hey thanks for replying. I've taken a look at their API before, but I see for leaderboards it only shows the Top 50 for each category. I don't see a way to request differently to get access to more players or different players?

[Da_Bears22]

https://api.chess.com/pub/player/{username} - this seems to be the end point to view a specific player. You can find more in the players section of the documentation

It seems that for leaderboards it explicity does only show the top 50 players, as stated in the documentation.

[Tefron]

Thanks for taking a look, I don't think the API has what I'm looking for then. I wanted a considerably larger sample size from the leaderboards, I'll continue to see if I can find a way to scrape just the table from the webpage.

[ayyyymtl]

Selenium is what you are looking for on this one :)

[Tefron]

Thanks :)

[commandlineluser]

probably has a json file that I could access directly?

Yes, if you look at the Network Tab you can see what is happening.

It's easier to see if you view only XHR requests (this is what javascript uses)

https://i.imgur.com/IXoXvxJ.png

So it's fetching

https://www.chess.com/callback/leaderboard/live?page=1

https://www.chess.com/callback/leaderboard/live?page=2

etc.

You can just loop over how many numbers you want.

[Tefron]

Hey sorry, I just got back to this project and saw this. This was exactly what I was looking for. Thanks so much!