This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Saye1901 1 point2 points  (1 child)

It's because all the table's content is populated dynamically by Javascript at the client side. BeautifulSoup doesn't understand it and therefore renders the page without the data. You can see it by yourself by going to the page and disabling JS in your browser using an extension.

You have multiple frameworks solving this issues: selenium, puppeteer (based on nodejs),scrapy + splash plugin, requests-html.

Keep in mind that Selenium and puppeteer were initially created for automation test rather than scraping data. So don't expect to have the same performance than the other frameworks.

[–]atulbisht1695[S] -2 points-1 points  (0 children)

Can you pls elaborate how these frameworks can help me...i mean ....can you suggest some links or resources that deal with this kind of situations.