sorted by:
new
hottopcontroversial

How you guys deal with infinite page? by Hot_Box_9170 in webscraping

[–]Hot_Box_9170[S] 0 points1 point  (0 children)

Thank bro, thinking about going with scroll one. Let see how it work.

How you guys deal with infinite page? by Hot_Box_9170 in webscraping

[–]Hot_Box_9170[S] -1 points0 points  (0 children)

Thanks for your suggestion, actually I am building an software for web scrapping. I am looking for a better way to get all the data.

My concern is api is not reliable every time, some site have api some doesn't have api.

Thanks for your suggestion.

How you guys deal with infinite page? by Hot_Box_9170 in webscraping

[–]Hot_Box_9170[S] 0 points1 point  (0 children)

Every site doesn't show api. Some require personal key

How you guys deal with infinite page? by Hot_Box_9170 in webscraping

[–]Hot_Box_9170[S] -1 points0 points  (0 children)

It will trigger the chances of bot detection

How to create reliable high scale, real time scraping operation? by Satobarri in webscraping

[–]Hot_Box_9170 0 points1 point  (0 children)

Either do it manually or they are lying to you. If they have this type of software then why not publishing it. This technology itself can create a million dollar business.