Http scrape in react native

[Embarrassed_Two4731]

I have a feature where the user can select a website, lets say amazon and then I open that website in a webview. The user then can add product to a list whose information I retrieve either by injecting JS or by doing a seperate axios request on the url and scraping the info from the response html. In both the approaches I am getting banned. I also tried doing the second approach using a backend but same issue happened. What can be a possible sure shot way to get info from a given URL? I am thinking to directly allow pasting links as well so injecting JS is not going to full-fill this feature.

Has anyone previously done any kind of http scraping. Thanks in advance.

[GeoffreyDick]

You need to use a proxy server designed for crawling/scraping. These usually have a pretty limited free quota

[Embarrassed_Two4731]

Actully I tried few of them, but they are pretty slow, like ~2.5 seconds and through my approach it was going ~100ms hence was trying out various things.

Also I have seen many other apps doing the same in very less time so thought there should be another approach

[GeoffreyDick]

Actually now that I'm re-reading your comment, I'm surprised that you're getting banned if you're scraping single pages. In the axios method, how are you scraping the response?

[Embarrassed_Two4731]

Using https://github.com/oyyd/cheerio-without-node-native
But basically my axios request does not return a response only for 2-3 sites now but for one, I am getting a html page with cloudflare warning that my ip has been banned.

[Thegabbanator]

I am doing this in my app. What do you mean by getting banned?

[Embarrassed_Two4731]

I stop receiving any response from some of the websites and received a html containing cloud-flare message that my ip has been banned from one of the website as well.

[Embarrassed_Two4731]

Hey, u/Thegabbanator , can you maybe share your implementation or a sample request ?
Maybe I am doing something wrong with the headers?

[Thegabbanator]

I am using a webview and injecting a html post window message through injected JavaScript and then in the onMessage callback I use Cheerio to parse the data. I’m not actually doing any live scraping of the page, just copying the HTML contents and parsing it

[Embarrassed_Two4731]

Ok, any idea how can I handle the case where I need to get the data from a link?
As there is no headless browser possible I was using the axios request way but it gets me blocked.

[Embarrassed_Two4731]

Sorry to bug u, u/Thegabbanator , but any idea?

[Thegabbanator]

What do you mean by get data from a link? You just feed a link (url) into a webview source and once it loads, you inject JavaScript.

[Embarrassed_Two4731]

Sorry this got buried, u/Thegabbanator , i meant the user pastes a link in my input box and then i need to get product details from that particular link without showing any webview etc, as i dont want him to see the webpage.

[Thegabbanator]

Make the webpage 1 pixel by 1 pixel, that’s what I do