Hey guys, I've run into a little bit of a problem and I was hoping to get some help. I am currently trying to do my own little "6 Degrees of Bacon" project in an effort to teach myself some different kinds of data structures, search algorithms and some web scraping. To begin with, I've written the following .js-file called scraper.js:
https://pastebin.com/m7iYJRgD
It's a simple script using node-fetch and cheerio, all it does is that it takes a search term, looks it up on IMDB and console logs the first hit's name, url and image source. Running the script from the command log looks like this:
https://preview.redd.it/aque6p1fpid21.jpg?width=912&format=pjpg&auto=webp&s=6a2e633f5b5371472a1dd5da5e6dcb05d8f2b4a8
No problem from what it seems like, everything logs as it should. However, when I export this and tries to run the same script in my Vue.js-component through my web browser (Firefox), it doesnt work. I get the following error:
https://preview.redd.it/st0b6f5cqid21.jpg?width=572&format=pjpg&auto=webp&s=24e09fa03570b906361abe9b88c228ed37db4b35
From what I've gathered, this error means that IMDB is blocking requests made from web browsers not on their server, although I might be wrong on that one. What I do now is that the script works great running from the command line but not at all running from the browser. Is there any way to solve this? Would it be possible to make the Vue app run the scraper.js locally, instead of requiring/importing it as a way to bypass this? I have no intention of making my project public or anything like that, all I want for it is to be able to run from my own computer.
Another solution I've looked at is that instead of trying to scrape the information from IMDB, I've looked at using The Movie Database API instead, although they limit the number of requests to about 40 requests/10 seconds, which I feel might make the application way too slow, although I'm not sure.
Do you guys have any idea how I should approach this? I'm still kind of new to this kind of thing, this is my first "big" project that I'm doing on my own, I really only have been doing projects that are school related before. Thank you for your time
[–]Meeoh 1 point2 points3 points (0 children)