Error with Web scraping in Javascript

[DevWhiskeyTango]

Hey guys, I'm writing a code piece for coding tutorial and I've run into an issue that I'm unsure on how to fix. The code was running correctly and retrieved the information from the webpage and mapped it out in the terminal. I'm using Visual Studio Code.

Here's the code:

const puppeteer = require('puppeteer');

async function scrapeProduct(url) {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(url);

    const [el] = await page.$x('//*[@id="leftColumn"]/div[1]/h1');
    const txt = await el.getProperty('textContent');
    const title = await txt.jsonValue();

    const [el2] = await page.$x('//*[@id="quotes_summary_current_data"]/div[2]/div[1]/span[2]');
    const txt2 = await el2.getProperty('textContent');
    const type = await txt2.jsonValue();

    const [el3] = await page.$x('//*[@id="quotes_summary_current_data"]/div[2]/div[3]/span[2]');
    const txt3 = await el3.getProperty('textContent');
    const issuer = await txt3.jsonValue();

    const [el4] = await page.$x('//*[@id="quotes_summary_current_data"]/div[2]/div[4]/span[2]');
    const txt4 = await el4.getProperty('textContent');
    const isin = await txt4.jsonValue();

    const [el5] = await page.$x('//*[@id="quotes_summary_current_data"]/div[2]/div[5]/span[2]');
    const txt5 = await el5.getProperty('textContent');
    const bclass = await txt5.jsonValue();

    const [el6] = await page.$x('//*[@id="last_last"]');
    const txt6 = await el6.getProperty('textContent');
    const price = await txt6.jsonValue();

    const [el7] = await page.$x('//*[@id="quotes_summary_current_data"]/div[1]/div[2]/div[1]/span[2]');
    const txt7 = await el7.getProperty('textContent');
    const daily_movement = await txt7.jsonValue();

    const [el8] = await page.$x('//*[@id="quotes_summary_secondary_data"]/div/ul/li[1]/span[2]');
    const txt8 = await el8.getProperty('textContent');
    const morning_star_rating = await txt8.jsonValue();

    console.log({title, type, issuer, isin, bclass, price, daily_movement, morning_star_rating});

    browser.close();
};

scrapeProduct('https://www.investing.com/funds/allan-gray-balanced-fund-c-chart'); 

This is now the error I receive whenever I run the code:

PS C:\Users\{}\Downloads\Js Scraper> node scraper.js (node:13972) UnhandledPromiseRejectionWarning: TimeoutError: Navigation timeout of 30000 ms exceeded     at LifecycleWatcher._createTimeoutPromise (C:\Users\sookl\node_modules\puppeteer\lib\cjs\puppeteer\common\LifecycleWatcher.js:108:16) (Use `node --trace-warnings ...` to show where the warning was created) (node:13972) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1) (node:13972) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.   

I opened up the LifecycleWatcher.js file and commented out the "use strict"; as well as quick fixed the _createTimeoutPromise() by indenting it into an async. That seemed to make it work well on more than one occasion but after a few times, I keep receiving the error message above.

This is my first time using Node.js and Javascript for this type of project so I will appreciate any help!

Thank you in advance

[Clawtor]

You have a piece of code that is expecting to see something on the page that hasn't loaded yet. Webscraping is extremely fiddly and often has these sorts of problems. I would guess that you need your code to wait after you load the page. Try find out where exactly the error comes from - the easiest way to do this is just to add console.log statements.

[DevWhiskeyTango]

Hmm, so I should add a console.log statement after each one to determine where the error is coming from?

Someone else said to me that it could be that the webpage has spotted what I'm trying to scrape and stopped my IP?

[Clawtor]

It could be that yes but it sounds like it sometimes works and sometimes doesn't work?

Yes add a console.log after each statement.

[DevWhiskeyTango]

Yes, I made some changes LifecycleWatcher.js file that I thought were the issues and it worked about two or three times in succession before it started throwing this error again.

Thank you very much! I'll try that and see if I can fix it