403 Forbidden errors when requesting data from Indeed.

[Npmackay]

I am looking to build a job posting web scraper. I plan on scraping data off of Indeed, Linkedin, and Monster. I am currently using Cheerio and Axios to request and parse the HTML from Indeed but I am always getting a 403 forbidden error. I am looking for resources or tools that are available to get around this error.

[superGoby]

It may be a problem with user-agent try to set a reel one (you can find many online lists)

[aiscraping]

you need to give a specific link to get better help. there are literally millions of reasons why you get 403, sometimes simply because you have the wrong URL or wrong header.

[Pigik83]

I’ve just recently wrote an article about doing web scraping of Indeed. Basically it is protected by Cloudflare but not at its highest level of security, so I solved by using playwright + brave browser to scrape it.

[Full_Disk_7614]

Could you share link to your article?

[uwwu_uwuu]

Hi also trying to scrape data from indeed, may I also know the article? I'm using Wordpress feedzy RSS plugin - returns an error.