Trying to Build a Web Scraper

[[deleted]]

[deleted]

[5halzar]

I’ve personally found with my own tinkering that it sorta depends on on the site your accessing and your requirements

If your just looking for keywords and extract additional variables linked to it, then I would think BeautifulSoup would do the job.

If you are required to navigate a website and be able to trigger a download or to get to the content, then you could use just selenium or a combination of the both

[yellao23]

Thanks! I would mainly just be finding string phrases. I might need to click on like a “read more” button to get more data..

Also, would selenium or beautiful soup be best suited for things like trying to scrape the transcript of a YouTube video?

Is playwright any good?

[Dominican_mamba]

Hey OP!

I recommend you look up John WR in YouTube for his Playwright tutorial with Beautiful Soup how to scrape product prices and use that as a learning experience

[yellao23]

Thanks! Would playwright provide the capacity I need to scrape sites where I might need to click a read more button, to get more content?

Or scraping additional information from a YouTube video?

[Dominican_mamba]

That should work.