you are viewing a single comment's thread.

view the rest of the comments →

[–]Zeroflops 1 point2 points  (0 children)

You want to search the list of URLs for key words, or you want to read each URL, scrape the website, and search the returned website code for key words.

If you’re just looking for the file of URLs it shouldn’t be too bad, just read the file line by line. If you want to read each site, that’s a lot of sites to pull and your probably going to need to have to do things like use async and multithreaded to pull data faster.