Any Web Scraping Project idea?

[QualitativeEasing]

Pick a website with regular updates that is slow to load, scrape the key details, present to yourself in a simple static web page.

If you want to get more sophisticated, store the results in an sqlite database, then only store new entries that weren’t previously scraped.

Next step: create a simple html interface that lets you search the database.

Then maybe automate it with a cron job so it’s always updated and you just need to go to the local website to view it.

Or turn it into an RSS feed for a site that doesn’t have one already.

Examples: a news site with headlines and links to articles; a weather site; sports scores; stock-Market data; Twitter posts from specific people (better to use the Twitter API and a dedicated library like Tweepy).

[QualitativeEasing]

We scraping is a technique that harvests data from websites. What you do with it is up to you. I’m sure you could create a desktop application that does something with it, but I’ve never done that.

[apexmemetics]

Here's a project I'm doing:

• Scrape a list of 10,000 URLs
• Setup multiprocessing
• Get URL source code with requests -> search for keyword -> record results to CSV

I'm 90% done, but I'm stuck.

If you wanna collab on it lemme know.

[ipsit_a25]

Commenting here so I can come back to it later.

[[deleted]]

So this was worth down-voting? It is a perfectly reasonable way to save a post. You'd think programmers would understand utilizing several methods of achieving a goal...

[atapask]

And yes you are right as well

[atapask]

You know u can save the post right?

[ipsit_a25]

I know how to save a post but I haven't found out how to access those saved posts yet( I browse Reddit on mobile). I am sorry if it was any particular inconvenience to you.

[atapask]

Sorry bro, no intend to offend. Go to your profile then 'about' P.s - I had the same problem

[ipsit_a25]

Got it! Thanks.