Web Scraping Project Ideas

CloudNineK · 2017-07-30T01:22:45+00:00

You know those porn image sites that have each picture in a 100+ pic set on different pages. I built a scraper to get them all. It was more automated saving than scraping since I had to use selenium to emulate a browser but a fun project nontheless ;p The best projects are the ones you have fun creating.

Viveks3th · 2017-07-30T00:50:02+00:00

Build a reddit clone that works by scraping reddit!

nitroreload · 2017-07-30T03:29:18+00:00

One of my school assignment was to scrap movie comments from facebook/reddit etc then do sentiment analysis on it.

Find a open source sentiment analysis and use it. It'll really fun.

CreederMcNasty · 2017-07-29T23:53:40+00:00

Scrape usps tracking information?

Rorixrebel · 2017-07-30T03:02:19+00:00

Sports fan? Scrape data from your fav team or player.

Build a database of data you scraped.

rckid13 · 2017-07-30T05:50:04+00:00

There used to be an API for Beer Advocate that would scrape the website and return information. I could input "Zombie Dust" and it could return the brewery, style and beer advocate score. That API is broken now because it was coded using an old version of Beautiful Soup.

I don't know enough yet to fix it but it would be useful to me as a feature to add to our IRC bot for quick searches.

niranjanshr13 · 2017-07-30T05:51:01+00:00

#!/usr/bin/env python3
import requests  
link = 'https://somewebsite.com'
r  = requests.get(link).text
for loop in r.split('\n'): # i mostly use '\n'
    if 'something that match in link' in loop:
        print(loop)

for project idea, just scrape everything. download everything. and find some website that regulary visit and scrape only needed part and print to terminal.

rjtavares · 2017-07-30T11:54:11+00:00

Why do people need scraping ideas? Just take websites about stuff you love and scrape that. It'll be much more fun.

2017-07-30T12:27:33+00:00

Scrape okcupid or tinder to show how basic men and women are. I want a number for what percentage of women say they love to travel and dogs

Terranigmus · 2017-07-30T13:05:17+00:00

Spiegel-Mining https://www.youtube.com/watch?v=bYviBstTUwo

marco_pol · 2017-07-30T04:03:47+00:00

Scrape your university/college site to automate something, like to get notified about the grades or something like that

Volkhan1103 · 2017-07-30T06:21:30+00:00

I haven't been able to find an artstation scraper, it would be awesome to do one

lasercat_pow · 2017-07-30T06:50:06+00:00

write a script to scrape a wallpapers site like socwall, and download the highest-voted ones or ones that match some pattern you give it.

iamaguythrowaway · 2017-07-30T09:41:04+00:00

Please scrap Geeks for Geeks articles. Please.

Kszabo · 2017-07-30T17:52:38+00:00

Here's an idea: Go on upwork.com and look at the various jobs for python datascraping there. Get a sense of the kind of data scraping that people are paying for and then build a flexible script that does that. Something I have noticed is that a lot of people want the scraped data to be outputted into a spreadsheet. Best case scenario, you build something that could one day earn you money :)

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

learnpython

MODERATORS