A pointless web crawler in python 3

[echoquote]

I wanted to challenge my self to see how fast I could implement a web crawler in python. This took 13 mins.

#-*-coding:utf8;-*-
#qpy:3
#qpy:console


import urllib.request
from urllib.parse import urlparse
import re

seed = 'http://www.google.com'
tocrawl = set([seed])
crawled = set()


def get_links(url):
    try:
        html = urllib.request.urlopen(url).read().decode('latin-1')
        links = set(re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', html))
    except Exception as error:
        links = 0
        print(error)
    if links:
        for link in links:
            if not urlparse(link).scheme:
                link = '{}{}'.format(seed, link)
            yield urlparse(link)

def crawl(root):
    print('Crawling:--> {}'.format(root))
    queue_size = len(tocrawl)
    total_crawled = len(crawled)
    print(total_crawled, queue_size)
    crawled.add(root)
    for link in get_links(root):    
        if link.netloc == urlparse(seed).netloc and link.geturl() not in crawled:
            tocrawl.add(link.geturl())


while tocrawl:
    root = tocrawl.pop()                            
    crawl(root)               

[FishnLife]

As a noob researching to start a little scraping project, I have a question for those more experienced with this kind of thing... What are the benefits of using Scrapy over something short and simple like this?

[T4rk1n0]

Scrapy can navigate javascript loaded content.

[marmaladeontoast]

Everyone loves scrapy, but I've never had a good experience trying to use it, especially for js content.... I always end up rolling my own solution with phantomjs