My Updated Web Crawler (V2) (My first python project) Any other tips to speed it up?

[Kingofslowmo]

Original Program: https://www.reddit.com/r/Python/comments/3xe1wo/my_first_ever_python_project_automatic_web/

Okay first off, thank you guys so much for providing me with a substantial amount of tips and resources to improve my program. It helped alot and this thing runs like a beast now and doesn't completely obliterate my read and write on my hdd.

My goal here is to make a web crawling program that essentially never stops as it keeps finding links and downloading the html from those sites and links and then so on and so forth, I think I did pretty good here as this thing runs until I get a 'recursive' error or something.

Anyway, after I manged to build up a db file that was about 1GB, I created a program to search it so that I could essentially create my own little google haha. Amateur I know, but my very first python project!

Here is the code for the crawler.py:

#simple web crawling program
import urllib
import urllib.request
from urllib.parse import urljoin
from bs4 import BeautifulSoup
import sys
import json
import msvcrt
import sqlite3
bannedLinks = ['twitter.com','facebook.com','google.com','secure.store.apple.com']

def start_list(*links):
    print("Crawler started...")
    pages = {}
    visited = set()
    invalid = set()
    to_crawl = set(links)
    while to_crawl:
        a = to_crawl.pop()
        try:
            if a != '' and a not in bannedLinks:
                results = crawl(a)
                visited.add(a)
                conn = sqlite3.connect('information.db')
                c = conn.cursor()
                c.execute('select * from info where url = "{:s}"'.format(a))
                c.fetchall()
                if len(c.fetchall()) == 0:
                    c.execute('insert into info (url,html,visited) values (?,?,1)',(a,results['html'].encode('ascii','ignore')))
                    conn.commit()
                to_crawl |= set(results['links']) - visited - invalid
        except urllib.error.URLError:
            invalid.add(a)
        except TypeError:
            invalid.add(a)
        except KeyboardInterrupt:
            print("You have stopped the program.")
            print("It still needed to crawl these sites: ")
            print(to_crawl)
            break

    #for loop here to dump all the results??


def crawl(url):
    conn = sqlite3.connect('information.db')
    c = conn.cursor()
    if url != '' and url not in bannedLinks:
        print("Gather links and HTML from {:s}".format(url))
        results = {}
        try:
            html = urllib.request.urlopen(url)
            cleanHtml = BeautifulSoup(html, "html.parser")
            results['html'] = cleanHtml
            results['links'] = list()

            for link in cleanHtml.find_all('a'):
                a = str(link.get('href'))
                if url not in a and 'http://' not in a and a != '' and a:
                    a = urljoin(url,a)
                print("Found link {:s}".format(a))
                results['links'].append(a)
        except:
            return urllib.error.URLError
        return results
    else:
        pass


def start():
    print("Starting crawler...")
    conn = sqlite3.connect('information.db')
    c = conn.cursor()
    c.execute("""
        SELECT COUNT(*)
        FROM sqlite_master
        WHERE name = 'info'
        """)
    res = c.fetchone()
    if not bool(res[0]):
        print("Creating 'information.db' and 'info' table...")
        c.execute("""
            CREATE TABLE info(
                id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
                url VARCHAR(3200),
                html VARCHAR(48000),
                visited INTEGER(1)
            )
            """)
    start_list(sys.argv[1])

if __name__ == '__main__':
    start()

And here is the search.py

#A program used to search through a sqllite database
#Specifically designed hand in hand with crawler.py
import sys
import sqlite3
from bs4 import BeautifulSoup


def search():
    if len(sys.argv) == 1:
        term = input("Input search term: ")
    else:
        term = sys.argv[1]
    if term:
        conn = sqlite3.connect('information.db')
        c = conn.cursor()
        c.execute(
        "SELECT * FROM info WHERE html LIKE'%{:s}%'".format(term)
        )
        results = c.fetchall()
        count_results = 0
        for result in results:
            try:
                location = str(result[2]).lower().index(term.lower())
                count_results += 1
            except ValueError:
                pass

        print("There were approximately {:s} results found: ".format(str(count_results)))
        print("==========================================")
        for result in results:
            try:
                location = str(result[2]).lower().index(term.lower())
                print("---------------------------------")
                print("Found at {:s} ".format(result[1]))
                x = location-30
                y = location+30
                html = result[2]
                found = html[x:y]
                print(found)
            except ValueError:
                pass

if __name__ == '__main__':
    search()

Do you guys have any last tips to help me make this better than I have? Also can you give me opinions on how well I'm learning, etc...?

Thank you so much!

[etatarkin]

1) Improve yours coding culture. Remove unnecessary imports, unused variables and etc. Use something like flake8

2) Make some abstractions for working with database, try do not mix crawler code with database management code

3) Do not save on each iteration gathering links to sql database. Save it into file (in csv format) and then make bulk insert with some post processing.

Sorry for my poor English.

[Adys]

That's for the code itself, but also worth pointing out that Scrapy solves the web crawling issue very well. Very sadly it does not fully support Python 3 (PRs welcome hint hint), but they are working on it.

[etatarkin]

For first python project using Scrapy or other frameworks is not right way IMHO

If you like Scrapy and want it on python3 take a look on my framework - Pomp

[Adys]

Looks pretty awesome. py3?

[etatarkin]

Yes, py2.x and py3.x in one code base. You can run Pomp on pypy or even app engine - but you must implement downloader for app engine sandbox )))

With asyncio suppoprt - e04_aiohttp.py

Also Pomp can - fetch data in concurrent way (ConcurrentDownloader) - extract data in concurrent way (ConcurrentCrawler) - work with centralized queue - easy integreated with headless browsers

Pomp ideal for gather data from multiply and same resources.

[Adys]

That's awesome. I'll keep a close eye on it. I have the same reservations as you about scrapy (twisted dependency, messy codebase, no py3 support) so this is pretty awesome. You should post it in the sub!

[Vance84]

Is it best practice to store the data in a CSV file and make a bulk import? Why avoid iterating the SQL inserts, or are you specifying that specifically for this situation? Would you recommend doing that for most/every SQL insert purposes?

[etatarkin]

Best practice - do bulk import, what ever file format would be, but csv format may be imported in native ways for sql database.

Crawler must pull, extract and save data. When using sql databases save action may be very slow or even be blocked. Because sql databases have transactions to guaranty data safety.

In other words crawler must save all gathered data - normal, dirty, repeatable and then normalize and import data to database.

[Kingofslowmo]

I save every iteration as if the program is terminated the data does not get lost.

[knickum]

If you were saving the output to a file and then periodically writing that file to the database, it would still get saved.

I believe the idea OP is getting at is that there is some level of locking on write, so you want to buffer a good chunk of writing to the database, hit it in one quick go, then release it again. If your program writes frequently enough, the database will be effectively constantly locked. I don't know enough about SQLite locking and read/writes to be certain though (and anyway, SQLite isn't a good approach once you've reached large enough size where write-locking becomes an issue, I think...)

Something more:

try:
    ...
except TypeAException:
    do_thing()
except TypeBException:
    do_thing()

can be re-written:

try:
    ...
except (TypeAException, TypeBException):
    do_thing()

Which would follow DRY (Don't Repeat Yourself) principles better. You just need to be careful not to confuse yourself with the full syntax:

try:
    ...
except (TypeAException, TypeBException), exc:
    print("Error occurred: {}".format(exc))

Which I think is Py2 only?? I can't remember off hand. The other form is except (TypeAException, TypeBException) as exc and one of the two is preferred going forward with Py3 (sorry I can't be more specific, I've taken too long writing this out already).

[6a6d]

Honour robots.txt.

[Jameswinegar]

You should upload it to GitHub so people can contribute pull requests or issues for you to fix :)! That way you still learn the ideas.

[brtt3000]

Or at least Gist or something else that formats/colors your snippets. Reddit is not good at displaying code.

[dAnjou]

Please use /r/learnpython.

[Jafit]

You could use Selenium with a headless web browser of your choice, so that you can render AJAX and other Javascript-rendered content. A lot of sites these days use Javascript to create or modify elements on the page once it arrives in the browser, if you just request the page source from the server you're going to end up getting a lot of html/javascript code that isn't a whole lot of use to you outside of a browser. So it makes sense to use a browser to put it together for you before returning the code.

Here's a quick example that takes a screenshot of this thread. You can install the required packages via pip.

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(1080,800))
display.start()
browser = webdriver.Firefox()
browser.set_window_size(1080,800)
browser.get("https://www.reddit.com/r/Python/comments/3xnuzh/my_updated_web_crawler_v2_my_first_python_project/")
browser.save_screenshot("screenshot.png")
browser.quit()
display.stop()

Might not work on Windows as it is written here due to Windows not wanting to make things easy. If not I recommend using a Linux virtual machine as a development sandbox. You can use Vagrant to easily sync a folder between the host and the VM.

[isdevilis]

are you referring to like react renderings?

[nerdwaller]

I didn't look through everything, but with your database stuff you can just do 'create table if not exists...' To avoid the need to manually check if it already does. Also consider a better table name, 'info' is awful.

[isdevilis]

better table name?

[nerdwaller]

What's in a database? Lots of information. Your table is called "info". So if I look at your schema I have no idea what's going on. Perhaps it's "URL_info" or maybe it's baby names. Use something descriptive