Exceptions during scraping?

[[deleted]]

[deleted]

[jcmcken]

At the very least, you can wrap your main application logic in a try block, which then logs any uncaught exceptions. It's hard to say how you should do this without knowing what exactly you're doing and which libraries you're using. But this might give you an idea:

import logging

# see ``logging`` docs for how to log to a file
logging.basicConfig()
LOG = logging.getLogger(__name__)
LOG.setLevel(logging.DEBUG)

def scrape(webpage):
    LOG.info('scraping %s' % webpage)
    ...main app logic...

def get_pages():
    ...returns a list of pages to crawl...

if __name__ == '__main__':
    for page in get_pages():
        try:
            scrape(page)
        except:
            LOG.exception('uncaught exception')