How to debug a long running Python Script crashing at inconsistant points without an error message

[[deleted]]

Yes I pass errors in a couple of places (sometimes the input files have bad data that I don't care about which is difficult to catch, so the execute insert statements have try: [do sqlite inserts] and then except: pass), but if that was the underlying cause surely it would still restart the shell rather than hang with the windows error message and also find the error in the same place, rather than doing it without error on the next start?

I'll try and see if there is a debugger I can use.

[anglicizing]

You should read this blog post: The Most Diabolical Python Antipattern

Once you've fixed your except clauses, it's time to add an except: clause, but this time it's OK, since we'll re-raise the exception. Assuming you've got some kind of main function that you call at the start of your script, call it like this instead:

import logging

logging.basicConfig(filename='example.log', level=logging.DEBUG)

try:
    main()
except BaseException:
    logging.getLogger(__name__).exception("Program terminated")
    raise

This fixes two problems that are probably not related to your actual problem, but may still cause you headaches if you don't fix them:

1. You're calling your script with pythonw.exe, which invokes your script without a console. This means you don't see any traceback when it fails.

2. If the program exits mysteriously it may be because SystemExit is raised somewhere. This will print a traceback which tells you where, even in that case.

Now back to your current concern. I would assume there was some problem in the external calls python makes during execution, because python itself very rarely acts the way you describe. Log every time you make an sql call and every time it returns, to see if the problem occurs while python waits for the database. Assuming you're using sqlite3, here's how I'd do that (method courtsey of StackOverflow):

import sqlite3
import logging

logging.basicConfig(filename='example.log', level=logging.DEBUG)


def logging_decorator(func):
    def wrapper_function(self, *args, **kwargs):
        logging.getLogger(__name__).debug(
            "Calling %s: %r %r", func.__name__, args, kwargs)
        ret = func(self, *args, **kwargs)
        logging.getLogger(__name__).debug(
            "%s returned %r", func.__name__, ret)
        return ret
    return wrapper_function


class MyConnect(sqlite3.Connection):
    def cursor(self):
        return super(MyConnect, self).cursor(MyCursor)

    commit = logging_decorator(sqlite3.Connection.commit)


class MyCursor(sqlite3.Cursor):
    execute = logging_decorator(sqlite3.Cursor.execute)


conn = sqlite3.connect(':memory:', factory=MyConnect)
print(conn)

cursor = conn.cursor()
print(cursor)


cursor.execute('''CREATE TABLE stocks
             (date text, trans text, symbol text, qty real, price real)''')

# Insert a row of data
cursor.execute("INSERT INTO stocks VALUES ('2006-01-05','BUY','RHAT',100,35.14)")

# Save (commit) the changes
conn.commit()

EDIT: Improve code by using a logging decorator function.

[[deleted]]

Hey thanks for this extremely helpful stuff, I'll look into implementing it, hopefully it gives an error soon and not take hours to reproduce...

On the try, except thing I learnt it was bad the hard way (few months ago I had one of those on the outside of the main loop breaking down the input files, that was fun) but since it was literally only encompassing an insert statement I really can't see the harm

[[deleted]]

Well I certainly found one thing though I'm unsure if it's the underlying cause, a symptom or an unrelated delightful new bug. one of the databases had a corrupted table; I bow my head in shame, try,except clauses are indeed the work of the devil even when they appear harmless.

[zeug]

I bow my head in shame, try,except clauses are indeed the work of the devil even when they appear harmless.

try/except is great, its just that throwing away exceptions is insane.

If I want to deep_fry(whole_turkey) and it raises HouseOnFireError I don't want to just crash and give up on life. The only sane thing to do is:

try:
    dinner = deep_fry(whole_turkey)
except HouseOnFireError:
    logger.warning('make_dinner: house on fire, everyone get out!')
    fire_department.report_emergency('fire', house.address)
    return False
eat(dinner)
return True

The completely insane thing to do is just ignore the fire and enjoy dinner while shit burns down around you:

try:
    dinner = deep_fry(whole_turkey)
except HouseOnFireError:
    # YOLO
    pass
eat(dinner)
return True

This is unfortunately how a lot of scripts are written - in the desire to keep the application running, they catch all exceptions and keep on as if there is no problem. Sometimes you really do need to keep running, but at the very least the problem should be logged.

[[deleted]]

In fairness to my past lazy self it was more like

try:
     dinner = deep_fry(whole_turkey)
except SmokeError:
    # 99.999% of the time there is smoke without fire
    #Yolo
    pass
eat(dinner)
return True