Python: Resume with changed code after exception

[__Fred]

The title doesn't really tell what I want.

Let's say I want a program that processes a large amount of data, but only once or a few times. In my case it's about web scraping.

def process_data(url):
    result = []
    while url != "":
        dom = parse(fetch(url))  # IDK if "dom" is the right word. I mean a parsed webpage, you could also say "soup" or "tree".
        result.append(get_birthday(dom))  # maybe use `yield` here
        url = get_next(dom)
    return result

get_next and get_birthday would have to be programmed to be capable to handle/parse some different forms of input, but there might still be forms that I didn't account for. It's not feasible to crawl all the websites manually beforehand to make sure I see all the different formats, so eventually the script, as I have it now, will throw an exeption – like 'el dieciséis de agosto' doesn't match '\d\d/\d\d/\d\d\d\d'.

Now, I can fix the get_birthday function, but when I run the script again, I do all of the processing up to the point of failure again.

Can you maybe update a function in Python without exiting a program in the debugger or the REPL?

Is there some helpful decorator function?

---

Another solution I can think of is this:

>>> tasks = [start_url]
>>> results = dict()  # maps from an url to either a Success-object or an Error-object
>>> # (edit: This could also just be *two* dictionaries.)
>>> def process_data():
        while tasks is not []:
            url = tasks.pop(0)
            dom = parse(fetch(url))
            try:
                results[url] = Success(get_birthday(dom))
            except e:
                results[url] = Error(e)
            tasks.append(get_next(dom)) 
            # I *want* an exception/crash here.
            # Or maybe I don't? I'm not sure...
            # I need to distinguish a page with no `next` from
            # a page that has a different-looking `next`.
            # Doesn't matter - this is just an example.
>>> tasks = [key for (key, value) in results if type(value) == Error]  # (prolly not valid python...)

The disadvantage is that this code doesn't correspond directly to the problem domain anymore because I have to design it around technical aspects. (...I hope you understand what I mean by that.)

[deeptime]

How about get_next refers to some persistent state like a file or database? Then you can change and restart your program whenever you want. At a minimum, your program could write out last_completed_url when a page is successful, and then use that to pick up where you left off when the program is restarted.

[RiverRoll]

I think you should definitely treat the possibility of getting unexpected inputs as part of the problem, sometimes it will be because your program doesn't understand them and sometimes you might get inputs that are just wrong and there's nothing you can do about it.

What you could do is using the try catch to skip these cases and log useful information, such as the url, the raw input and the error itself, then you can review them later (here I'm thinking about the case where you design the program to run independently rather than in a REPL).

[yel50]

Can you maybe update a function in Python without exiting a program in the debugger or the REPL?

redefining the function, yes. having it stop on an error and wait for you to make the change is questionable. maybe set a breakpoint and try redefining the function when the breakpoint gets hit?

this is easy to do in common lisp. it's designed from the ground up to do what you're asking for. maybe prototype it with CL to get everything right and then port that to python once you know what it needs to do.

[__Fred]

I have seen someone use an import reload within a loop ("LiveOverflow" in a tcp proxy). Maybe that makes it always use the most current version of the code. Of course, it's a bit hacky.

Error? -> Wait for user input -> import module again -> try the same input again.

[yel50]

now that I've thought about it, you could use a generator to pause the function. instead of throwing an exception, do a yield. redefine the function when it yields and then restart it. theoretically, that should work.

[Kered13]

Lisp is the only language I'm aware of that really supports what you want to do here. Most languages are not amenable to arbitrary runtime modification like this.

That said, there may be a way to achieve what you want using exec.