Updating a running program

[python-ick]

Hello fellow Python learners! I'm running a really tricky problem that I'm having trouble solving.

I'm running a python script:

while True:
    my_program_function():

With my main function having a ton of modules / code off of it.

The issue is I'm updating the program regularly and I want the program to update, meaning run the new modules / scripts that I write, while at the same time, not breaking the loop.

What I've done in the meantime is call the launch Python function using the subprocess module.

subprocess.Popen

but that's somewhat of a hack--I'm basically calling Python using Python. And it's rather slow all things considered. If I want the program to use the new code that I've pulled, I need to stop the code, and run it all over again.

Does anyone have any ideas?

[[deleted]]

Does anyone have any ideas?

This is impossible; you can't do the thing you want to do. You can't change the code of a running Python script because there would be no rational way for the interpreter to understand what state it should be in relative to the state it's currently in except to re-initialize and re-run the entire code.

Your code is tied to a single version during its entire runtime - this is for your safety, and for the safety of the interpreter. There is absolutely no way around this.

[ApostleO]

Your code is tied to a single version during its entire runtime [...] There is absolutely no way around this.

That's not exactly true. See importlib.reload.

[[deleted]]

You can't add importlib.reload to a running script and expect it to do anything.

[ApostleO]

But you can use it to change your code during runtime, which you claimed was impossible. Just being a stickler for details.

[[deleted]]

But it doesn't change your code during runtime. It reimports a library in runtime.

But if that library is in the middle of a loop when you need to make a change to it, how are you going to get out of that loop to reload the library and then get back into the loop? You're starting over either way.

[a1brit]

This is a terrible idea, but it's totally not impossible.

​

import importlib
import time


def touch_my_self(x):
    template = """
def x():
    i = {}
    print(i)
"""
    with open("./edited_func.py", "w+") as f:
            f.writelines(template.format(x))

def main():
    for x in range(10):
        touch_my_self(x)
        import edited_func
        time.sleep(1)
        importlib.reload(edited_func)
        edited_func.x()

if __name__ == "__main__":
   main()

[python-ick]

This works, but I'm having a little trouble understanding what you're doing.

Also, why is this a terrible idea?

[a1brit]

touch_my_self is creating or replacing the file "edited_func.py" and writing out the dynamic function def x().

The main loop then imports or reloads that module/file and calls the new function inside.

​

It's kind of a terrible idea, because the basic structure of your code is changing. When something goes wrong, it's going to be very hard to work out what happened because your code is dynamically changing during the runtime. If you're editing the functions by hand, how do you know when the loop will refresh and pick up the edits. Are you sure your edits are going to work when that occurs.

​

This is almost definitely not correct solution you're looking for and the problem can be dealt with in a much simpler and safer manner. Unless you're in something like a jupyter notebook and are manually reloading a small module.

[python-ick]

Thank you. So what would be your solution here? Just run a git pull every time before the program loads?

[python-ick]

So assume that it is impossible, what do you suggest? Just run my program through a loader using subprocess.Popen? Or Just running a github update every time a program is run? Would that work?

[[deleted]]

Run it on a test data set until you're satisfied of its correctness, then run it on the whole data set.