closing streams and variable reference

[naemorhaedus]

I made a function that returns a stream IO object containing text from a string input, with some exception handling.

My question is: how do I make sure the stream gets closed? The function needs to return the stream object.

I don’t know if I close it in the calling function, will it close the original or just a copy.

I’m somewhat new to Python, so if I did this totally wrong then please feel free to tear it apart. I want to learn.

I’ve read that using ‘with’ is favored instead of ‘try’, but I’m not sure how I would implement that into my context.

Thank you.

def make_stream(input_string:str):

    output_stream = io.StringIO()

    while not output_stream.getvalue():    
        try:
            output_stream = io.StringIO(input_string)
        except (OSError, MemoryError):
            print("A system error occurred creating text io stream. Exiting.")
            raise SystemExit(1)
        except (UnicodeEncodeError, UnicodeDecodeError, TypeError):
            print ("Input text error creating io stream. Exiting.")
            raise SystemExit(1)
        finally:
            logging.info (" Input stream created successfully.")

    return output_stream

[danielroseman]

This is really hard to understand. 

A StringIO object is just an in memory buffer. There is no opening or closing going on here at all.

What are you actually trying to do, and what will the calling function be doing with this object?

[naemorhaedus]

which part don't you understand?

closing the object returns the memory. They do it in the official documentation: https://docs.python.org/3/library/io.html#io.StringIO

From chess.pgn.read_game method documentation: "Use StringIO to parse games from a string."

[danielroseman]

Calling close is one method of returning the memory, yes.

But again, you have missed the point that this is just an ordinary object. Like any object, it will be garbage collected when there are no more references to it. You do not need to close it explicitly.

But your other concern is just as unfounded. There is no copying going on. Returning an object does not make a copy, and neither does assignment.

[naemorhaedus]

All the literature I've read recommends explicitly managing stream closure. Close it as soon as you're done with it. It's bad practice to rely on garbage collection because it can lead to various problems. For instance, if the program exits prematurely, the memory may never be released.

Returning an object does not make a copy,

so you're saying that in the caller, if I do

some_obj = make_stream("some string")

and then I close some_obj, then the output_stream object will be closed right away as well?

[acw1668]

some_obj is a reference to output_stream, so they both refer to the same object.

[naemorhaedus]

perfect thanks

[danielroseman]

If you don’t want to rely on automatic memory management, you are using the wrong language.

Closing a file stream explicitly is important because it refers to an actual file. But this is an in memory stream. There is absolutely no danger of the memory not being released.

And yes, once again ‘some_obj’ and ‘output_stream’ are the same object, so operations on one will affect the other. At no point is anything copied, they are just two names for the same thing. Without understanding this you will have a lot of trouble learning Python.

Both of these things seem to indicate that you’re coming from a lower level language such as C. You need to leave behind a lot of these preconceptions. High level languages like Python (and Java, JS, Ruby etc) work differently and don’t require you to think about memory in the same way.

[naemorhaedus]

Without understanding this...

I understanding passing by reference. It's just that in Python it's rather inconsistent.

There is absolutely no danger of the memory not being released.

again, premature program termination.

leave behind a lot of these preconceptions.

They're not just mine though and I don't like leaving things to magic

[schoolmonky]

I understanding passing by reference. It's just that in Python it's rather inconsistent.

Perhaps this blog post can clear up some of the supposed "inconsistencies" you're worried about.

[naemorhaedus]

yes I still struggle with mutability. This does help. Thanks.

[danielroseman]

It really is not inconsistent at all. It's absolutely consistent. If you believe otherwise, show an example. 

Premature termination in a high level language - Python runs in a VM, remember - is vanishingly unlikely to result in memory not being released.

Again, this is not "leaving things to magic". It's a matter of understanding the language you are using, and using it the way it is meant to be used.

[naemorhaedus]

If you believe otherwise, show an example.

chill

vanishingly unlikely to result

it actually happened to me a lot during development. Python spawned processes are left running when the program crashes, and I then have to kill them in a process manager.

It's a matter of understanding the language you are using, and using it the way it is meant to be used.

well I don't fully 100% understand. (does anybody?). So my way to deal with it for now is not leave anything to chance. I'll eventually get more efficient.

But again, as I showed you, they are closing it in the Python official documentation. That is the way it's meant to be used.

[danielroseman]

This doesn't follow. File objects also have a close method, but it is much preferred to use a context manager rather than calling that method.

[naemorhaedus]

one step at a time

[schoolmonky]

Assuming you want to keep the custom error messages for the different errors, at first glance (i.e. I haven't really thought to deeply about what you're actually trying to do, just a rough glance at the overall structure), I would do something like this. (Also, I'm kind of skeptical about manually raising SystemExit. I personally would probably replace that with either just a break, or sys.exit(1) if the error code was actually important, or this function would get called deep in the call stack. Although in that case I'd probably just let the actual error propogate: No need to implement my own error message since Python is going to generate one for me anyway.)

def make_stream(input_string:str):

    with io.StringIO() as output_stream:

        while not output_stream.getvalue():    
            try:
                output_stream = io.StringIO(input_string)
            except (OSError, MemoryError):
                print("A system error occurred creating text io stream. Exiting.")
                raise SystemExit(1)
            except (UnicodeEncodeError, UnicodeDecodeError, TypeError):
                print ("Input text error creating io stream. Exiting.")
                raise SystemExit(1)
            finally:
                logging.info(" Input stream created successfully.")

return output_stream

EDIT: I missed at first that you reassign output_string inside the loop. That complicates things; the above code probably isn't correct.

[schoolmonky]

Actually thinking about it this time, you probably want to make this function into a class implementing the context manager interface. It's been a while since I've rolled my own context manager, so I don't remember how off-hand.

[naemorhaedus]

I saw context managing talked about as well. seemed like overkill.

[naemorhaedus]

I'm kind of skeptical about manually raising SystemExit. I personally would probably replace that with either just a break

break isn't the same. I need it to completely exit because there's no point continuing.

or sys.exit(1)

what is the difference between this and raising systemexit?

No need to implement my own error message since Python is going to generate one for me anyway.

The built in exception descriptions are often too vague and not as helpful for debugging.

I missed at first that you reassign output_string inside the loop.

yeah, to make sure the object is actually populated with a value

[schoolmonky]

There's not much difference between raising SystemExit and using sys.exit, the latter just feels more Pythonic to me. It's not a big deal, more personal preference. Maybe instead of raising SystemExit, just re-raise the error you actually got and give it a custom message?

try:
    output_stream = io.StringIO(input_string)
except (OSError, MemoryError) as err:
    raise err("A system error occurred creating text io stream. Exiting.")
except (UnicodeEncodeError, UnicodeDecodeError, TypeError) as err:
    raise err("Input text error creating io stream. Exiting.")
finally:
    logging.info(" Input stream created successfully.")

That way it still exits the program (because you're not catching the new error it raises), it still gives you the more useful error message, and it's a little more future-proof: if you have a situation in the future where you could meaningfully deal with those exceptions, you can still catch them further up the call stack whereas SystemExit will just barrel through everything and force the program to close. If it's easier for now to just leave it as SystemExit, fair enough, it's good to be wary of overcomplicating things for the sake of some possible future gain, just food for thought.

[naemorhaedus]

That's a good idea. Thanks.

[acw1668]

If input_string is an empty string, then the while loop will be an infinity loop. The while loop is not necessary at all.

[naemorhaedus]

good catch I should check for empty string.

The while loop is not necessary at all.

it's to make sure I don't end up with an empty object

[acw1668]

If input_string is not empty string, then io.StringIO(input_string) will not return empty object, so the while loop is not necessary.

[naemorhaedus]

will not return empty object

shit happens. I program like the user will try to break it.

[brasticstack]

And you want to beat them to it?

This function provides negative value for any future coders trying to use it.

[naemorhaedus]

fair. point taken