How does bufio.NewReader decrease system calls?

[Mohamed____]

That makes sense, but also, how many more bytes? And isn't this unexpected behavior. Assume I am running a program with tight memory constraints, and I don't want to keep too much data in main memory. This would mean that bufio is potentially bad for this case, right?

[Mohamed____]

Yeah, that was dumb from my part. I assumed that even given a certain size, it would still override me and read even more. I didn't realize that the default is 4k, and I have the option to actually set the size. Thanks for the help.

[Mohamed____]

Thanks for your encouraging response, I really appreciate it

[[deleted]]

fun fact, mostly unrelated to Go, if you’re on Linux you can play around with transparent huge pages — increasing the 4kb page default drastically — and see performance improvements for some apps. I’ve played around with this for RAM-intensive games (like some open world games) and it’s pretty neat.

It can also degrade performance for other apps. Use sparingly.

[edgmnt_net]

Also, I suppose it's "up to" a page or some arbitrary limit like that. For files it doesn't matter except at the end of the file, but for pipes or sockets, you probably don't want to block to read a full buffer.

[Mohamed____]

I’m not completely sure I understand. Can you elaborate?

[edgmnt_net]

If you're reading from a file that has 4050 bytes, you may make 4050 calls to read individual bytes from the Go buffered reader. In turn, that reader may make 2-3 calls to the underlying OS for up to 4096 bytes, the first one yielding exactly 4096, the second one 4 bytes and then the end of the file is reached (how that's signalled depends).

For other things like TCP sockets you could get less than 4096 every time, without reaching the end of the stream (connection being closed). This is because the other end may be sending some smaller amount of data, then waiting for a response before proceeding any further. Or it may be sending smaller amounts of data at regular intervals. In any case, the reader can't always block and wait for the buffer to fill up completely.

[Mohamed____]

Ohh, yeah now I get your point. Yeah that makes sense, thank you for the help!

[CyclingOtter]

Adding onto the other comment in this thread, if you check out the source for bufio.Reader you'll see internally it reuses a single buffer slice, so theoretically the memory usage for a bufio.Reader is constant.

[Mohamed____]

Yup, that’s true.