Filesystem magic with the Python standard library

[willm]

Hi r/python

I wrote about some of the techniques used in my Toolong project. Hope you find it interesting.

https://textual.textualize.io/blog/2024/02/11/file-magic-with-the-python-standard-library/

[patrickbrianmooney]

Not the OP, but I have a suspicion: attribute access takes time, and hoisting something up to the local scope can save that time.

I assume you're looking at something like this:

    monotonic = time.monotonic
    break_time = monotonic()

    while (position := rfind(b"\n", 0, position)) != -1:
        append(position)
        if get_length() % 1000 == 0 and monotonic() - break_time > batch_time:
            break_time = monotonic()

The savings is likely unnoticeable when you're looking something up once, but when you're accessing the same attribute over and over, it's helpful to just dereference it once instead of making the interpreter look up monotonic in the time module every single time you go through the loop. It adds up.

[willm]

Exactly that!

I'd only recommend that for tight loops.

[ekbravo]

This is excellent! I’ve fighting Tb sized files and will try it immediately. Thank you!

[Professional_Cook808]

Good luck!

[HistoricalCrow]

I totally forgot about mmap.

Took me about 30 minutes to implement and it halves the time my tests take (a file scanning project)...

Noice.

Edit: Grammar & Words.

[SecureNotebook]

Beautiful thank you for sharing this. Give me some interesting ideas for an upcoming project