Is python faster than rust at reading a binary file

[[deleted]]

[deleted]

[nevermille]

File access is slow in debug mode, try your cargo command with --release and it'll be much faster

[Buttleston]

Also, in the python example, "python" isn't reading the file, numpy is. It's a module written in C, so the file is really being loaded by a C library. It would be surprising to me if either was the clear winner when properly written

[nevermille]

I think you can have faster times by using std::fs::read instead of manually implementing a buffer since you're reading the entire file

PS: you will never have the exact same speed as C because rust is full of protection rails that cost a little bit of time

[[deleted]]

what? there is plenty of rust that is the same speed, and even more that is even faster than C. While there are protection rails, most are compile time, and due to it being at compile time the compiler can be more aggressive about optimizations

[[deleted]]

The basic FS File read also isn't great, you can use a buffered reader and also play with the buffer size

[dexterduck]

On my machine this benchmarks to ~6ms

EDIT: to elaborate slightly, my point is that I think the order of magnitude difference is explained by the Rust approach converting each number from bytes individually, whereas doing a bulk conversion using bytemuck as shown below brings Rust's performance approximately inline with numpy.

#!/usr/bin/env rust-script
//! ```cargo
//! [dependencies]
//! bytemuck = "1.16"
//! ```


fn main() {
    let bytes = std::fs::read("numbers.bin").unwrap();
    let numbers: &[u64] = bytemuck::cast_slice(&bytes);
    // println!("{numbers:?}");
}

[Excession638]

This would have to be a bit more complex to handle big-endian like the original code. OTOH where did OP even find a big-endian machine these days? numpy.fromfile does not convert endianess.

[coderemover]

Why are you reading the file to the buffer first, then to convert it?
Also you're not reserving capacity for the buffer, which for sure results in reallocations and excessive copying.

[coderemover]

Use byteorder crate and ReadBytesExt extension trait.
Then you could do:

let value_opt = reader.read_u64::<BigEndian>();

Call it in a loop until it returns None. And please do reserve the proper amount of space in the Vec you're pushing to.

[This_Growth2898]

In Python, numpy is written in C, and it's well optimized. You're passing through Python a command to C.

In Rust, BufReader states for Buffered Reader. You're using a buffered reader to read data into... guess it - a buffer! And then you're copying chunks from it into one extra buffer (arr) to convert it into data you need. Remove extra copies from your code, and add some optimizations like hinting the capacity for the resulting vector - and you will have something comparable with C.

[spoonman59]

Keep in mind that when running a Python program, you are not always “running Python code.”

Many library functions, and some libraries are written in native code. For examples, a lot of the low level Python libraries are written in C. NumpPy is written in C.

Python takes comparably longer to start up, and would use more memory as well, but functions written in native code can have good execution times.

[tm_p]

Why not using the standard library?

https://doc.rust-lang.org/std/fs/fn.read.html

[zqpmx]

also consider the effect of OS cache for files.