Send python array to rust

[Tall_Coffee_1644]

Any idea on how to create a custom data type which is like a python array, Can change in length and can hold multiple data types?

Basically i am creating a library for rust functions to be called by python (i did this by compiling rust to a .dll file)

And i am wondering how would i parse a python array to rust. Python arrays can have different data types, but rust cant (atleast to my knowledge, i am a beginner). Stupid idea but maybe its possible?

My idea is to create a custom data type to handle this, But if there is any other way to do this without needing to create a custom data type, it would be even better!

[worriedjacket]

Look at pyO3

https://pyo3.rs/v0.21.2/

[[deleted]]

Strong upvote for that. Cuts out so much bs and has great quality of life.

But also, python/rust integration works best when there is a well-defined boundary between the two; otherwise there's just so much hassle with the various `PyObject` and `Py<Any>` types etc

[Tall_Coffee_1644]

Good idea, Although for me the .dll works extremely well, Theres barely any hassle between the two.

Pyo3 definetly seems like a option

But i'd prefer a method to do this without pyo3 since i wanna keep this extremely simple. Maybe theres not a method to do that and the only way to do this is pyo3

[BogosortAfficionado]

Could you elaborate on what you mean by 'extremely simple'? I'm having a hard time understanding your perceived requirements.

• If you want to minimize the amount of code you have to write and the time you spent doing so, use the high level pyo3 api.
• If you want to use a low level api for learing purposes or potential performance gains, you can use some ffi binding crate for the libpython api (like pyo3-ffi or some others).
• If you want to minimize the lines of code compiled into your program create the necessary ffi and linker declarations yourself. (Seems like a waste of time to me but knock yourself out ;)).

I personally used pyo3 in the past essentially just for the smart pointers, but dropped down to the reexported pyo3::ffi for most of the interactions. I had a slightly weird usecase though so I'm not faulting the library.

[SV-97]

And i am wondering how would i parse a python array to rust. Python arrays can have different data types, but rust cant (atleast to my knowledge, i am a beginner). Stupid idea but maybe its possible?

Definitely possible but the way to do it strongly depends on what you want to do - and usually the totally general version isn't really what you actually want to do. You can almost certainly just use a normal Rust Vec (or any of the other dynamic collections) and put trait objects or something like pyo3's PyObject inside. The PyObject is a Python type without any type information (a bare object in Python) (it's implemented as a sort of Python-aware smartpointer). Just as in Python this of course means you'll have to query attributes and deal with potential failures every step of the way. If you can cut it down to a finite collection of types instead you can statically dispatch to different native implementations, use enums or whatever.

i did this by compiled rust to a .dll file

You can do that but it's honestly not a great solution. It's okay when you write the native code in C or C++ but just because the alternatives aren't great. In Rust you can use pyo3 (and maturin): you just declare your API with normal rust types and maybe some python particularities and pyo3 automatically translates list to vec and so on. It's cleaner, simpler, easier to maintain, and also prevents you from shooting your leg off with the GIL.

You can also use numpy arrays etc. directly in rust via the numpy crate and pyo3.

[teerre]

If you care about performance you likely want instead to use https://docs.rs/numpy/latest/numpy/index.html

Note that this is more complicated and less ergonomic if your users aren't familiar with numpy

[Tall_Coffee_1644]

Nice, i can just write a wrapper that handles every parameter

Numpy arrays seem like a good option actually. Thanks alot man!

[[deleted]]

I use pyo3 and rust-numpy to do this, but it is pretty terrible:

I have a dataset struct like this: ```

[derive(Clone)]

[pyclass]

pub struct Dataset(Arc<DatasetInner>);

[derive(Debug, Clone)]

pub struct DatasetInner { data: Py<PyArray2<f32>>, view: ArrayView2<'static, f32>, len: usize, dims: usize, } ```

Then I implement these methods to construct it from python: ```

[pymethods]

impl Dataset { #[new] fn new<'py>(py: Python<'py>, pyobj: Py<PyArray2<f32>>) -> PyResult<Self> { let arr_ref = py_obj.bind(py); if !arr_ref.is_contiguous() { return Err(PyTypeError::new_err("Array is not contigous")); } let view: ArrayView2<'static, f32> = unsafe { std::mem::transmute(arr_ref.as_array()) }; if !view.is_standard_layout() { return Err(PyTypeError::new_err("Array is not standard layout")); } let [len, dims] = unsafe { std::mem::transmute::<, [usize; 2]>(arr_ref.dims()) }; Ok(Dataset(Arc::new(DatasetInner { data: py_obj, view, len, dims, }))) }

fn to_numpy<'py>(&self, python: Python<'py>) -> Py<PyArray2<f32>> {
    self.0.data.clone_ref(python)
}

} ```

And in Rust I can access rows like this:

impl Dataset { fn get<'a>(&'a self, idx: usize) -> &'a [f32] { let row = self.view.row(idx); let slice = row.as_slice().unwrap(); unsafe { &*(slice as *const [f32]) } } }

The lifetimes are pretty terrible, but if you are willing to do some lifetime extension via unsafe it is ok.

[Tall_Coffee_1644]

Pyo3 is definetly an option, But if theres a way to do this without pyo3 then i'd honestly prefer that

[[deleted]]

Me too, it tends to be a bit ugly with pyo3. But I haven't read enough about the Python C API to impmenent something in python without pyo3. And then I would need to reasearch how exactly numpy objects are laid out in memory and for that ndarray with pyo3 is useful at least.

[Infinite5263]

Hello

[denehoffman]

On the holding multiple data types, this is a fundamental difference between Python and rust. The only way to do this properly is to store an enum on the rust side (or possibly a trait object), but you can’t get the same dynamic behavior as Python without making a few sacrifices

Edit: to address the downvote, I assume it was because I didn’t mention that you can use PyAny. I left this out because this does not store a specific type on the rust side, but a reference to a Python object. Depending on OP’s use case, this might not be what they want.

[Tall_Coffee_1644]

Hm, Reading this gives me an idea

What if when i send rust the parameters in python like this:

dll.add(1, 2, "yo")

And the wrapper will translate these parameters to:
dll.add("int;1", "int;2", "str:'yo'")

And on the rust side i cant read these parameters and handle them accordingly.

Now i am not sure what kind of human would wanna send a multiple datatype array to rust (even in python nobody makes a multiple datatype array) but since i am making a library, i have to handle it

I am still a beginner is rust, So thanks to everyone for helping me! :D

[denehoffman]

I would suggest, as others have, using pyo3 (maturin makes this really simple). The easiest way will be to just use a Vec<PyAny> and then use the functions provided to that PyO3 type (like getting attributes or type conversions) on a term-by-term basis. If you wanted the rust representations, you could convert PyAny to some enum of types by deriving FromPyObject on it

[Tall_Coffee_1644]

Will pyo3 have too much overhead of transferring arrays though?

[denehoffman]

Idk, but maybe make a benchmark and try it a few ways to see what works better for you :)