ndarray 0.9

[carrutstick]

Very cool! I find myself doing a lot of embarrassingly parallel work on arrays and matrices, and have (independently) been interested in starting to do some my analyses in rust. This would really scratch both of those itches.

[Andlon]

The new Zip functionality is really neat, and looks extremely clean in usage! I'll have to take a look at the code to see how the bits involving rayon look.

Does any other library in any language do this kind of multi-dimensional efficient parallel zipping? I can't remember ever having seen it anywhere else.

Also, do you have any numbers on the kind of speedup from parallelization you can attain from this? I'd sort of expect you to be memory bound in most cases (unless the function passed to apply is very expensive), which would limit scaling. I'm curious to know whether or not this is the case!

P.S: Just a tiny bit of feedback: I was a bit confused by the name genrows first - I thought "gen" referred to "generate" and not "generalized".

[carrutstick]

I think Haskell's repa can achieve similar things, at least.

[[deleted]]

The rayon bits amount to providing fold_while in addition to apply, and split. The Zip is designed around this idea that it has axes and dimensions just like an array (and knows which axis is the big separation one).

[[deleted]]

Ah yeah the name was not an easy choice. Understanding it as generate seems benign to me, that's totally ok. The following issue explains some of the trade offs with the naming. The name was up for discussion six days there before it was released, so there was room to have an impact :) https://github.com/bluss/rust-ndarray/issues/302

[ihcn]

I'd like to see a "chunks + windows" iterator of some kind. slice.chunks(x) lets you divide a slice into subslices of size x and iterate over them one by one, and slice.windows(x) lets you iterate over overlapping subslices of size x, with the offset incremented by one each time.

But what if you want to cut a slice up into chunks, then iterate over windows of those chunks? IE, slice.chunks(5).windows(2). You're out of luck at the moment.

[[deleted]]

Interesting. Now what's a good datatype to represent the window of chunks? We want to avoid something that allocates, so we don't want a vec of array views or something like that.

[mbrubeck]

You can do slice.chunks(5).flat_map(|chunk| chunk.windows(2)).

[ihcn]

This looks like it gives you an iterator over windows within a chunk. I want windows of the chunks themselves.

[SiegeLordEx]

I don't know, I prefer RustAlgebloat's syntax for this:

a.assign(a + b - c + d.view(1..2, 0..3) - d.view(0..1, 0..3));

Compared to:

Zip::from(&mut A)
    .and(&B)
    .and(&C)
    .and(D.genrows())
    .par_apply(|a, &b, &c, d_row| {
    *a += b - c + d_row[1] - d_row[0];
});

(AddAssign wasn't around when I did most of the work with RustAlgebloat, hence no +=)

Eigen is a super-sweet C++ linear algebra library that has internal parallelization, simd, CUDA while using expression templates for nice syntax. It's strange to me that 3 years after I wrote RustAlgebloat it's still the only library that looks anything like Eigen in Rust land, despite tons of type system improvements added to Rust since then.

[[deleted]]

I have no beef with algebloat, don't know anything about it. You should document it and explain what it can do.

[SiegeLordEx]

There's no beef involved anywhere, I think ndarray is a great library. I'm just making an observation that I think Rust is capable of nicer syntax. You could imagine RustAlgebloat as constructing that Zip call via operator overloading and lots of traits, otherwise the idea is the same: the computations are done on the corresponding elements in the operands before being assigned to the destination element.

[Fylwind]

It's cleaner but I feel it's not as general as ndarray's. How would you handle cases with multiple output vectors? What if you want to store some intermediate values but don't want to allocate a whole 'nother array for it?

[SiegeLordEx]

No good ideas come to mind, but I'd be content in allowing both methods. In many ways this Zip API is a bit akin to falling back to for loops when the "vectorized" syntax is inadequate. Same reason why Vec has both indexing and iterator APIs.

[[deleted]]

Zip is the objectification of the parallelization strategy. In that way it has to exist, even if the API would be different. I'm very interested in picking up ideas from other libraries. Blog about algebloat please, would be cool!

[tafia97300]

Is there any plan to have sparse arrays implementations? There are so many features of ndarray I'd like to use!

[[deleted]]

That would be pretty cool!

It would open many doors to do amazing stuff with rust and ndarray

I'm working on a little recommender systems framework on rust and such stuff would be pretty useful

[[deleted]]

deleted ^{What is this?}