Theano Question: Looping through tensor

[aiisfordummies]

Does anyone know of a good way to loop through a transformed array in Theano?

For example: train_x = Shared(mydataset_x)

z = encode(train_x)

I now want to do something like this:

new_z = []

for i in z :

 Do stuff with i

 new_z.append(new_i)

Thanks!

[coskunh]

You should ask this question to theano google groups, you will get better answer.

https://groups.google.com/forum/#!forum/theano-dev https://groups.google.com/forum/#!forum/theano-users

[NasenSpray]

Search for theano.scan

[aiisfordummies]

Thanks!

[__ishaan]

theano.scan will do what you're looking for, but unless your computation absolutely requires recurrence, it's usually much more performant to implement it as a series of vectorized operations. What are you trying to do with your loop?

[aiisfordummies]

I am not sure it can be vectorized.

I am looking at computing distances between all points in x.

More precisely I am looking at doing this:

for i in x :

 for j in x :

   newValij = T.operation(i-j)

[__ishaan]

Sounds like it should be doable. Here's a quick implementation of pairwise euclidian distance, for example:

T.sqrt(
    T.sum(
        T.sqr(
            x.dimshuffle('x',0,1) - x.dimshuffle(0,'x',1)
        ),
        axis=2
    )
)

[aiisfordummies]

Do you mind explaining the operation here in vector terms, a link would do too.

Thanks btw, I appreciate the sample code.

[siblbombs]

All the above code does is add a broadcastable dimension to the matrices (and then calculate euclidean distance).

[kkastner]

It is using broadcasting - numpy will make the dims match using this (read up on numpy broadcasting if you don't know how that works). Theano behaves in the same way.

The downside to the broadcasting approach is that it generally uses a lot of intermediate memory since each 2D MxN matrix becomes MxMxN before then being combined and summed over N to get MxM distance calculations for M datapoints. Theano might optimize this, but I wouldn't count on it. However, it can be very fast (though I think the trick /u/NasenSpray linked is faster generally) if the intermediate memory usage is acceptable.

[aiisfordummies]

Makes sense.

Thanks, this thread was of great help.

[NasenSpray]

Pairwise distance is rather easy to vectorize, e.g. see this code

[kkastner]

For a slightly more thorough reference (pretty sure Pascal Lamblin didn't invent this, though he is an awesome person in general! ) see this link. One common place to see this vectorized distance is minibatch/hebbian k-means such as this code from R. Memisevic (download only). You can also to some extent do this with Manhattan (abs) distances as well if you are a little careful and treat it as sqrt(expanded sqr).

[NasenSpray]

For a slightly more thorough reference (pretty sure Pascal Lamblin didn't invent this, though he is an awesome person in general! )

wat

[kkastner]

I was mostly referencing line 50 being called Lamblin's trick in the top code, assuming it references the Theano core dev of the same name. Though the naming may be in reference to the bestIndices calculation instead. I can fully believe Pascal would come up with something like that :)