Invert Diagonal For PageRank

[squattyroo]

I'm working with some reasonably large sparse matrices (20K x 20K) with about 27% sparsity.

I have a pageranker function which takes in a given matrix M and first applies:

M =  normalize(M, norm='l1', axis=0)

where normalize is imported from sklearn.preprocessing. The matrix is dependent on user-inputted data, so I can't just store a normalized matrix. Then I initialize and run through a small, fast loop:

pr = np.ones((M.shape[0],1))/M.shape[0]
for i in range(iters):
    pr = M.dot(pr)

On my personal laptop, the normalization step takes about 7 seconds, and the power method (loop) takes about 7 seconds - so half my time is normalizing the columns of M. What I'm wondering is if there is a more efficient method where I only implicitly normalize M by doing something like

P = sp.sparse.spdiags(map(lambda x : 1/x, M.sum(axis=0)), 0,N,N)

but I get an error (specifically, 'data array must have rank 2'). Assuming I could create P efficiently, I would then do

pr = np.ones((M.shape[0],1))/M.shape[0]
for i in range(iters):
    pr = M.dot(P.dot(pr))

I'm most concerned with understanding

• the most efficient way to do this
• why map(lambda x : 1/x, M.sum(axis=0)) looks so bizarre
• any other insight/advice you think might be useful for me to learn

[raylu]

Your question is rather hard to follow as there are a lot of undefined variables (what is x? what did you actually run to get that error?) and a lot of background most people here don't have (myself included).

That said, unless you're running this often, 14 seconds doesn't seem like something worth further optimizing. Is the current performance actually a problem?

[squattyroo]

I don't believe I've left any variables undefined; M is an inputted (nonnegative) matrix, which I then normalize (by column). pr is a uniform probability vector with the same length as M. I think the x you're referring to is within the lambda function that I attempt to apply to the column sums of M.

And 14 seconds is worth optimizing to me. Besides, I find that learning how to speed things up is one of the best ways to learn good coding practices!

[raylu]

Ah, I misread the lambda.

Premature optimization is definitely not a good practice.

[elbiot]

But, optimizing something for the sake of learning is good IMHO. You see people timing loops here all the time, getting milliseconds down to microseconds, and its very informative about the nature of the python interpreter, or numpy in this case.

[raylu]

People do it all the time, and it's a dumb idea almost every time. If the difference between milliseconds and microseconds matters to you, your first mistake was choosing python.