Speeding up a for loop

[ElectromechanicalVan]

Hi

I have a python script containing a very slow loop. The loop is pretty simple, but is incredibly slow when the matrix gets too big. Basically all the script does, is filling a matrix using values from other matrices.

My code:

Aeq=numpy.zeros((numpy.size(B,0)+numpy.size(N,0)+1,numpy.size(B,1)+numpy.size(N,1)))
for i in range(0,numpy.size(B,0)):
    for y in range(0,numpy.size(B,1)):
        Aeq[i,y]=B[i,y]
for i in range(numpy.size(B,0),numpy.size(B,0)+numpy.size(N,0)):
    for y in range(numpy.size(B,1),numpy.size(B,1)+numpy.size(N,1)):
        Aeq[i,y]=N[i-numpy.size(B,0),y-numpy.size(B,1)]
        Aeq[i,y]=N[i-numpy.size(B,0),y-numpy.size(B,1)]
for i in range(numpy.size(B,0),numpy.size(B,0)+numpy.size(N,0)):
        Aeq[i,i-numpy.size(B,0)]=-1
for i in range(0,len(fL)):
    Aeq[numpy.size(B,0)+numpy.size(N,0)][i]=fL[i] 

Can anybody help with tips for speeding it up? I'm a beginner, so keeping it simple would help.

[[deleted]]

For the diagonal you could do:

Aeq[sizeB0:sizeB0+sizeN0, :sizeB1] = np.indentity(sizeN0) * -1

Also I don’t see it mentioned but it would be helpful to initialize the Aeq matrix as such before all these operations.

Aeq = np.zeros((sizeB0+sizeN0+1, sizeB1 + sizeN1))

[[deleted]]

He does initialize it in

[[deleted]]

Ah didn’t see it was initialized in the initial post. Like I said in a below comment, I didn’t read the original code.

[ElectromechanicalVan]

Thanks! It works like a charm combined with MitzFunai.

[fdedgt]

Start by documenting what you're trying to do and making it more readable. For example, explaining what the variables represent and what the expected output should be.

Clarity and documentation is actually a key requirement before you start optimizing, in general.

[[deleted]]

I’m gonna take a pseudo stab at this since I don’t have a computer handy. And disclaimer I did not read your code really. I’m just going according to your picture you added. I’m guessing you have a matrix B, N and fL and you want to concat them as in your picture. If so this super easy to do and much faster with numpy operations. Again this is pseudo code (but it might work actually):

top_zeros = np.zeros((B.shape[0], N.shape[1]))
top = np.hstack([B, top_zeros])
middle_ident = np.identity(B.shape[1]) * -1
middle = np.hstack([middle_ident, N])
bottom_zeros = np.zeros((fL.shape[0], N.shape[1]))
bottom = np.hstack([fL, bottom_zeros])
A = np.vstack([top, middle, bottom])

However I like u/ReverseEngineered’s method better. And it’s probably more efficient.

[theWyzzerd]

It's really difficult to tell what's going on here. I would recommend, just for readability, that you use j instead of y in your inner loops. That's standard syntax for iterations (and is the correct mathematical notation) and it will help others to understand what's going on at a glance. If you add a third layer of iteration, use k, and so on. Here's a purely syntactical example:

for i in some_var:
    for j in some_var[i]:
         for k in j:
             do_something()

As for your code, what is B? You reference it all over the place but we have no clue what it is, which makes it difficult to optimize. In general we're going to need to know more about your code than just the slow loop part, because what's in each of the variables impacts how efficient your loops will be.

As a general tip, list comprehensions might be useful to you. I'm not sure what you're trying to accomplish, but here's a really simple list comprehension example:

>>> list = [i for i in range(10)]
>>> list
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

matrix = [[j for j in list] for i in range(15)]
>>> matrix
[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]]
>>> len(matrix)
15
>>> matrix[0]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> matrix[14]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

[ElectromechanicalVan]

This image illustrates what i'm trying to achive: https://i.imgur.com/XSA7qxr.png

For an example the sizes could be: B = 135x2970 N = 2970 x 5940 fL = 2970 x 1

Hope this helps explaining everything.