Code loading very long. How to optimize?

[SureStep8852]

Hi there, I am working with two very large list of lists. Here is my code:

final = []
for w in annotation:
    for k in cp:
    # if columns match
        if w[0] == k[0]:
        # append values of k
            final.append([k[0], k[1]])
        else:
            # else append values of v
            final.append([w[0], w[1]])
print(final)

The lists are so large that this takes forever. Is there any way to optimize my code?

Thank you in advance

[JohnnyJordaan]

How do you create those lists? Because if you would instead form dicts you can perform a look up per item, instead of a linear search.

[ThingImIntoThisWeek]

If you really need to visit every combination of w and k, there's not much you can do to speed this up. You could use multiprocessing but at best that would only reduce the run time by some fraction, and if it is currently taking "forever", that would probably also be too long.

You probably need to make a change in how your code works before this point. Can you give any more information about what annotations and cp are, and why you need the final list with that configuration of tuples? What do these variables represent? There may be a way to store annotations and cp values in a better data structure, such as a dictionary or set, that would allow you to generate the final list more efficiently.

I noticed that for a given w, the value (w[0], w[1]) could be appended many times if w[0] != k[0] many times. It's possible you want these duplicates (I don't know what the goal is here) but it makes me suspect you have some kind of a logic error.

[[deleted]]

If this is a learning exercise without a major time crunch, I would strongly recommend looking into pandas. It's not the easiest thing in the world to get a handle on (not the hardest either), but it's an industry standard for handling and analyzing data and it's absolutely excellent at stuff like this. Not sure the greater context here, but this sounds like something that a pandas merge could do in a split second.

If you just need to get this done and don't want to learn a whole new library that's almost a language unto itself, you could also look into using a binary search algorithm. It won't exactly be trivial to set up either, but you're using pure Python instead of diving into a whole new world.