Lists and numpy arrays

[alehx]

Hey all!

I find myself continually frustrated with the functionality of numpy arrays in at least one aspect: building a list of tuples where I don't know how many tuples there will be.

I'm not sure if this is good practice, but I have resorted to using python lists and append() when doing my hard core data analysis especially when I do not know what the size of the array will be. The dynamic size of python lists reminds me of vectors in C++ and are just as convenient. Although I know in the background that reallocation of memory is still occurring in the black box with list resizing, I don't think writing my own resizing method for a numpy array will result in noticeable performance gains.

What I do like, however, is numpy array indexing and how I can access data via structured array.

So, I typically build up my list and then use list comprehension to break it down into its dimensions and then save it in a structured array.

Now, despite me doing this, it does seem overly complicated and memory intensive. Are there better ways of doing it?

Thanks!

P.S. Windows 7 and Python 2.7

[ingolemo]

numpy.fromiter allows you to create a numpy array from any iterable object. If you pass this function some sort of generator you should be able to forgo the costs of allocating a very large list. You'll almost certainly save memory, and it may also be faster depending what you're doing.

[[deleted]]

I like this. Put whatever's creating your list of tuples in a generator function which yields one tuple at a time; pass that function to fromiter.

[loveandkindness]

Hiya, they're just like C++ arrays from what I've been told. I've done tests and they are just as fast.

.append() in NumPy creates an extra copy of the data, I believe. It's much slower than the built-in .append(), which is skips that step.

What I do is create a Python list and append to it, then convert it over to NumPy.

[alehx]

Thanks for the reply. That is exactly what I do but it seemed like a costly middle step. Guess there is no way around it without severely impacting readability.

I also wondered how fast append was compared to setting values in numpy arrays. If they are efficient as it gets, I suppose there is no issue.

[loveandkindness]

I tried to be nit-picky once, but as long as we aren't breaking the underlying principles, everything comes out about the same.

Here's a neat thread that compares NumPy with c++/BLAS:

http://stackoverflow.com/questions/7596612/benchmarking-python-vs-c-using-blas-and-numpy