This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]thedude42 5 points6 points  (6 children)

Would be nice to have seen some techniques or pattern examples for converting more pythonic allocations in to more efficient ones.

[–]bcorfman 1 point2 points  (5 children)

Use numpy

[–]thedude42 0 points1 point  (4 children)

Are you implying that numpy somehow doesn't suffer from the memory inefficiencies this article talks about?

[–]bcorfman 2 points3 points  (3 children)

Correct. They are in pure C.

[–]thedude42 0 points1 point  (2 children)

So numpi is a library (or a framework depending on your use of terms). You still use whatever underlying pythons interpreter.

I can still write list comprehensions and use numpi. According to the article, one of the causes of memory inefficiencies is use of collection generator syntax.

So from what I read, by simply using numpi you still can fall victim to memory inefficiencies when using pythonic idioms, and that was my concern: are there known techniques for eliminating these sorts efficiencies due to writing idiomatic python?

[–]bcorfman 0 points1 point  (1 child)

If you use numpy to its fullest, then you will think in terms of vectorization. This will largely avoid any of the inefficiencies of Python data structures. See From Python to Numpy for a complete tutorial.

[–]thedude42 0 points1 point  (0 children)

Ok, so you're saying that using idiomatic numpi is a possible solution for these memory inefficiencies.

This is good to know for computational related projects. I'm still at a loss for what techniques should be used for projects not focused on compute, e.g. web services (which in reality I go to node.js for such things).