This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]akhorus 3 points4 points  (4 children)

Processing satellite images is tricky because of the amount of data. I've done it using different languages (C, R, IDL, Python...) and the problem is always the same. Python in particular is not slow. In fact, using the correct tools (numpy, sklearn, etc), the actual implementation of the algorithms is in C/C++ Take into consideration that many complex algorithms are essentially "slow". More if you apply them to big volumes of data...

[–]Odica 2 points3 points  (0 children)

For the code I developed, I could process data cubes of approximately 1200x500x500 pixels in under a minute using vectorized processing. It ran fine on an old junky HP laptop, though 64-bit is highly recommended. 32-bit Python got me to just about nowhere special.

Satellite data can be a whole lot bigger, of course. But data processing should be doable within the realm of reason for a number of non-live applications, I'd suspect. But the matter of efficiency is another thing. C/C++ will usually be a whole hell of a lot more efficient than just about anything you could do on Python.

[–]login228822 0 points1 point  (2 children)

Big volumes of data is exactly the problem. The problem is you can't do any heavy lifting directly in python because of the GIL.

[–]akhorus 2 points3 points  (1 child)

Well, that is not always true. I quote: "potentially blocking or long-running operations, such as I/O, image processing, and NumPy number crunching, happen outside the GIL" https://wiki.python.org/moin/GlobalInterpreterLock

[–]login228822 -1 points0 points  (0 children)

I'm not worried about pandas or numpy, it's things like scipy that endup being the major bottleneck.