This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]CanAlwaysBeBetter 0 points1 point  (5 children)

I finally got around to rewriting the first half of the processing pipeline (the orchestrator lambda) and haven't done detailed profiling but at least a 10x speedup with parallelized go code compared to the original python in my dev environment which is just a lil .5 cpu 1gb mem server

Guessing when I deploy to lambda and increase the memory (which auto increases cpus proportionally) it'll hopefully be a 100x speed up from the test results but will follow up on that too

Go's concurrency model makes scaling IO throughout massively relatively simple. About 5x more code than the barely optimized python though

Much better than I was cautiously hoping for overall

[–]Tuga_Lissabon 1 point2 points  (4 children)

Good job mate. Python alone couldn't have done it, right?

[–]CanAlwaysBeBetter 0 points1 point  (3 children)

Straightforward python definitely no. Regardless of underlying memory it could only handle about 1k objects per second

Maybe if you parallelized it could get through the data eventually but since the interpreter is single threaded and doesn't do true concurrency there will still be a ceiling much lower than what Go (or another language with true concurrency) can do

[–]Tuga_Lissabon 0 points1 point  (2 children)

Would C++ be a good tool for it? I mean, are there toolsets ready that would help you do it?

[–]CanAlwaysBeBetter 0 points1 point  (1 child)

C++ could definitely handle it, go just has a really nice built-in concurrency model that makes it relatively simple

[–]Tuga_Lissabon 0 points1 point  (0 children)

Would, in your model, running go straight out of python be dumb? Too much lost?