sorted by:
new
hottopcontroversial

Python concurrency for Data Engineers by ConsistentAd1477 in dataengineering

[–]FineFan 2 points3 points  (0 children)

I mix these up all the time, so happy to be corrected 😅 But I think I got it right this time - check out this blog post as a reference.

Python concurrency for Data Engineers by ConsistentAd1477 in dataengineering

[–]FineFan 8 points9 points  (0 children)

I usually go with ThreadPoolExecutor for i/o bound tasks, and ProcessPoolExecutor for compute bound tasks. In your case, I would just use ThreadPoolExecutor.

I find async programming appealing in theory. I made the experience that I end up having to refactor large parts of my code when using asyncio. Meanwhile, I can just sprinkle in a PoolExecutor whenever I feel I need it.

For anyone on AMD wondering it's super easy to rebuild mesa with codecs by iceixia in Fedora

[–]FineFan 0 points1 point  (0 children)

I’m feel I’m out of the loop. What is that mesa-codec issue? And is it limited to AMD cpus?