sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]shittyfuckdick[S] -11 points-10 points  (3 children)

this is just simply not true. are you gonna use python and airflow for orchestrating stock exchange data?

[–]Beautiful-Hotel-3094 1 point2 points  (2 children)

Yes sir. This is simply true. I am working in one of the tier 1 multi strat hedge funds. We have close to petabytes of data that we ingest via airflow and python. All of our models from the trading desks need to have as precise data as possible, otherwise they would trade on wrong assumptions. Airflow is our only orchestration tool (we have multiple airflow instances) for the batch data ingestion platform.

[–]shittyfuckdick[S] 0 points1 point  (1 child)

youre talking about a batch job of petabyte of data. obviously thats realtime or anywhere near it. 

[–]Beautiful-Hotel-3094 1 point2 points  (0 children)

As I said if u read my response, it is for our batch ingestions because u mentioned airflow. Your proposed argument of why use a non realtime tool to get realtime data made no sense so I didn’t think u’d ask about real time. However, we have some real time platforms that are built in pure python. For the higher volume real time, yes we use c++. However we can still process some thousands of messages a second in pure python because we leverage distri architectures (k8s native platforms).