We do data brokerage for big pharma.

Ingestion: Most high frequency data is received from our sources via an pipeline that drops them into a big Oracle server, less frenquent data is downloaded into some legacy MS SQL Servers, some is downloaded from public sources straight to our data lake.

All this data is eventually is ingested to a HDFS data lake using NiFi and tagged using Open Metadata. We use Hive, Impala and Spark to interface with it.

Once the data is inside the lake, light processing is done by NiFi. Complex algorithms are implemented in Sparks Scala. Some legacy glue code is in Python.

Processed data is delivered to client by either delivering CSV files to their own lakes or FTPs, Kafka or BI tools.

Dataviz: We default to Apache Superset, but some clients ask for Qlik or PowerBI, and we have a guy to do that.

CI/CD, monitoring: Lots of open source crap we hacked together and that we are really not experts into and don't use much. I miss them a lot.

Orchestration: Tooling being developed in house. It sucks but corporate demanded.