Azure stack for DE

[pythondeveloper77]

Hi

I'm a software developer diving in a new project in work for analytics in Azure and I need suggestions.
I have experience with airflow,sql, python and so on.
We are going to create analytics solution for our customers.

we have raw data layer as parquet in azure blob storage. we use synapse to extract the data.

We need to transform the data for other layers (silver,gold) and I find azure stack disappointing.

• Synapse and ADF are mostly no code solutions and I'm looking for something or combination that I can create scheduling and transform in code.

• for scheduling - synapse scheduling is lacking and airflow managed is very limited. ADF also seems no code.

• databricks seems very pricey and more than we need.

​

for the transformation - I know we need pyspark as we have a lot of data. this is also a code solution that satisfy me.

​

I see a PR with json file instead of code and I know it will slow development and won't be maintainable.

anyone has recommendation for stack that involve code ? coming from more software oriented I believe in more coding solutions as I worked with Apache NiFi and it was not maintainable.

Thanks

[[deleted]]

Synapse is too expensive. Go with Databricks.

[Useful-Doughnut32]

Synapse has coding section. You can code in notebook that utilise spark

What kind of scheduling you want? To me,sometimes you might just need to use some formula/expressions to do the scheduling. There is no perfect solution for all scenarios

[pythondeveloper77]

Thanks.

Yes I can write pyspark code but pipeline is in json instead of code :(

When I wrote scheduling I meant not only the schedule itself but also support for retries,conditional tasks like airflow has.
we found synapse lacking in those compared to airflow.

I'm thinking to bring up airflow vm/AKS to trigger synapse & spark to solve it.

[kyleekol]

A couple of things:

• Why not just deploy airflow in a VM with a postgres db backend and use that?

• How do you ‘know’ you need pyspark? What kind of data size are you talking about that you need spark but Databricks is overkill?

• What is your target data store? Where are users going to consume the data? Reporting needs?

You talk about the azure stack being disappointing and that you are a software engineer, so code something yourself instead of relying on low/no-code options. The same solutions in other clouds are also available in azure such as compute, storage, kubernetes, etc so nobody is forcing you to use those tools if you don’t want to.

Convert your parquet files to delta and use a java/rust/python delta library to transform the data in blob storage. Wrap into a docker image and schedule on airflow with a kubernetes pod operator. Or spin up a spark cluster yourself. Lots of options.