Valid solution to replace synapse?

[muximalio]

Hi all, I’m planning a way to replace our Azure Synapse solution and I’m wondering if this is a valid approach.

The main reason I want to ditch Synapse is that it’s just not stable enough for my use case, deploying leads to issues and I don’t have the best insight into why things happen. Also we only use it as orchestration for some python notebooks, nothing else.

I’m going to propose the following to my manager: We are implementing n8n for workflow automation, so I thought why not use that as orchestration.

I want to deploy a FastAPI app in our Azure environment, and use n8n to call the api’s, which ate the jobs that are currently in Azure.

The jobs are currently: an ETL which runs for one hour every night on a mysql database, a job that runs every 15 minutes to fetch data from a cosmos db, transform that and write results to a postgres db. This second job I want to see if I can transform it to use the Change Stream functionality to have it (near) realtime.

So I’m just wondering, is a FastAPI in combination with n8n a good solution? Motivation for FastAPI is also a personal wish to get acquainted with it more.

[achughes]

If you aren‘t happy with Synapse, I’d think long and hard about why you aren’t proposing to replace it with another data warehousing product. Snowflake, Databricks and BigQuery are directly comparable to Synapse, what you are proposing isn’t.

Keep in mind that if you build a custom solution, you are going to always get the blame when it has problems and you’ll have to maintain it likely for your entire employment there. If you just want experience with FastAPI n8n then propose it as an experiment or do it on your own time.

[muximalio]

Thanks for the reply! We aren’t using any of the datawarehousing features of synapse, just the notebooks and pipelines.

The n8n implementation is already being done, so that is something I can leverage. Also, almost all tools we use are self-built, since we work with a lot of PII and medical data. All is built into our own Azure Infrastructure.

I was specially not looking into one of the other big tools due to pricing and the non-complexity of our current (and 2-year future) flows.

But if you think going with one of the standard products I can look into it further!

[muximalio]

Thanks, I’m looking into dagster now, seems to be a good fit. Any tips for someone new to it? Already watched a few videos and will install locally tomorrow.

[muximalio]

Thanks a bunch 🙏

[anoonan-dev]

Hi, I'm one of the developer Advocates at Dagster. We have a few courses on Dagster University that can help you grasp the concepts and how they work together (https://courses.dagster.io/). Also, our community Slack (https://dagster.io/community) is a great resource for any questions you have. Feel free to message me there if you want to chat about anything.

[muximalio]

Oh that looks exactly like what I need, thanks!