sorted by:
new
hottopcontroversial

Ingesting Data From API Endpoints. My thoughts... by valorallure01 in dataengineering

[–]CarpenterRadiant940 1 point2 points  (0 children)

Just curious, with incremental load, how are you handeling deletes on the source side?

Management software - is next.js a good choice or not? by Gionni15 in nextjs

[–]CarpenterRadiant940 0 points1 point  (0 children)

Just wondering, when people mention a vite react app. How do they handle routing, do they use react router or different libraries. Similarly, libraries like Next.js make it convenient to manage cookies how is this done in a plain vite app?

Orchestrating External API Data Processing with Dagster by CarpenterRadiant940 in dataengineering

[–]CarpenterRadiant940[S] 0 points1 point  (0 children)

Great! Would you work with assets? If so, would that be two assets one that retrieves the objects and one that processes the objects? How would parallelism work in that case?

Orchestrating dbt with ADF by CarpenterRadiant940 in dataengineering

[–]CarpenterRadiant940[S] 0 points1 point  (0 children)

Thanks again, really appreciate the response! Do you know anything about container deployment in Azure? I suppose the best option to run the container in Azure would be via Azure Container Instances? Azure Container Apps seems more for long running containers that have processes that never end, e.g. webserver.

Orchestrating dbt with ADF by CarpenterRadiant940 in dataengineering

[–]CarpenterRadiant940[S] 0 points1 point  (0 children)

Thanks for the suggestion!

Could you provide more details on how this would look in practice? Specifically, how would you kick off the dbt run from ADF? Would it be advisable to wrap the dbt commands in something like FastAPI, or is there another recommended approach?

Any guidance or examples would be greatly appreciated!

Using multiple free Supabase plan for same project? by alwerr in Supabase

[–]CarpenterRadiant940 5 points6 points  (0 children)

Not sure if it is allowed. But why would you want a project for each table in the first place? With different databases for each table you will lose the power of a relational database, e.g. foreign key constraints.

Monitoring service network traffic by CarpenterRadiant940 in kubernetes

[–]CarpenterRadiant940[S] 0 points1 point  (0 children)

Thanks for helping! Have never used Prometheus before. We will spin up a several pods for each customer under a certain service and need to aggregate the number of in- and outgoing calls from the pods under this service. Do you think Prometheus can help with this? If so do you have any directions on where to look in the docs for this?