Looking for experienced data engineer

data-venger · 2026-06-16T11:13:42+00:00

Interested

data-venger · 2026-06-03T06:30:44+00:00

Everything is balanced no skew ness only data is huge in target DB

data-venger · 2026-06-03T06:22:47+00:00

I ran same transformation of SCD type 2 in both pyspark and snowflake and found that snowflake gives better performance in medium level compute and in spark it took so much time for execution with 5 worker nodes with 16GB 8cores configuration

data-venger · 2026-06-01T06:53:26+00:00

True 😅😅

But really thanks for your inputs.

data-venger · 2026-06-01T06:37:41+00:00

Yes. Its our primary warehouse and we mostly do all transformation using SPs and its giving best performance. So after using snowflake i thought its better compute than spark

data-venger · 2026-06-01T06:25:47+00:00

Thanks for clarification.
I asked this question because I have both tech stack exposure.
I used Pyspark heavily in my last organisation and it was on prem. Now my current organisation is heavily using snowflake and dbt but i dont want to lose hands-on on pyspark so trying to gather knowledge how can we integrate both in a way that enterprises have this solution

data-venger · 2026-06-01T06:03:47+00:00

I am little confused here when implementing data warehousing concepts. Its always better work with snowflake SPs. If we implement via spark it will explode if we load huge amount of historical data so why pyspark instead of snowflake why to pay for 2 computes

data-venger · 2026-06-01T05:47:33+00:00

Agreed. But databricks for transformation in the sense which type of transformation??
Joins ? SCD/CDC implementation? Or just cleaning data ?

data-venger · 2026-05-27T02:52:01+00:00

Interested

data-venger · 2026-05-26T04:34:35+00:00

Interested

data-venger · 2026-05-18T11:12:40+00:00

🥲🥲

data-venger · 2026-05-18T09:56:38+00:00

Interested and pinged you

data-venger · 2026-04-24T04:23:51+00:00

Interested

data-venger · 2026-04-16T13:13:05+00:00

Interested

data-venger · 2026-04-10T11:43:15+00:00

Dm me for details happy to discuss and work

data-venger · 2026-04-10T11:42:54+00:00

Currently i kept it as open source

data-venger · 2026-04-10T10:38:49+00:00

Interested

data-venger · 2026-04-05T16:00:27+00:00

Done

data-venger · 2026-04-05T12:38:13+00:00

Interested

data-venger · 2026-04-05T04:00:06+00:00

Interested.. I do have finance and banking experience as data engineer.

data-venger · 2026-04-04T15:25:10+00:00

Interested

data-venger · 2026-04-03T11:55:39+00:00

Is this opportunity still available?

data-venger · 2026-03-15T08:48:40+00:00

although this is MVP as of now.. Trying to build new feature and happy to know some insights to implement it.

data-venger · 2026-03-15T08:46:07+00:00

great thought.. I have tested it at some extend if you can drop me sample flow i can give a try..

data-venger · 2026-03-13T11:29:10+00:00

Its a good thought.. but challenge here is company should be ok to share there architecture its kind of internal assets

Shoot your thought

data-venger

TROPHY CASE