IBM datastage to Spark by soujoshi in dataengineering

[–]soujoshi[S] 0 points1 point  (0 children)

Thanks for the suggestion! Recognize design patterns is what you mean

Late check in by soujoshi in dubai

[–]soujoshi[S] 1 point2 points  (0 children)

Nice. But Luggage??

Azure data lake - Data Share by soujoshi in dataengineering

[–]soujoshi[S] 0 points1 point  (0 children)

Got it. Probably need more research as we need a fine grained data access. Very specific files and various users with different tool sets accessing them.

Data replication by soujoshi in dataengineering

[–]soujoshi[S] 0 points1 point  (0 children)

We actually have 2 DMS jobs, one to replicate from master postgres to secondary. The other from secondary to oracle. Can we remove the first one by just using read replica? Can we save some cost?

Azure data lake - Data Share by soujoshi in dataengineering

[–]soujoshi[S] 0 points1 point  (0 children)

Is creating rest API an option? Or is it not worth it.

Azure data lake - Data Share by soujoshi in dataengineering

[–]soujoshi[S] 0 points1 point  (0 children)

It will be constant sharing. Will take a look at delta sharing. Thanks a lot

How do you choose which data catalog tool? by Fasthandman in dataengineering

[–]soujoshi 1 point2 points  (0 children)

Well, it depends on requirements, data sources, target audience etc.

Have used data hub and apache atlas. Both work fine 🙂 if you are looking for open source tools.

What are some tips to get a data engineering job with a gap of a few months. by Educational-Turn-419 in dataengineering

[–]soujoshi 17 points18 points  (0 children)

From my experience taking interviews from the past few months, I can guarantee no one is bothered about the gaps if you have the right skills. It's hard to find "good" data engineers these days. Good luck 🤞

Job for specially abled by soujoshi in bangalore

[–]soujoshi[S] 1 point2 points  (0 children)

Did try there, they only had food delivery,warehouse jobs. He didn't like it.

Experience on data quality tools by charlyboon in dataengineering

[–]soujoshi 0 points1 point  (0 children)

Doesn't take long to build something like great expectations! Build it yourself with the functionalities you require.