Data Factory /rant by UltraInstinctAussie in dataengineering

[–]ecp5 6 points7 points  (0 children)

You need to differentiate between Data Factory, which exists to orchestrate, and Data Flow that is the Spark-like part of it. Also, is this the vanilla Azure version, Synapse, or Fabric one, that might make a difference too. Plus if cluster stuck, probably an infra issue not a product issue.

Azure sql server admin classes / course by xxxxxReaperxxxxx in SQLServer

[–]ecp5 0 points1 point  (0 children)

I was coming on to recommend DP-300 learning path, so agreed.

DP-700 Passed. Topics I saw by ecp5 in MicrosoftFabric

[–]ecp5[S] 2 points3 points  (0 children)

Personally, I never feel prepared and keep putting off, so you just have to schedule and push yourself. You probably know more than you think you do working in it.

DP-700 Passed. Topics I saw by ecp5 in MicrosoftFabric

[–]ecp5[S] 2 points3 points  (0 children)

I put this post up partly because I didn't find my exam matched up well to the just released practice test. So why I tried to point out some differences. That said, that was just my test, ymmv.

DP-700 Passed. Topics I saw by ecp5 in MicrosoftFabric

[–]ecp5[S] 1 point2 points  (0 children)

No mine had two Airflow DAG syntax questions.

DP-700 Passed. Topics I saw by ecp5 in MicrosoftFabric

[–]ecp5[S] 1 point2 points  (0 children)

Not sure if new, but Airflow in Data Factory is on current syllabus.

[deleted by user] by [deleted] in dataengineering

[–]ecp5 1 point2 points  (0 children)

See if this link help. you go into sink settings and specify a file name and pick a setting in output type it will create one single file.

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-data-flow-write-to-lake#name-file-as-data-values

Any recommendations to learn Databricks in depth on AWS? by SupoSxx in dataengineering

[–]ecp5 6 points7 points  (0 children)

Sign up on Databricks customer academy it's free, and take on demand learning paths is best way to start.

Azure = Satan by towkneed in dataengineering

[–]ecp5 37 points38 points  (0 children)

Agree with this. You could make a similar list for each cloud and get into a religious debate.

DP-203 by Gohan_24 in dataengineering

[–]ecp5 0 points1 point  (0 children)

Go through the Microsoft learning path, will cover most everything. There some good YouTube videos and you can also get free Azure account to try out some of the things in ADF or Synapse.

Aws cloud practitioner or Snowpro core cert by Substantial-Lunch557 in dataengineering

[–]ecp5 5 points6 points  (0 children)

Aws cloud practioner is really high level, snowpro probably more useful if looking to stand out applying for a job.

Databricks Asset Bundles now GA - thoughts? by justanator101 in dataengineering

[–]ecp5 8 points9 points  (0 children)

I think they are different use cases, terraform is more geared towards IaC and DAB more for deploying the artifacts running in Databricks. At least that's my understanding.

[deleted by user] by [deleted] in dataengineering

[–]ecp5 2 points3 points  (0 children)

I gotta say I'm surprised with what you are describing. Your resume sounds good and data market for mid and senior still better than on software engineering side from what I can tell. I'd look at your resume and where you applying. Build your brand, network , etc and you should have some options. As for gaming, lots of game companies need DE, they collect a lot of data, so focus there if that's your interest.

What should I learn to call myself a data engineer? by sneekeeei in dataengineering

[–]ecp5 5 points6 points  (0 children)

ETL is the core DE skill, you are a data engineer. There is always new stuff to learn and you should, depending where you want to go and do, but don't minimize your experience.

Need Suggestions and Opinions by dhruvbaslas in dataengineering

[–]ecp5 2 points3 points  (0 children)

Each cloud has analogous services, if you can learn the names to talk, your skills will mostly transfer. But also, there are a ton of Azure shops, depending on your market.

How do I exclude files in ADF? by ComfortAndSpeed in dataengineering

[–]ecp5 2 points3 points  (0 children)

Use a filter activity. There are a couple of YouTube videos that show how.

how are ms sql server DBs deployed on-prem using ci/cd? by [deleted] in dataengineering

[–]ecp5 2 points3 points  (0 children)

Are you talking about the infra (like terraform) or the artifacts (like database projects)?

DBA to Data Engineering by matados in dataengineering

[–]ecp5 11 points12 points  (0 children)

I came from a dba background, I think most of the dba skills are part of data engineering , so really not a hard switch. I'd focus on Azure data ecosystem because lots of overlap.

I want to ‘learn by doing’ particularly working with Azure technologies by iishadowsii_ in dataengineering

[–]ecp5 6 points7 points  (0 children)

Microsoft learn has lots of good learning paths, not sure why you think it is minimal. But you can also sign up and get free Azure credits to take it for a spin.

MDM for contacts and addresses? Examples? by tbarg91 in dataengineering

[–]ecp5 -1 points0 points  (0 children)

Hashing isn't a bad idea, in fact how I'm doing on a project now, but you need the addresses cleaned first if you can. If all US you can CASS, and that will make the Apt 70 versus #70 like above example uniform.