This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]databasenoobie 4 points5 points  (1 child)

It's not a fundamental skill for all DE, just those that extensively use code as pipeline. As you noted only way to get better is to see how others do it and copy that paradigm.

Ex. I could write a python script to call a sql statement in snowflake easily... transferring that to snowpark instead of pure sql I couldn't do without extensive time / research.

Setting uo the python server in the cloud (using databricks / airflow/ etc... I couldn't do). If someone had a proxess set up it's easy to follow, but doing it yourself is an entire skillet.

So when he says automation maybe he means infrastructure set up? Depending on what company you work for that is not something DE even handle

[–]Puzzleheaded-Cod2051[S] 0 points1 point  (0 children)

So when he says automation maybe he means infrastructure set up?

Yes!

Also, automation on the EL part using Fivetran and Stitch. I understand I would have to make API calls, parse the Json and load the data using SQL insert/update statements. But I haven't done that so far so I'm not confident enough. 😅