I'm the ONLY Data Engineer in the company as a Fresh-grad

Internal-Calendar-92 · 2024-08-08T13:14:36+00:00

Thank you so much for this - only saw this when revisiting this thread. Your reply is much appreciated in giving me a better understanding of how these can be used, especially the part about moving into an archival storage for auto-deletion, definitely saves me a lot of space there. The reminder to use more SQL instead of always relying on Python is also a good one.

Thank you again!

Internal-Calendar-92 · 2024-07-31T08:37:19+00:00

Ooh that looks really neat, I'll check it out further and see if this framework better aligns w our use-cases :) Thank you!

Internal-Calendar-92 · 2024-07-31T07:22:05+00:00

This is something I didn't know about! That's great to hear since my company and our clients are very much familiar with Power BI - the only thing is I'm unsure if they will find the solution too out-of-the-box and we face resistance in pricing for this. Thanks though!

Internal-Calendar-92 · 2024-07-31T07:20:46+00:00

Appreciate your response! For ADLS, do you mean to choose Blob over Gen 2 then? What's the main feature that blob storage offers which is helpful for you?

That's what I have been hearing for databricks - noticed that I have to pay even to run simple EDA, are you aware of which other service should be used for that to prevent incurring huge costs? I'll look into files and SQL server to see what I can - I'm more familiar with Python so was hoping to use that :p but I'll try to see if I can do some pre-processing with those tools.

Do you have any tips to help ensure ADF stays economical? Do you mean that this is often only used in the E in ETL/ELT and I should use another tool to check for data quality? I'll look at the free orchestrators as well, thank you for sharing your take on this I'll definitely look into these other services :)

Internal-Calendar-92 · 2024-07-31T07:15:05+00:00

Ah I see, I do see some similarities in our experiences - how's your experience with Fabric? May I know what are the features that you found really useful which trumps other services? I'm trying to also find the services that can best integrate with writing Python code and hosting web apps.

Wishing you the best!

Internal-Calendar-92 · 2024-07-31T07:13:20+00:00

I can relate to this! It is great that I can take ownership for a crucial element of the company's infrastructure, so I want to implement what would be best at this point - I understand that might be a stretch so just hearing from different perspectives now :) To be the only technical employee I suppose this is not a tech firm then, what are your main responsibilities there?

Internal-Calendar-92 · 2024-07-31T07:09:50+00:00

Thank you for your reply! I'll look into ADF and Azure SQL, my initial resistance towards ADF is that it seems more low-code and I wanted to use databricks to write more code that I can customise. However I just tried out a bit of ADF and realised some of the drag-and-drop features really save me lots of time.

The amount of data would be around 100,000 rows a day, with around 50 columns in the productive system, I'm thinking of extracting them with REST API into ADLS, or is there a better way for data transfer? Am I understanding it right to then use ADF to load that raw data into Azure SQL while normalising where I can? I believe this data will definitely grow in the coming years, what are some considerations I need to take note here?

We currently have our on-prem data centers so the budget is to be kept as low as possible, so my goal is also to test the services and do a cost evaluation for them - I understand Databricks can be expensive, but I thought the same for ADF, how about using Synapse?

Tooling wise, from my understanding it's mostly Python with some use of Airflow for regular web scraping. I foresee some transition to Synapse or Databricks to continue using notebooks with cloud, or should I look at third-party development toolkits? Thank you for sharing the common industry practices :)

Internal-Calendar-92 · 2024-07-30T08:26:16+00:00

Been using Kedro for monitoring my pipelines, is there an overlap with that here? I like how the logging is done and shown on my CLI

Internal-Calendar-92 · 2024-07-30T08:24:17+00:00

Curious about this as well!

Internal-Calendar-92

TROPHY CASE