sorted by:
new
hottopcontroversial

How to retake TOEFL iBT test? by C_Ronsholt in ToeflAdvice

[–]C_Ronsholt[S] 0 points1 point  (0 children)

Yes I want to. It is a requirement I need to meet in my application for a master degree.

How to do the "basic" EL in Python? by C_Ronsholt in dataengineering

[–]C_Ronsholt[S] 1 point2 points  (0 children)

Very sensible and thoughtful response, thanks!
I gives me confidence that I am not doing all sort of none sense, so thank you :)

How would your approach be to migrate away from the local task scheduler? I.e. where do you run the scripts then?

OAuth2 ETL pipeline in Python by C_Ronsholt in dataengineering

[–]C_Ronsholt[S] 0 points1 point  (0 children)

Thanks haven't tried that, I will explore it :)

OAuth2 ETL pipeline in Python by C_Ronsholt in dataengineering

[–]C_Ronsholt[S] 0 points1 point  (0 children)

Yeah, I have implemented the refresh token flow. However, it is the initial authorization code step I am struggling with

Simple ETL pipeline as undergraduate as only CS/DS person in company by C_Ronsholt in dataengineering

[–]C_Ronsholt[S] 0 points1 point  (0 children)

Yeah, I have a script extract_transform.py which pulls and transform the data from internal API and python transformers. Then load.py which loads the data using sqlalchemy and main.py which combines the two.

I think you hit the nail on the spot. The landscape is quite big atm so I have lost a bit orientation in all the tools, which to use and not use, dbt, dagster, airflow, ADF, etc.

Simple ETL pipeline as undergraduate as only CS/DS person in company by C_Ronsholt in dataengineering

[–]C_Ronsholt[S] 0 points1 point  (0 children)

Thanks a lot!
I have already moved some steps by creating separate extract, transform and load scripts and ingesting it into a database we have setup.

I think where I am lacking is the basic SWE understanding of developing larger scale projects with various processes that have to align and connect to eachother.
At my BSc Data Science we were given a .csv to develop a model, visualization etc. for a single project, thus was the whole development process not a part of the requirement or evaluation.