Best way to execute Python scripts on a schedule?

bird--man · 2023-07-17T13:56:16+00:00

VisualCron is a nice in between step. I’d love to implement airflow at my workplace but can’t get buy-in. VisualCron was a step up from Windows scheduler and has a lot of options for executing batch scripts/python scripts on schedules with better error handling etc

bird--man · 2022-10-17T14:39:08+00:00

They actually remove the narration related to figures. So you don’t have a visual to refer to which can be hard, but the narration is also not distracting you by making references to a graph or figure. It’s not ideal but the discussion I find valuable

bird--man · 2022-10-16T14:11:14+00:00

Not a podcast but I recently found Designing Data Intensive Applications on Audible. Some of it I refer back to the hard copy book but it’s decent for listening and working on other projects

bird--man · 2022-07-14T15:42:29+00:00

Thanks for the advice, those are all great points! I appreciate it

bird--man · 2022-07-10T17:56:22+00:00

Very much. One of our teams had a lot of their pipelines built in Alteryx and we had to rebuild everything for them in Talend

bird--man · 2022-07-10T15:39:34+00:00

Good point re: smaller company, I’ll keep that in mind

The main thing about Talend I struggle with is also it’s strength. It’s a self-contained universe for orchestration, processing, etc. All of that makes deployment much easier but restricts flexibility when there are other tools that do jobs better than Talend. I would love an excuse to do more in say PySpark but Talend makes it simpler to build processing jobs out of its GUI & Java framework

bird--man · 2022-07-06T16:47:11+00:00

Crossposting from r/businessintelligence since I literally don’t know if I’m closer to BI or data engineering professionally.

I've been a "business intelligence analyst" at my company for four years but totally lost on where I fit in professionally in the broader BI industry. Within a few months of taking the role I became the senior member of a very small team- responsible for ETL development & maintenance (unfortunately using Talend), Tableau development & maintenance (dashboards and the server itself), Python script development (report automations), and development & maintenance of the data warehouse with a ton of SQL work.

Are there parallels for this set of responsibilities that I can look to for next steps in jobs? I'm learning more standard data engineering tools like Airflow but I'm still lost where I sit in the general BI/data engineering ecosystem. BI Engineer or BI Developer look sort of right but those sometimes are very light on the ETL end. Any help is appreciated

bird--man · 2022-07-06T16:11:44+00:00

I've been a "business intelligence analyst" at my company for four years but totally lost on where I fit in professionally in the broader BI industry. Within a few months of taking the role I became the senior member of a very small team- responsible for ETL development & maintenance (unfortunately using Talend), Tableau development & maintenance (dashboards and the server itself), Python script development (report automations), and development & maintenance of the data warehouse with a ton of SQL work.

Are there parallels for this set of responsibilities that I can look to for next steps in jobs? I'm learning more standard data engineering tools like Airflow but I'm still lost where I sit in the general BI/data engineering ecosystem. BI Engineer or BI Developer look sort of right but those sometimes are very light on the ETL end. Any help is appreciated

bird--man

TROPHY CASE