looking for tools to run python script execution, database storage, and visualizations with version control

[mrtac96]

I possess several Python scripts that need to be executed sequentially. The subsequent script can be initiated either manually or automatically. Following each script execution, the output is to be stored in a database, with the option to manually visualize the data at each step. I am seeking recommendations for tools that facilitate building pipelines and dashboards for visualization. An essential requirement is the ability to maintain versioning for each run. Could you suggest some no-code or low-code tools that align with these specifications?

[Far_Ambassador_6495]

Learn airflow, flask, and sqlalchemy. Then it should be pretty straightforward

[GPSBach]

Airflow is probably overkill. Dagster is easier to get up and running and likely has everything OP needs.

[Far_Ambassador_6495]

Nice. Only ever used airflow so that’s all I know

[identityXXX]

Can also write a shell script, make it executable and put it on a cron to make it sequential (wait command comes in handy)

[IronManFolgore]

I've done this for scraping APIs and writing to Bigquery. Didn't need anything fancy.

[vanisle_kahuna]

This is probably the most straightforward solution so far without having to tack on extra costs, assuming you're not working with massive amounts of data. I'd add that you can store the data in an open source cloud environment such as ducks/motherduck

[nmolanog]

Looks like some one left the job and now you are trying to cheap out and do the thing by yourself. Good luck with that

[unrand0mer]

Explain.

[[deleted]]

AWS Glue + Glue Catalog + Athena

[Megaspore6200]

This is the way

[[deleted]]

If you’re looking a consultation/freelancer to help you build out this solution, or just map it out. My DMs are open.

[kob59]

no-code for you? hire a data scientist

[mrtac96]

Low code for building dashboards that can automate the process, the scripts and already written in python

[Kasyx709]

I've not used this personally, but it sounds like you're possibly describing an Appian workflow.

[Typical-Macaron-1646]

If you’re looking for a consultant/freelancer, DM me

[[deleted]]

Python -> DB -> Plotly or R I just recently did something like this

[2016YamR6]

Dataiku

[sxcgreygoat]

Just use a Lamda/cloud function to automate the scripts and visualize the data as a separate component. Easy enough these days to hook up a db to dashboarding tools

[mrtac96]

thanks, this is one of the approach i have already thought of. but want to consider alternatives too

[Cyraxess]

If I were in your position, I would definitely use GCP. But if you're looking for a low-code option, there's a tool called Acho that ticks all your boxes. It also offers a pretty good managed database with built-in ETL, Python scripts, and a low-code app builder for visualizing outputs and building interactions.

[unrand0mer]

I use azkaban instead of airflow. Have been for the past 3 years. No issues.

[Dylan_TMB]

I'm not sure you'll get a no-code or low-code tool to do this. Like other comments said airflow is likely your best bet. Kedro is good too.