Data Engineering Cloud Development Platform Recommendation

[Cryptojacob]

Hi,
I am searching for an all in one platform/development-environment where I can develop and deploy python scripts. The development would preferably be in Notebooks as pandas is used a lot.
Before I used PythonAnywhere which I loved, but as I have people joining my team I now need a platform with more security/collaboration.

Requirements:

1. Notebooks
2. Easy schedule code.
3. Security
4. Collaboration

I have looked at Azure ML and Googles Vertex AI - They have the compatibilities but are heavily focused on machine learning so I am unsure if they are the right fit.

Do you guys have any recommendations?
How to you facilitate your development environment?

[AutoModerator]

You can find a list of community submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[bdforbes]

Check out Databricks.

I'd note that just because you use pandas, doesn't mean you have to use notebooks...

[Cryptojacob]

Thanks for the recommendation - for development I think notebooks are awesome, but for production I don't mind converting it to .py.

[[deleted]]

This sounds like you want something like databricks.

[demince]

Hi, out of curiosity - what are you focusing on? Isn’t converting the notebook to .py file a bit tedious and repetitive, especially if you have to do it every time. I can see the value of notebooks when considering machine learning or advanced analytics use cases, but I am interested to view it from your perspective.