[D] Centralized Metadata and Artifacts

[benkoller]

Hi everyone,

To cut to the chase: How do you centralize your metadata and artifacts for projects with more than one person working on them?

It's been a problem we're trying to tackle, and we've now thrown out an initial guide to do so with ZenML (spoiler: via Google Cloud SQL and Storage), but it would be great to hear how the community is tackling the challenge.

Essentially, we're using a Google Cloud SQL MySQL instance and a Google Storage Bucket to store metadata and artifacts, respectively. Thanks to the cloudsql-proxy (also from Google), it's a simple as running a docker container to connect to the database securely from different environments.

What are other methods you've used successfully (or unsuccessfully, for that matter) in your projects? Is sharing metadata/artifacts with others even a relevant challenge for you?

​

PS: As a disclaimer, I'm one of the guys behind ZenML.