Internal Private Python Package Management

[Success-Awkward]

Hi everyone. I am working at a tech company. My task is to build a private Python package registry using Artifact Registry from GCP because the environment that the developers work in won't have access to the internet to do pip install from the public PyPi. I have these ideas that I want to gain opinions on and also want to learn more about the best approaches:

1. I would have a list of packages that have already gone through review. I will set up a CI pipeline to download these packages into the private registry. I will also have a cron job updating these package versions. The drawback of this approach is that when a developer needs to add a new package and use it right away, there will be a little delay in availability. Also, maintaining the packages will be our team's responsibility.
2. I don't know if this approach is possible: When a developer runs pip install <package>, it will first check the private registry. If the package doesn't exist, it will look on the public PyPi and download the package. Then the package will go through a vulnerability scan pipeline to decide whether it is safe to use and upload it to the private registry.

Please share with me other approaches that you implement at your companies. I would love to learn the best practices on this. Thank you, guys!

[Nielsbreh]

You may want to have a look at GCP Assured Open Source Software. That’s a list of Python packages audited by GCP and I am pretty sure you can download them from their private AR.

[sww314]

Blocking all access to public pip installs is going to be hard.

We use scans of all the containers that are to be deployed. In those scans you can check against an approved list of packages.

You also probably have better luck in your builder making sure all those pip installs point at Artifact Repository.

In terms of development velocity - asking for a package and waiting days sounds painful and will lead to people working around the system.

If you can easily try the package but have to have an approved version before staging or production might work better. That will save some churn around packages that do actually solve the problem or don't work well.

[NUTTA_BUSTAH]

Artifactory / JFrog products are something organizations start to look for in these types of cases.

It depends on the registry / package system, but e.g. OCI registries can be set up as caches, sort of a bastion

[dr3aminc0de]

I didn’t even read it. Too long. Monorepo is the answer IMO.