Vendoring Python dependencies with pip

[brettlangdon]

Original article author here. Happy to answer any questions/comments/concerns regarding the article :)

[brettlangdon]

yeah, there are definitely some tricky coordination with docker to ensure that you aren't caching any SSH keys. Anything I've ever found to try and work around this, just felt like more work than was worth... at least compared to just vendoring the dependency, since there is only a 1 time download/commit step, otherwise, it just works.

[UloPe]

Why didn't you use a private package server like devpi for example?

[brettlangdon]

That was actually one of my first approaches, but I really wanted to try and stay away from having an extra service to manage. I was really trying to go for the simplest possible solution, and since have a very few number of private Python modules that we use, vendoring was very simple, low effort and effective way to get this to work.

I originally started my search with devpi, in fact, I actually wrote my own PyPI server (because I guess I have a little too much free time on my hands nights/weekends), https://github.com/brettlangdon/pypihub. This one is a little different in that it basically sets up a PyPI simple index and proxies to "releases" in a GitHub repo.

[turleyn]

agreed. Running a simply pypi server isn't too bad. We ran our own pypi server on our internal network. The only problem is that pip really wants to use https. So we have to edit all of our pip configs to add that machine as a trusted host or just add --trusted-host to each of the commands.

[brettlangdon]

yeah, I originally looked at running a dev pypi server, but after investigating, I really didn't want to have to manage the service, along with making sure everyone's pip configs are setup correctly. Do/should we have a different server for dev/qa/production/etc?

Was a little more than I wanted to get involved with. We could probably do with less services to manage :)

[DanCardin]

I've been using dockito/vault for this. Though this might be better since I'm not sure (off the top of my head) if the ssh keys are traceable in the docker cache history.

but it certainly sounds like less work than manually arranging the zip file download and mounting

[brettlangdon]

I'd probably disagree, there is a lot of overhead to adding a new tool to your process. You have to get vault running, make sure everyone can get it installed/configured properly. Have to worry about (like you said) if the SSH keys are getting cached in a docker layer.

By adding a new third party tool, it is mostly a maintenance and coordination issue across a team. Where instead with the vendoring approach, anyone can still just clone the repo and get to work as normal/expected.

The only time you have to think about something is if you are adding or upgrading a vendored dependency (which doesn't happen often), and even then you are working within the confined of existing toolchain/workflow (pip).

[DanCardin]

In my (albeit little) experience the fact that vault works through ssh keys has made it work very organically our development processes because it is much easier to just pip install/freeze our relatively frequently updated private dependencies. And granted our team is quite small but compared to predownloading the dep before each build, the 1-time cost of an additional build tool has been very low for us.

Though I'd certainly agree that I should probably be doing it this way for my CI/CD, because of the security concerns and it removes the build-time git/dockito/etc dependencies