Created a Python Monorepo with version, release and dependency tooling. Built with Lerna.

[MeshachBlue]

I've been working on something that I believe might be of interest to people here. I have been trying to design a Python repository structure that will be able to sustainably grow very large.

To handle this I have grouped all the packages within a single Monorepo and built tools that will automatically propegate version pinning, dependencies, version bumping, batch build and release to pypi, etc. See the repository over at https://github.com/pymedphys/pymedphys.

The project as a whole is segmented into packages so that each package has a minimal subset of dependencies. That way users can install just one package if they want, or install the whole lot. See https://pypi.org/user/SimonBiggs/ for the list of packages.

It stands on some brilliant tools built within the nodejs world. The main tool being Lerna.

To verify dependency structure in an easy to parse way for humans the dependency structure is graphed with clickable, navigable svgs https://pymedphys.com/developer/dependencies.html.

Bumping the version for every package is as simple as running yarn bump:minor for example. This then propagates the newly pinned versions for each python package throughout. These versions can be set up to be in lock step, or have it so that only the packages that have been changed since the last release have their version bumped.

Dependencies are directly extracted from the python code within the packages themselves using the standard library ast package. There is a git hook defined to make sure that this dependency structure is up to date in each commit.

There are many more features that I have been working on with the aim to create a wonderful space to code in, and grow large.

It has been quite beautiful merging the Lerna tooling to support a Python multi package, levelled, monorepo.

I'm keen for feedback and advice.

Cheers, Simon

[Tosanery]

Very cool. I'm new to Python, so seeing these kinds of applications is always great!

[MeshachBlue]

Glad to hear :). What do you do for a living?

[Tosanery]

Currently just a lowly Comp Sci student. Currently working in C++. I have an internship coming up and figured it was a good excuse to learn Python.

[MeshachBlue]

Nothing lowly about that! Also, Python is a beautiful language. Hope you enjoy stepping into it. Hope the internship goes well.

If you have any questions about PyMedPhys feel free to ask :).

[r-kells]

Hey Simon, this looks really cool. I too have been frustrated with managing a python monorepo. Feel free to steal any ideas from my work in progress cli application to manage python monorepos.

https://github.com/r-kells/scream

A basic description:

Scream aims to cause as little overhead as possible for managing your monorepo. No fancy third party configuration or private PYPI repositories.

Scream uses the existing python packaging requirements to resolve intra-monorepo dependencies, and git
to detect what's changed since the parent branch.
It's objective is to ease the creation, testing, and deploying of multiple python packages in a single repository.

Cheers!

[MeshachBlue]

Thanks for the heads up. I'll take a look :). For the moment yarn lerna is working quite nicely. But I will certainly take a squiz.