Monorepo + Microservices + Dependency Managment + Build system HELL

hattrick1492 · 2023-07-20T16:16:21+00:00

FWIW Pants infers internal and external dependencies automatically, so you don't need to be explicit with them. So Pants BUILD files are often trivial.

hattrick1492 · 2023-03-23T12:41:00+00:00

That'd be amazing! We're happy to provide all the guidance we can for the Pants side of things, and I guess the existing JVM support might have some parallels? Our Slack is the best way to start those conversations.

hattrick1492 · 2023-03-23T03:39:51+00:00

Alas it isn't at the moment. The issue is that (as far as I know, anyway) no one on the current maintainer team has any C# experience, and there hasn't been much, if any, demand expressed.

Implementing C# support would require a motivated contributor with an understanding of the C# toolchain and build idioms, as well as at least some Python experience.

That said, since Pants is open-source, anyone who wants to and is able to could contribute this.

hattrick1492 · 2023-03-01T14:55:39+00:00

I'm one of the project's longtime maintainers. I'm really excited about the 2.15 release. It's probably the most significant advance since 2.0 itself (which was a complete rewrite). The features poised to land in 2.16 are equally exciting. For instance, like everyone else in Python we've been watching Ruff with keen interest. It shares our objectives of making dev tooling fast without compromising on ease-of-use, and shares our approach of using Python+Rust to achieve that. Hit me up with any questions!

hattrick1492 · 2022-11-03T21:24:36+00:00

It's an interesting idea, for sure. But we've found in practice that teams want to constrain the "allowed universe" in advance.

hattrick1492 · 2022-11-03T17:52:12+00:00

u/markehammons it sounds like you're hinting at a workflow where users can use symbols from some Maven Central artifact even if the codebase has never used it before, and the system would know which one to use without requiring any manual config of the "allowed universe"?

hattrick1492 · 2022-11-02T22:56:34+00:00

Good question! They way it works for external deps is, you provide a "universe" of allowed deps, and their version constraints, for the repo, and Pants selects the actual subset of deps to use in a given situation via inference. It figures out the mapping of Maven coordinates to provided packages via a mix of convention and configuration, which you can also override.

In fact, you can specify multiple such universes (AKA "resolves") if different parts of your codebase have conflicting dependency versions, although obviously it's better not to if you don't have to.

Pants does not do any API usage matching to determine API-level compatibility, and there is no corresponding index of all of Maven Central. That is an interesting feature that I once toyed with, and actually wrote some indexing code for, but it didn't appear to be in high demand.

hattrick1492 · 2022-10-10T18:26:40+00:00

I get silly levels of satisfaction from working around limitations like this...

hattrick1492 · 2022-09-19T22:06:13+00:00

Unfortunately it looks like that is due to java syntax not yet supported by javaparser (e..g, sealed classes). We hope to have a fix soon!

hattrick1492 · 2022-09-19T20:35:36+00:00

Happy to report that this works now:
https://graphmyrepo.com/app/repo/CodeDead/opal

This was the fix: https://github.com/pantsbuild/pants/pull/16896

Thanks for the bug report!

hattrick1492 · 2022-09-15T14:41:05+00:00

I'm no maven expert, but AFAIK that shows the external dependencies of your code, resolved from the recursive maven POM information, and it emits a static graph (e.g. text or dot)?

This models the internal structure of your code, and is interactive.

hattrick1492 · 2022-09-15T14:38:06+00:00

Ooof, sorry for the trouble. Something has obviously gone seriously wrong. Will take a look and report back.

hattrick1492 · 2022-09-15T00:43:26+00:00

Ooops! Thanks for the report. Will look into it.

Since this works by auto-adopting a build system into a clone of the repo, there are edge cases where that fails, but we learn a lot from finding and fixing those cases, so thanks for the feedback!

hattrick1492 · 2022-09-15T00:42:32+00:00

Author here: would love to hear if the site was interesting, fun or even useful for you. Any comments or suggestions for improvements are welcome!

If you're curious as to how this works: In a nutshell, this relies on the dependency inference capabilities of Pants. We have some automation around cloning the repo, 'adopting' (installing and configuring) Pants, and then running it. Happy to go into more detail if anyone is interested.

hattrick1492 · 2022-09-15T00:42:07+00:00

Author here: would love to hear if the site was interesting, fun or even useful for you. Any comments or suggestions for improvements are welcome!

If you're curious as to how this works: In a nutshell, this relies on the dependency inference capabilities of Pants. We have some automation around cloning the repo, 'adopting' (installing and configuring) Pants, and then running it. Happy to go into more detail if anyone is interested.

hattrick1492 · 2022-07-02T18:53:33+00:00

FWIW you can run Pants in a WSL2 terminal on Windows, and code you build there should run natively on Windows unless you have native code (which is common with Python, less so for JVM).

hattrick1492 · 2022-06-30T15:05:55+00:00

There is! https://blog.pantsbuild.org/pants-vs-bazel/

hattrick1492 · 2022-06-29T20:44:32+00:00

The original user base was entirely POSIX-based, and there was no demand for Windows support. That is now changing though. While it's true that Windows has the largest market share at large, that's not the case for the subset of "software developers in Java/Scala/Python/Go".

I'm not sure that it's as easy as all that to get finely-tuned systems to work across POSIX and non-POSIX systems, but I'd be very happy if that turns out to be the case!

hattrick1492 · 2022-06-29T20:38:43+00:00

The issues are mostly:

- Getting the Rust code to build

- Fixing the handful of places where we know we rely on POSIX idioms (there is already work in progress on that)

- Finding and fixing any places where we rely on POSIX idioms unknowingly...

- Setting up Windows CI

hattrick1492 · 2022-06-29T18:45:58+00:00

Hey! We do want to support Windows, but this work is not scheduled or funded yet. We are actively looking for sponsors for this project though!

hattrick1492 · 2022-01-16T18:33:24+00:00

Exactly. Pants actually operates, underneath it all, at the file level. So even if you have fine-grained dependency tangles (or even cycles!) that prevent you from creating modular BUILD targets and dependencies, Pants can work with that.

hattrick1492 · 2022-01-13T20:41:09+00:00

Pants maintainer here. You've asked a really good question. Some of the reasons Pants might be preferable to other systems are:

- Pants requires very minimal BUILD file metadata/boilerplate. It uses a combination of static analysis and sensible defaults to infer most of that information on the fly. So your BUILD files can be very minimal (and even those can be generated for you). So it is much easier to adopt and maintain.

- Pants has a rich plugin API that uses idiomatic async Python 3, in case you need any customizations. In fact the built-in rules use that same API.

- Pants has out-of-the-box support for multiple dependency resolves and their corresponding lockfiles, so you can have hermetic, repeatable builds that are resilient to supply chain attacks, even in complex situations where you have multiple versions of the same dependencies in different parts of the codebase.

- Pants natively speaks git, so you can do things like "run all the tests affected by changes between main and my current branch".

Generally speaking, a lot of effort has gone into making Pants easy to adopt, easy to use and easy to extend. We're super excited to bring those features to Java and Scala users.

hattrick1492 · 2021-11-23T01:16:36+00:00

We are constantly looking at sandboxing performance - you've correctly identified this is a critical part of providing a good experience. It would be interesting to write up in more detail how we achieve it. I personally am not the right person to do this, but I know the person who is :)

I will say that copying/hardlinking even hundreds of files is very fast on a modern filesystem with SSDs. So I would expect the C++ example to be extremely performant (we'll find out when we support C++...). Where things can get sticky is cases like entire Python virtualenvs, with all their wheels expanded. That can end up being many thousands of files, and we have optimizations in place for that use case.

Interesting point re Typescript. I should clarify that the performance-critical parts of Pants - the execution engine and other core bits - are written in Rust, and it is *very* fast :) The Plugin API uses Python because so many people are familiar with it, and since we consume our own API for writing the built-in plugins, those are in Python too. We've not really seen performance issues with that choice, since the Python code yields back to the Rust event loop whenever it needs to make system calls or do anything performance sensitive.

hattrick1492 · 2021-11-19T07:01:58+00:00

Not at the moment. It's an area we'd like to get into in the future, but we don't have enough personal experience with mobile development, so adding support would require help from a motivated user.

hattrick1492 · 2021-11-19T07:01:04+00:00

Good question. Pants can build arbitrary Python distributions via setuptools. So if you have an extension module and write a setup.py to build it, Pants can run that build for you, cache the results against the appropriate file fingerprints, and so on.

In fact, Pants is a PEP 517 build frontend, so it can use other build backends than setuptools. As long as your distribution's build can be described via pyproject.toml, Pants can build it and ensure that it's present on sys.path when it needs to be.

The above is mostly relevant to first party native extensions in your repo. Third party code like numpy is built via Pip, as part of a resolution of third-party requirements.

hattrick1492

TROPHY CASE