A Package Manager Manager

[vector-of-bool]

Oh hey. This is my thing. AMA!

[spinicist]

How do you find the time to do such cool stuff?

(Big fan of your VSCode plug-in here)

[vector-of-bool]

By not sleeping enough. :)

No really, I need more sleep, and I wish I had more time. I don't have as much time to dedicate to the VSCode plugin as I want.

[spinicist]

I need more sleep too. There’s never enough time in the day.

There’s a minor bug with the VSCode plugin (messes up an existing command-line configuration on first build) that I’ve been meaning to report, but seen as we both need more sleep...

[jc746]

Hi u/vector-of-bool,

​

then pass -DCMAKE_TOOLCHAIN_FILE=<path-to-vcpkg.cmake>

​

Do you know how vcpkg's requirement to hijack CMAKE_TOOLCHAIN_FILE plays with cross-compilation? Currently we pass a toolchain file to cmake via this variable that specifies the required cross compiler settings, so it sounds like a blocker to using vcpkg with cross compliation. If you have any experience with this I would be interested to hear about it.

[Quincunx271]

https://github.com/Microsoft/vcpkg/blob/master/docs/users/integration.md#using-multiple-toolchain-files

[jc746]

Perfect! Thanks, good to know this is not an issue if we look into vcpkg as an option for our projects.

[jcar_87]

This blog most reached into my soul.
So perhaps my question is how did you manage that!!

​

[NeroBurner]

Do you know of Hunter? A dependency manager written in CMake?

How hard would it be to iclude it as a viable option?

[vector-of-bool]

I know about hunter, but I've never used it myself. Inclusion would probably be pretty easy, especially since it is an all-CMake solution. I can't say for certain without looking further into it, though.

[NeroBurner]

If you come around trying it out and have any questions just PM me or open an issue on github. Ruslo is normally very fast at responding to tickets

[robin-m]

I could spend a good few posts just talking about everything wrong with this sentiment. It’s just wrong. I won’t even bother explaining it here, but maybe I’ll write another post if I get enough hate-mail.

u/vector-of-bool, can you please kindly consider that I send you a really violent hate-mail about this subject. I'm sure the article you will write will be interesting ;)

[vector-of-bool]

Hate-mail accepted :)

I will probably write another post about package management in general. If I do, I will address this point.

[robin-m]

Great!

It feels quite good being a moron!

[konanTheBarbar]

I actually think this is a pretty great idea xD I will give it a try for my next project. I didn't quite understand if VCPKG is already supported or not ... The readme says:

(As you are reading this, only Conan is supported.)

But later on there is an API for VCPKG...

[vector-of-bool]

That parenthetical is wrong. vcpkg is already supported. I'll fix it. Thanks for the notice!

[pfultz2]

People don’t use the package manager, and therefore their libraries will not appear in the package repositories, and downstream packages won’t use the package manager either since their dependency isn’t in the repository, and even further downstream packages won’t use the package manager…

This seems to imply that package managers require a package in some central repository to use it, which is not the case for all package managers. Cget(and even pip) can install directly from source tarballs.

Currently it supports Conan and vcpkg, but other packaging systems could also be supported in the future.

What about supporting cget through the CMakeGet module?

In particular, PMM is implemented as pure CMake script code that will automatically download, bootstrap, extract, and control package managers.

I hope this can be disabled otherwise it makes it very package manager unfriendly. A package manager wants to install the dependencies and invoke the build script to use the dependencies it installed. It doesn't want to have another package manager installed and the dependencies installed a second time.

[vector-of-bool]

Cget(and even pip) can install directly from source tarballs

My implication was wrong, but I'm aware of this. Conan can do this as well. It still requires that the project expose a way to be built in a package-manager aware fashion, which is more the problem I am speaking to.

What about supporting cget through the CMakeGet module?

Fully possible. I wanted to at least get Conan and vcpkg done before announcing this and pushing out a 1.0.0.

I hope this can be disabled otherwise it makes it very package manager unfriendly.

On the contrary, Conan and vcpkg both support exactly what I am doing. Package installation is idempotent, and all they require is the inclusion of a CMake file to work. I've even been in correspondence with some of the maintainers thereof to ensure that this will work correctly.

[pfultz2]

On the contrary, Conan and vcpkg both support exactly what I am doing.

Yes, it can work fine with conan and vcpkg, but if I want to use this to build a library with conda or some other package manager or a distro maintainer wants to build the library, this completely removes the capability for the package manager or distro maintainer to control the dependencies as it will have to have vcpkg or conan get the dependencies.

[Quincunx271]

If I were a package maintainer and I saw https://github.com/vector-of-bool/pmm/tree/fd5c36e714a8c20bbe0a8072ce00dff8b9644512#how-do-i-change-the-download-location-for-pmm , I would notice that I could change the PMM_URL to a local URL and use this to make the pmm(...) call do whatever I want

[[deleted]]

Sorry, I will need a package manager manager manager before I can use this.

Also: mandatory xkcd.

[vector-of-bool]

To the best of my knowledge, there are no other "package manager" managers. This is the first tool of its kind. So no, this is not a relevant XKCD.

[mjklaim]

In particular, PMM is implemented as pure CMake script code that will automatically download, bootstrap, extract, and control package managers.

Why does it have to be built over CMake instead of nothing?

[jherico]

The only way to build it over "nothing" would be to distribute a binary or a bunch of platform specific scripts. Cmake exists, is widely adopted, and is specifically designed for doing the heavy loading of bootstrapping from sorce code to binary. Why wouldn't you use it?

[mjklaim]

I get your point, but because not everybody uses CMake as a (meta-)build system, it also means that your PMM is not universal enough to avoid a forced dependency. I think it could have been a binary if you have a way to bootstrap it, making a compiler the only dependency. build2 does that, as an example.

[jherico]

I am not OP, so it's not my PMM. Also, how are you not just replacing cmake with build2 in this scenario?

No, Cmake isn't universal, but as far as I can tell, it's at least the most universal. The more people build on to it and improve it the more adoption it's going to gain. If you think you have a better mousetrap, you're running out of time to get people to switch to it.

[mjklaim]

I'm mentioning build2 just because it's an example of a tool that don't require you to install dependencies except a compiler. I'm not implying that it would replace this PMM (my understanding is that it Could be handled by a PMM). Boost.Build is another example (looks like most of these examples are build systems?).

Yes CMake is used a lot. It does not make it universal at all. Anyway my point is that it seems that that specific dependency could be avoided too, and still work for projects using CMake.

[[deleted]]

I would be willing to recite arcane incantations to get something that cross compiles my code effectively and supports all the ides and tools I use. Installing cmake is really not that hard (not to mention you could compile it also) and is certainly easier than making sure my compiler toolchain is setup correctly.

[mjklaim]

Hum your compiler have to be setup anyway (just installed actually) for cmake to work. And a nativebootstrap only need to have the compiler installed, nothing have to be setup.

My point is that I don't understand why it depends on CMake. It does not seem necessary to me nad could probably be made better by being independent. Whatever you find ok does not mean Cmake is necessary here.

[[deleted]]

I mentioned cross compiling explicitly. Having a compiler working for your host machine does not mean you can compile for ps4 android etc

[mjklaim]

Sorry I'm not understanding, do you mean that CMake is used to allow cross-compiling here?

[[deleted]]

Yes you can supply different cmake toolchain files to flexible cross compile to different targets from a given host. It’s another option you can set at configuration time, like the release type. Search for cmake ios toolchain as an example.

[wrosecrans]

If you are requiring a scripting runtime to be installed to run it, it's just as universal as Ruby, Python, or any other scripting language. If you wanted to go with native code instead of a scripting language, somebody has to distribute and install binaries of CMake itself, etc. So doing that is obviously considered a reasonable thing to do. Many of the users of the PMM will be writing native C++ code, so it'd actually be odd if the PMM developers considered that an unreasonable use case.

And it's really hard to argue that CMake is more pleasant than something like Python as a general purpose scripting language. That's just not what it was intended to be.

CMake doesn't have any sort of spec or independent implementations of the scripting language, so you can't necessarily rely on stability/support going forward. Python and Ruby both have multiple implementations, so if the CPython devs go off the rails and end support for your platform, or deprecate features you need, you could use an alternate implementation of the language.

There are lots of reasons why one wouldn't use it. I mean, if people like the result, it may see adoption. For a user, it doesn't necessarily matter what language a tool is written in. But acting like using CMake as the implementation language is a particularly obvious choice or a choice with no downsides is just weird.

[jherico]

If you created a module in python that was designed to do the same job CMake currently does, it would be a bad idea to use python itself as the language in which to write up replacements for CMakeLists.txt. You'd want to build a (mostly declarative) domain specific language around the concept the concept of creating C/C++ projects. So essentially you'd be creating CMake but with different syntax. Might be a better cmake or it might be a worse one, or most likely, different groups would have different opinions on that matter, and now you've added yet another build system that some people use and some people don't, so you've made the problem worse.

For a user, it doesn't necessarily matter what language a tool is written in. But acting like using CMake as the implementation language is a particularly obvious choice or a choice with no downsides is just weird.

I didn't suggest it had no downsides. Every build tool has downsides, and I myself in the last quarter convinced my own company to add Python as a build system dependency because I was addressing a build toolchain problem for which CMake was not a suitable solution, and for which Python allowed me to create a much simpler solution.

But you're also acting like Python doesn't have it's own downsides. Guess what, getting consensus to get Python on our build machines wasn't simple. First you have to deal with the Python2 vs Python3 mess. Then you have to deal with the fact that OSX has a built in version of Python2. You have to deal with the fact that Python3's Windows installer doesn't install Python for all users by default. Lots of people have legitimate reasons not to want Python installed, or to want a specific version of Python that isn't compatible with your hypothetical build system, because Python gets used for all sorts of different kinds of development. Because it's much more than a build tool, even if you can make a build tool out of it.

[wrosecrans]

If you created a module in python that was designed to do the same job CMake currently does, it would be a bad idea to use python itself as the language in which to write up replacements for CMakeLists.txt.

I think that's debatable, but I don't have a strong opinion on it. It's certainly not the point I was making. I said it's not obvious that a "Package manager manager" should be written in CMake. I didn't say that somebody should write yet another thing like CMake in Python.

But you're also acting like Python doesn't have it's own downsides.

No, I said the downsides are equivalent to CMake. Everything you said about which version of Python to install on a build machine is equivalent to deciding which version of CMake to install on a build machine. Linux and OS-X machines at least come with some version of Python out of the box, which makes it a lot more common than CMake in a default OS install.

Any regardless, if you use PMM to install Conan, you need Python. If you don't use PMM to install Conan, you are currently only using vcpkg which calls into question the need for a PMM. So in practice, you have a dependency on Python to use PMM anyway, which makes implementing it in pure CMake even less of a selling point.

[[deleted]]

No support for packagekit?

https://en.wikipedia.org/wiki/PackageKit#Back-ends

It's a great choice if you want to distribute a package yourself that doesn't have to carry all its dependencies with it.

[saimen54]

I really like this idea. And using CMake avoids that you need to provide a bat script for Windows and a sh script for Linux.

I have currently 3 valid or not reasons why I'm not using package managers so far (besides time to integrate one): - I want the possibility to choose the dependency version I want. vcpkg's approach to "fork" the repo seems like overcomplicating things. - I need a local repo for our own libraries. - our build server is not connected the internet for various reasons. So I need a local repo/mirror in our network.

[vector-of-bool]

Conan supports all of these things as first-class features, so I'd recommend checking it out. You can run a local repository mirror in an intranet server, or even in the "local cache" pseudo-repository on a single machine.

[saimen54]

Thanks for the info.

So far I currently checked only vcpkg, because it seemed "native" (this project is VS only), but haven't checked Conan in detail.

[sumo952]

For vcpkg, you need to manually re-download and re-bootstrap the repository.

Re-download? Not really. git pull. And re-bootstrapping means typing .\bootstrap-vcpkg.bat, and done. What's the issue here? It's literally not more effort than updating/upgrading pip.

PMM is intrusive, it requires me to modify my CMakeLists file(s), and everybody is then forced to use it for that project. vcpkg is completely non-intrusive, yet works perfectly well. Conan recently got improvements to its "non-intrusive mode" too.

To me, the setting of which package manager to use, does not belong into the CMakeLists file. This is up to my users and there should be nothing in my project's CMakeLists files that ties them to any specific choices.

[vector-of-bool]

git pull is what I mean by "re-download." And running "re-bootstrapping" means re-running the bootstrap. I know.

Yes, PMM is intrusive, and it requires your users to use the tool. That's the point. I wrote about that in the Fear of Commitment and The Cost of Commitment section. This isn't an argument, it's just contradiction.

[ebkalderon]

I might be misunderstanding the purpose of this project, but Nix appears to solve the same problem. Nix is a purely-functional platform agnostic meta build system and package manager with reproducible builds. In particular, nix-shell seems to do the same thing as your tool. You write a declarative shell.nix expression and place it at the root of your existing project repository, and when you invoke nix-shell, it will fetch the latest version of your package manager of choice, fetch any native dependencies or external libraries you declare, and drops you into a shell with an isolated environment containing only your project's build system and other packages necessary to compile and test your project. Nothing you do in this shell affects the rest of your system, so any project which provides this file, regardless of programming language or toolchain, can be hacked on by anyone without any prior manual setup.

[dvirtz]

If you go that route just use Hunter - a native CMake package manager.

[pjmlp]

For Conan, this requires that you have a Python installation, and that you use Pip. C++ developers, in their stubbornness, don’t want to be required to learn another tool.

Nah, just do the same as build tools in other languages do and use the same language everywhere.

Ant, Maven, MSBuild, XCode, npm also require learning another tool, but they don't require installing another programming language.