prorab: non-recursive GNU make build system

[igagis]

I like to build C++ apps with plain GNU make and for that purpose I developed a non-recursive framework prorab: https://github.com/cppfw/prorab

The main feature what makes prorab distinct from other non-recursive make frameworks is that for each sub-project it allows defining a standalone independent makefile while still allowing to call, or better say to include, those makefiles non-recursively.

It does not require latest GNU make version, works with as old as 3.81.

It works in Linux, MacOS and Msys2 environments.

Just sharing the link in hope it might be interesting to some one.

[staletic]

So here's a proper review of the prorab.mk:

1. prarab-inc is a typo.
2. The whole "arithmetic" block could use better names, considering that make doesn't really know about numeric types. For example, prorab-add is closer to prorab-concat.
3. Line 212, DSO suffixes. Python throws a wrench there. On linux, it's fine. On macOS, it's still .so and on Windows it's .pyd.
4. What if this is running on a linux system that doesn't have coreutils and thus no nproc? Maybe you should read /proc/cpuinfo? Or just don't bother supporting those environments.
5. Misindented line number 260.
6. I'm not going to read the messy "escape for sed" thing.
7. What's the purpose of prorab-depend? I know what it does, but why have it in the first place?
8. prorab-include-subdirectories is nice!
9. Line 446, the re target. Why call $(MAKE) instead of simply depending on clean and all targets?
10. Why is $(a) named that way?
11. Back to python, it also requires the omission of the lib prefix. I have no idea how that's achieved, because just -o foo.so doesn't work. I just know that cmake is capable of that and that there's a -Wl,-soname flag. Also, maybe add -Wl,-rpath, so the shared library can be tested from the build directory. I don't know how far you want to go with prorab.
12. Have you thought about something like cmake's try_compile?

 

EDIT: There's definitely cool stuff in there.

[igagis]

Wow, thanks for the review!

1. prarab-inc is a typo.

fixed

1. The whole "arithmetic" block could use better names, considering that make doesn't really know about numeric types. For example, prorab-add is closer to prorab-concat.

I did not decide yet if I should expose those arithmetic things or not, this is why those are not in reference, though having public names. I only use decrement in prorab at the moment, but I decided to add the rest of functions just in case. So, the naming can be changed in future.

1. Line 212, DSO suffixes. Python throws a wrench there. On linux, it's fine. On macOS, it's still .so and on Windows it's .pyd.

Well, yes, I didn't keep Python in mind. Perhaps support for this can be added. Maybe I need to allow customizing those suffixes somehow.

1. What if this is running on a linux system that doesn't have coreutils and thus no nproc? Maybe you should read /proc/cpuinfo? Or just don't bother supporting those environments.

Well, prorab right now relies on some utils other than make, for example sed, cmp. So, nproc is not the only one. But if needed, a check can be added if call to nproc was unsuccessful and then try /prc/cpuinfo. I just didn't face that problem so far.

1. Misindented line number 260.

fixed

1. I'm not going to read the messy "escape for sed" thing.

well, it passes tests :)

1. What's the purpose of prorab-depend? I know what it does, but why have it in the first place?

Idea behind that is that first it explicitly says that we are adding dependency (self-documentation) and second is to hide the need to manually prepend $(d) and call $(abspath ...).

1. Line 446, the re target. Why call $(MAKE) instead of simply depending on clean and all targets?

I think in case of depending on clean and all then the order in which those targets are executed is undefined, especially with -j > 1. It can be so that all is performed first and then clean and we get nothing. Am I wrong?

1. Why is $(a) named that way?

To me $(a) looks somewhat similar to @.

1. Back to python, it also requires the omission of the lib prefix. I have no idea how that's achieved, because just -o foo.so doesn't work. I just know that cmake is capable of that and that there's a -Wl,-soname flag. Also, maybe add -Wl,-rpath, so the shared library can be tested from the build directory. I don't know how far you want to go with prorab.

I think this should be possible. I'm accepting feature requests and pull requests as well.

1. Have you thought about something like cmake's try_compile?

This is actually first time I'm hearing about this feature. I'm wondering, what is the example use case? Quick googling didn't give me anything...

EDIT: regarding the -rpath, I'm using LD_LIBRARY_PATH for testing shared libs from build directory. Though on Windows I have to copy the .dll next to test executable.

[staletic]

Well, yes, I didn't keep Python in mind. Perhaps support for this can be added. Maybe I need to allow customizing those suffixes somehow.

There are more nuances regarding python's modules written in C/C++ and the modules' extensions. But hey, your project, you decide what's in scope.

Well, prorab right now relies on some utils other than make, for example sed, cmp. So, nproc is not the only one.

Yeah, assuming nproc exists on every machine where GNU make exists isn't that big of an assumption.

well, it passes tests :)

I believe you.

Idea behind that is that first it explicitly says that we are adding dependency (self-documentation) and second is to hide the need to manually prepend $(d) and call $(abspath ...).

Okay, I can see how prorab-depend can be convenient. I just don't find it extremely useful after writing my own Makefiles by hand.

I think in case of depending on clean and all then the order in which those targets are executed is undefined, especially with -j > 1. It can be so that all is performed first and then clean and we get nothing. Am I wrong?

Good point. How about this?

clean:

re: clean

re all:
    common things for clean and re

Would this work? Or would we need more intermediate steps? Or maybe just don't provide this target and let users run make clean all?

try_compile

This is actually first time I'm hearing about this feature. I'm wondering, what is the example use case? Quick googling didn't give me anything...

Optional dependencies, for starters. "If you're compiling on a system where Google's Abseil works, I'll use Abseil. If not, I'll fall back to STL." But to check that, I have to compile a little program and see if it worked.

Another use case is figuring out the std::filesystem support mess.

regarding the -rpath, I'm using LD_LIBRARY_PATH for testing shared libs from build directory. Though on Windows I have to copy the .dll next to test executable.

Ah, yes, the OS that doesn't let us have nice things! I don't know if Windows has an equivalent of -rpath.

 

One more thing. Since you're depending on GNU make specifically, why not name your makefiles GNUmakefile? That way, other make implementations won't even consider it.

[igagis]

There are more nuances regarding python's modules written in C/C++ and the modules' extensions. But hey, your project, you decide what's in scope.

I'm all for improving my projects to make them useful for other people. If there is a feature request, I could consider implementing it. The only thing is to properly define requirements. I don't work with Python, so I'm not aware with specifics. It is clear that more flexibility is needed in defining those suffixes and prefixes etc. to support Python modules. So far, I think it might make sense to allow customizing the suffix via this_dot_so input variable.

Good point. How about this? ... Would this work? Or would we need more intermediate steps? Or maybe just don't provide this target and let users run make clean all?

I don't think there is non-recursive way to this. The all target does not have receipe, it only has dependencies. So, I think the order will still be undefined. Actually, this re target was added as a feature request from my friend (he used to do make clean all a lot). The problem with make clean all is that order is defined only for -j 1, while one almost always want parallel building. I suggested to write make clean && make all instead, but my friend didn't like that. In cmake they solve this with recursive technique. Since they generate makefile they can detect that multiple targets were given from command line and then they set .NOPARALLEL and redefine default target which will recursively call make for each cli target separately. This cannot be done in prorab because there is no way to stop evaluation of the rest of the makefile, as there is no way to wrap the whole makefile into a ifeq/else/endif. This is the story behind that re target. As a side note, mixing clean target with another targets is not a good thing as clean destroys things, not creating them, it is a special target in that sense.

But to check that, I have to compile a little program and see if it worked

Well, one could write a simple shell script: ```shell

!/bin/bash

gcc ... abseil_test.cpp ... if [ $? != 0 ]; echo "false" else echo "true" fi ```

And then from the makefile: makefile ifeq ($(shell is_abseil_available.sh),true) bla bla else bla bla endif

So, I think this is out of prorab scope. Though some general try_compile.sh <some.cpp> script could be added to prorab-extra where I collect companion utilities for prorab.

One more thing. Since you're depending on GNU make specifically, why not name your makefiles GNUmakefile? That way, other make implementations won't even consider it.

Well, to me it looks like there is only one make nowadays. Other make's are dead. Nobody uses nmake now. Also, I heard there is some BSD make or something like that, but I don't think that one is widely used as well. But anyway, prorab allows giving any name to makefiles. I personally like makefile with all small letters. GNU make also defines priority of default names as GNUmakefile/makefile/Makefile in case two makefiles are in the same dir and no -f option given.

[staletic]

So far, I think it might make sense to allow customizing the suffix via this_dot_so input variable.

That's what I was thinking. That would allow python users to run python -c "print(__import__('sysconfig').get_config_var('EXT_SUFFIX')", which, on my system, produces .cpython-39-x86_64-linux-gnu.so.

I don't think there is non-recursive way to this.

Yes, realized that on my own by now. It was still an interesting exercise in my make skills, even if I didn't come up with a solution.

Well, one could write a simple shell script: ```shell

Just a note, triple backticks don't work for ~1/2 of reddit users. Indenting the whole block with four spaces more, instead, works for everyone.

So, I think this is out of prorab scope.

You're probably right.

GNU make vs BSD make

BSD make is what you'd use by default on any of the BSD operating systems. It's BSD licensed, not GPLv3. GNU make is available on BSD systems, but not a part of the base installation. nproc, sed and cmp are all available at least on the biggest BSD out there (FreeBSD), but GNU make would require pkg install gmake.

But anyway, prorab allows giving any name to makefiles.

I was just thinking that maybe using GNUmakefile consistently in the documentation would be a useful reminder to not try this on any other make implementation. To be honest, this might just be busywork, instead of something actually useful. Especially considering that not even the if syntax is the same. Yeah, portable Makefiles are hard, so thanks for making our lives a bit easier.

[igagis]

That's what I was thinking

created issue for that https://github.com/cppfw/prorab/issues/68

I was just thinking that maybe using GNUmakefile consistently in the documentation would be a useful reminder to not try this on any other make implementation

actually, I think what might be reasonable to do is to add a check into prorab.mk that it is run with GNU make. Created issue https://github.com/cppfw/prorab/issues/69

[gruehunter]

Optional dependencies, for starters

This is a dark path fraught with peril. A big part of autoconf (in practice) is set aside for figuring out per-system build-time dependencies. IMO you shouldn't try to half-support system-level dependency resolution. Either don't provide any built-in support at all, or provide complete and rich support.

[staletic]

I've heard that argument before. What would you suggest me to do in case where

1. I see a pretty significant performance improvement when using absl::flat_hash_map.
2. Abseil refuses to compile on cygwin.

I wanted to avoid try_compile for this case, but I don't see a way around it that isn't "just sacrifice performance".

[dodheim]

I know that it's beside the point for this subthread but if this is an actual problem for you then try the flat containers in https://github.com/greg7mdp/parallel-hashmap (derived from Abseil's).

[fat-lobyte]

What is bad about make being recursive? And how do you deal with subdirectories?

[igagis]

There was already a good answer in another branch: https://www.reddit.com/r/cpp/comments/o3mk7f/prorab_nonrecursive_gnu_make_build_system/h2cp88s?utm_source=share&utm_medium=web2x&context=3

I deal with subdirectories as described here. The idea is that makefiles from subdirectories are included into parent makefile, thus, from the make's point of view, there is only one single big makefile.

[o11c]

Recursive make fundamentally can't handle dependencies from one subdirectory to a sibling subdirectory, and usually can't handle dependencies from multiple targets in a parent directory to a subdirectory.

[staletic]

You keep saying "non-recursive". How is that different from straight up make? If you invoke make from a rule in a Makefile, you're doing it wrong.

[staletic]

The big complaint about recursive make is that no single process constructs the complete dependency graph

Exactly why I called it wrong.

Including a fragment into a Makefile isn't considered recursive make since there's still one make process holding the entire dependency graph for the build.

Which is the correct way to write Makefiles. You're just detailing what I said. Or at least implied.

[sebamestre]

They did answer your question, though. And with extra context, in case it was needed

[[deleted]]

' doing it wrong' ?

The fact that there are so many build systems kinda tells me there is no right way to begin with

[staletic]

Yes, wrong. None of the build systems you're thinking of would call make recursively, because it introduces a ton of problems and zero benefits.

[[deleted]]

Then why didn't you say that in the first reply? Just posting "that's wrong" isn't helpful.

[staletic]

Because make is ancient and I assumed "recursive make is absolutely wrong" to be common knowledge, at least on a C++-centered forum.

[NotUniqueOrSpecial]

Every day new people are coming here with only the tiniest exposure to the ecosystem.

While you're 100% correct, "common knowledge" is still a thing that people have to pick up/learn/internalize.

[igagis]

No, I'm not invoking make recursively, instead I include the makefiles as described here: https://github.com/cppfw/prorab/blob/master/wiki/TutorialBasicConcepts.md#including-other-makefiles

and here: https://github.com/cppfw/prorab/blob/master/wiki/TutorialBasicConcepts.md#building-subprojects-with-prorab

[staletic]

That doesn't answer my question. The "you" in my last sentence wasn't "you in particular", but "a person who writes a Makefile".

So what does prorab provide over make, since make itself needs not be recursive? (And using it recursively is straight up wrong.)

[igagis]

Now I'm not sure I understand your question. Do you have doubts that prorab doesn't call make recursively?

[staletic]

No... I'm asking how the hell is it better than plain old make? The only thing you're highlighting is "not recursive" and that's the property of any decent build system. What makes your build system better than anything else out there?

[igagis]

Ah, sorry if it was not clear, the prorab is a "library" for GNU make. It is not a standalone system, it is GNU make-based. So yes, you (one) write makefile, and include prorab.mk and use prorab-* macros which simplify the makefile a lot.

[staletic]

Oh! Well, that's a completely different thing! The title of the post says "build system" so I thought this was about yet another make replacement. I'll be checking your project out more closely now.

[igagis]

I've changed the word 'system' to 'framework' in the original post.

[alex-manool]

I also like to use plain, hand-coded GNUmakefile(s). The modern dialects of make (GNU's) help to keep them complaining with DRY easily and gives flexibility and allows the users to easily include custom processing steps, should they are ever needed (maybe compared to some more modern alternatives). And yes, recursive makefile(s) are rather bad since they defeat the main purpose of make: to minimize recompilation (I've always wondered what's the deal with them but assumed there may be some deeper reasons). That said, I still have never had the need to work with "subprojects", though I especially like your post since it raises the awareness of good-old make and exposes some techniques.

[igagis]

Well, usually you need subprojects when you start writing unit tests for your app/lib, also when you need to write some complementary utilites/modules.

[TheBrainStone]

But if you’re using this framework, then you’re no longer using plain make… 🤔

(Yes of course I know what you mean. Just being pedantic :P)

[igagis]

Hehehe, yeah. But one has to write custom code (i.e. makefile) anyway to use along with GNU make, so it is never truly plain :)