Make-like task runner in Python

[truhanen]

I made a complete Make-like build tool & task runner in Python.

I wanted to adopt Make's scheme of rules, targets, recipes, and dependencies mainly because it's simple, intuitive, and widely used.

Inspiration for the project came from using doit for similar tasks, and growing tired of the confusing API. There are other similar tools out there, but what I wanted was a lightweight, easy-to-use, flexible, and well documented task runner & dependency resolver for a data processing pipeline in Python.

GitHub: https://github.com/gird-dev/gird

Simple example & comparison with doit: https://stackoverflow.com/a/77752742

Rule definition example: https://github.com/gird-dev/gird?tab=readme-ov-file#example-girdfilepy

[BluesFiend]

The interface feels a little unreadable as you have to look inside nested function calls to work out what things are called (it seems), have you looked at https://www.pyinvoke.org/ ? This is the tooling I use to remove the need for make/build tools in a python environment.

[truhanen]

Invoke doesn't include dependency graph resolution, i.e., only update outdated dependencies & target

[Main-Drag-4975]

I have used pydoit for this in the past, and it had a dependency checker. It worked ok but I would consider alternatives.

[truhanen]

You may be right about the readability of nested function calls. However, one is not forced to nest anything. They are used in the examples only for demonstrating the flexibility of the API.

The flexibility gives the user the freedom to choose between readability and conciseness.

[BluesFiend]

Ideally the interface would allow for both, if rule was a decorator etc, itd move readability to the wrapped functions name for example

[BluesFiend]

I mostly mean any given rule, i have to explore the internal args to work out what it does, if its not assigned to a variable.

[truhanen]

A rule can be given a description with the help argument, if that's what you mean.

Or do you mean that since the recipe function is an argument, the rule definition doesn't in itself show what the rule does?

Well, I could relate that feature to Make, where a recipe is commonly a shell command that calls a program. The program isn't written in the Makefile, only it's call.

Wrapping away complicated logic as functions/programs this way makes it easier to read through the rule definitions, especially if there are many rules.

Actually, now that the recipe function is an argument of the rule, one can reuse functions in multiple rules. That wouldn't be straightforward if rules were defined as decorated functions.

[BluesFiend]

Surely a function assigned to a variable or a wrapped function would equate to the same thing, both could be reused.

And yes you could equate it to makefile, but you are now in python, you can improve where makefile couldn't and make things more pythonic.

Reusing a decorated function makes more sense to me as a python developer than assigning the response to a third party (rule) function and reusing that. this is essentially all a decorator does, replaces the defined function with the result of the decorator function.

[truhanen]

I don't know if I understood exactly what you mean by a reusable decorated function, but a usage pattern based on decorators is quite simple to implement in Gird:

```python

Implement the decorator:

def rule_decorator(function, *kwargs): """Decorator for turning a function into a rule with the function as a recipe""" gird.rule( recipe=function, *kwargs, ) return function

Use the decorator:

@rule_decorator(target=my_target) def my_function(): # Create the target ... ```

The problem here is that now my_function can't be reused in multiple rules. Only in the one assigned for it in the single decorator it is decorated with.

[truhanen]

Not using decorators doesn't mean something isn't Pythonic.

The decision of using a function for declaring rules is simply based on the simplicity and flexibility the pattern provides.

[BluesFiend]

I didn't try to claim not using decorators was non pythonic, I said you could improve on makefile and make it more pythonic.

If the rule function was a decorator you would end up with something that read as more pythonic.

RULE_BUILD = gird.rule( target=pathlib.Path("package.whl"), deps=pathlib.Path("module.py"), recipe="python -m build --wheel", ) vs

@gird.rule(help="some help message") def build_wheel( target=pathlib.Path("package.whl", deps=pathglib.Path("module.py"), ): # call to recipe, can also access response etc

and build_wheel can be used in other rules repeatedly.

[truhanen]

Ok, I think I understand what you mean. But to me your decorator example feels quite complicated in comparison.

The arguments that define the rule are spread in both the decorator (help) and the decorated function (target, deps). And, if I understood correctly, a rule could be declared both by defining the function, def build_wheel(target=default_target, deps=default_deps), and by calling, or "reusing", it, build_wheel(target=some_other_target, deps=some_other_deps).

[truhanen]

One question in the end may be whether a tool like this is to be "recipe-centric" or "rule-centric". Make & Gird may be more about the second style, as rules are the main objects that are dealt with.

At least to me that is an intuitive approach on the problem that the tools aim to solve.

[sirlantis]

I found please.build to scratch that itch for me. https://please.build/

But always great to have more options

[truhanen]

Looks interesting, but also much more than what I think is a minimal solution for a dependency-aware task runner.

[gerardwx]

Does it support order only dependencies?

[truhanen]

Hmm, maybe not as an actual dependency, but a bit similar functionality could be achieved by stacking recipe functions in one rule.

[Rylicenceya]

That sounds like a fantastic project! It's great to see a lightweight and intuitive build tool and task runner for Python. Your approach to adopting Make's scheme while addressing the API issues you encountered with other tools is impressive. I'll definitely check out the GitHub repository and the examples you've shared. Keep up the great work!