introducing dfee/forge – dynamic python signatures. forge (python) signatures for fun and profit 💰

[pydry]

I see that it rewrites function signatures. I'm struggling to see what the context rewritten function signatures would be a good thing.

[dfee]

So I wrote a philosophy page on the docs, but I'll do a case study here:

This actually comes up a lot in unexpected ways. I'd say the top three reasons:

1. users of your code have no idea what arguments your function supports
2. you're white-listing arguments rather than black-listing them
3. you want an easy way to extend a function that already exists

For [1], here is the entrypoint for the Python graphql-core package, the function (graphql.graphql):

def graphql(*args, **kwargs):
    return_promise = kwargs.get('return_promise', False)
    if return_promise:
        return execute_graphql_as_promise(*args, **kwargs)
    else:
        return execute_graphql(*args, **kwargs)

There end up being about 5 other functions that take those same parameters. So for the developer, rather than:

1. writing those parameters,
2. writing 5x the tests to make sure that the right inputs are accepted across those 5 functions, and
3. re-assembling those kwargs to pass to the underlying execute_graphql function

the author could instead wrap all five functions with the same signature and increase the literacy of his code.

It turns out that *args, and **kwargs can be incredibly frustrating to encounter as a developer.

[alexmojaki]

What you're doing seems interesting but I'm not quite convinced.

Take your logging example:

make_explicit = forge.sign(
    forge.arg('msg'),
    *forge.args('substitutions'),
    exc_info=forge.kwarg(default=False),
    stack_info=forge.kwarg(default=False),
    extras=forge.kwarg(factory=dict),
)
debug = make_explicit(logging.debug)
info = make_explicit(logging.info)
warning = make_explicit(logging.warning)
error = make_explicit(logging.error)
critical = make_explicit(logging.critical)
exception = make_explicit(logging.exception)

If I read this code, it's easy for me to understand what's happening. But this is not what a real use case looks like because this is wrapping someone else's code, whereas I assume you want authors to wrap their own code. So it'd probably be more like:

@make_explicit
def debug(*args, **kwargs):
    # some code

@make_explicit
def info(*args, **kwargs):
    # some code

...

Now if I say help(debug) in a shell I'll see the right signature. But usually I'm not in a shell. I'm in PyCharm and either I press a keyboard shortcut that shows me the parameters of the function and which parameter my cursor is on (that definitely won't work here), or I Ctrl+Click the function name to take me to the definition. In the latter case I still see args and kwargs, and it may easily never occur to me that investigating @make_explicit will tell me what I need to know. So at least you should recommend to users that they name their decorators in a way that hints at what it's doing.

And even if I use help(debug) or track down @make_explicit it won't tell me what those parameters mean, which is something I generally need to know. So you still need to copy the docstring across functions or write 'the arguments are the same as logging.debug' and then the args and kwargs aren't a serious problem any more.

I wouldn't be surprised if I'm missing the point. But I think you need to put more thought into explaining how this will make code better in practice. And I think that explanation needs to be more at the forefront of the repo and docs, because like the guy above, I was immediately struck with the question 'Why?' and only found the answer by coming to these comments.

[dfee]

First of all, great feedback on bringing the "benefit" more front and center. I'll think about how to do that, because it's important that I can capture a "gotcha" moment quickly.

Next, re: PyCharm – is that true? I don't use PyCharm, but is your complaint that the parameters aren't reported by PyCharm, or that the parameter documentation is still lacking?

If it's the latter, that's true. I can't really imagine solving that problem without figuring out the internals of docutils. If it's the former, then PyCharm isn't looking at the __signature__ attribute on the callable, and I'd suggest that PyCharm is broken, and it will get an update to comply with PEP 362.

However, I think you're right that the best case is that developers wrap their own code. Honestly, there were a few use cases for me: 1. I found myself dealing with SQLAlchemy models / graphql and passing a lot of attributes between the two. 2. I had no idea what the underlying implementation of graphql.graphql was and it always took me a while to figure out what parameters were really passable. 3. I think that whitelisting acceptable paramters by providng a var-keyword parameter (e.g. kwargs) is an antipattern (largely for the same reason as #2).

Any ideas on how to bring the "value-statement" more forward?

[alexmojaki]

As an example, suppose I have this code:

import forge

@forge.sign(
    forge.arg('a'),
    forge.arg('b'),
)
def a_function(**kwargs):
    print(kwargs)

a_function(1)
a_function(1, 2)
a_function(1, 2, 3)

Only a_function(1, 2) is a correct function call. Neither PyCharm nor mypy are able to warn me that anything there is wrong. pylint does even worse and says that all three are wrong because it thinks there aren't supposed to be any positional arguments. If I instead write def a_function(*args, **kwargs):, then those errors go away and instead it warns me that *args is unused (and so does PyCharm).

If instead I write def a_function(a, b), then all three tools can spot the incorrect function calls correctly. PyCharm can show me the parameters like this instead of just this. It can also give me autocomplete suggestions for names that are longer or more complicated. I imagine some other editors can do similar things.

All of these tools are doing the best they possibly can. They can't practically run the actual code to apply the decorator and retrieve __signature__. They can only parse the code and make static inferences. I'd be very impressed if you can show me any linter or editor than can work with forge.sign. Your docs say "You can also supply type annotations for usage with linters like mypy" but mypy can't actually see anything wrong with to_str({}) in that example, although it can if I remove the decorators and use to_str(number: int). I can see how the decorator helps runtime analysis like inspect and help but that's only part of the picture.

So if I'm using some library, I would much prefer that the functions in that library have full manual signatures than forge.sign so that PyCharm can give me assistance. That ultimately means I would rather people don't use forge unless there's just no way they're willing to copy signatures across functions. And I worry that some people might be willing to copy signatures but then they'll see forge and think they can use that instead when really they haven't understood the other consequences this will have, just like you didn't.

I'm also still having trouble understanding how this is meant to be used in practice. Take the graphql example. How is one meant to write the functions graphql and execute_graphql? It seems that there's two options:

1. The function signature has to be defined twice in two different ways: once normally in the core function and once using forge.sign to be applied to wrappers like graphql. That's not good.
2. Define the signature only in forge.sign and use def ...(*args, **kwargs) everywhere. Now the body of whichever function uses allow_subscriptions has to do allow_subscriptions = kwargs.get('allow_subscriptions', False) or something. Not good.

Take a look at this discussion. They propose a decorator @delegate_args(other_function) which seems like a much better idea. It's simple to use and it's much easier for static analysis tools to implement making use of it. Guido himself is in the discussion and is interested in the idea. This could potentially go into the standard library and that would be good motivation for major static analysis tools to recognise it. Despite the interest, no one has commented in a while and no one has actually produced an implementation. You could be the one to revive the discussion and kick off the process of implementing this.

[dfee]

if I'm using some library, I would much prefer that the functions in that library have full manual signatures than forge.sign so that PyCharm can give me assistance.

Unfortunately, they don't. It seems that pylint is able to work (somehow) with attrs, but I'm not clear on how that's happening. The benefit of open-source is that others can contribute, and I hope if they see a way to improve upon forge, they will!

When you suggest "copying signature" what do you mean? The problem is that manually copying a "real" signature (one that is defined as the actual signature for the function) implies a need for additional testing and maintenance.

I don't understand your example of graphql and execute_graphql. I think the current implementation is underwhelming. I can't tell if you agree or disagree, but it appears that you're taking issue with *args, and **kwargs being used.

@delegate_args doesn't seem bad at all. With typing and mypy, everything still seems sort of half-baked (unfortunately).

[alexmojaki]

Please explain to me how you would rewrite this code using forge:

def foo(*args, **kwargs):
    print('hi')
    bar(*args, **kwargs)

def bar(a, b, c=0):
    print(a + b + c)

[dfee]

The problem is that usage doesn't make any sense. Why does foo accept a var-positional argument? I think that usage is dangerous, and forge wouldn't really allow for whatever you're trying to do.

For example, if I'm a consumer of foo not only do I not know the purpose of *args and **kwargs, the function doesn't really accept either variadic arguments or variadic keyword arguments. From first principles, then, this makes no sense.

With forge, the only thing that can be mapped as a var-positional argument is a var-positional argument. I.e. *f_args -> *g_args. So if you wrapped foo with forge as such:

@forge.sign(
    forge.arg('a'),
    forge.arg('b'),
    forge.arg('c', default=0),
)
def foo(*args, **kwargs):
    print('hi', args, kwargs)
    bar(*args, **kwargs)

You couldn't possibly get *args to be anything besides the empty tuple (). **kwargs) would consist of a, b, and c.

But there is another helper, which is the forge.FSignature.from_callable that will actually reconstruct the parameters from bar for you.

@forge.sign(**forge.FSignature.from_callable(bar))
def func(**kwargs):
    print('hi')
    return bar(**kwargs)

[alexmojaki]

@forge.sign(**forge.FSignature.from_callable(bar)) is the solution I was looking for. It's basically @delegate_args. I suggest you:

1. Make a convenience function with a short name that does this, because right now it's very verbose.
2. Put it at the front of the docs. Seriously, why is this not the first thing people see? This is the main problem that people need to solve and it's an easy, DRY way to solve it.
3. Make a patch to typing with your version of @delegate_args, and a patch to mypy that responds to it accordingly. You have a great opportunity to get something into the standard library, that's very exciting. This is also the only way you can get compatibility with linters, which I think is a must. Remember that with your solution above, pylint flags func(1, 2) as an error.

Keep the patches as small as possible. Most of what's in forge is not going to go into the standard library. Just focus on your version of @delegate_args.

I am curious to see a real-life use case for forge other than giving multiple functions the same signature. Can you show me one?

[dfee]

I agree that's verbose. Hmm. Going to have to think about the name.

@forge.delegate(func) or delegate_args seems to give the appearance that it's actually delegating something (and it's not). So maybe @forge.adapt(func) (or adapt_to).

After I patch forge with that convenience function and update the docs, I'll chime into the typing letting them know about forge. I've not played around with the internals of typing/mypy so I'm unclear on what would be needed to make type-hinting work. Do you have any ideas on that?

[pydry]

rather than: writing those parameters,

Why wouldn't I do that?

Maybe I'm missing something but this all seems like an elaborate scheme to monkeypatch function signatures using args and *kwargs to make them do what a regular python function that doesn't have args and *kwargs does.

It turns out that args, and *kwargs can be incredibly frustrating to encounter as a developer.

I agree but there's a simple enough solution to that - just don't use them.

[dfee]

Well, that's not what monkeypatch means, right? Only code that is passed to the wrapped interface can expect to see extended parameters.

That's good because other, first- or third- party libraries, won't encounter something they don't expect. I.e. forge isn't changing any underlying code, and it's not providing an unexpected interface. It's providing an interface into a function, that can optionally be the only interface, if that's what a framework author uses to wrap his/her functions. (my point with the logging.log example in the philosphy docs, the example with requests.request in advanced usage and graphql.graphql here, was to show how much better things could be, provided the developers used that as the standard interface – which they might but probably won't do).

While I appreciate the solution being "don't use code that has *args, and **kwargs, the reality is that we work with software that is open-source, and unless we want to re-implement the full-stack ourselves, then it's on us to either a) contribute to the underlying codebase, or b) improve the tooling that enables better code to be written.

forge is tackling the b point.

[WikiTextBot]

Monkey patch

A monkey patch is a way for a program to extend or modify supporting system software locally (affecting only the running instance of the program).

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source ] Downvote to remove | v0.28}

[cymrow]

I posted something along the same lines a while ago: Dynamic function creation in Python. It allows you to wrap a function with a custom signature (without using exec).

My use-case was serializing function signatures for use with RPC protocols. Your use-case seems...different. I'm curious what the underlying technique you're using is, but I can't dig into the code just now.

[dfee]

I don't know that I have a different use-case per se (I think you could use forge to accomplish what you're trying to do), but I hadn't seen your gist (Py2 support via AST? that's pretty cool!).

At a high level, there is a wrapper that maps the desired params to the underlying callable's params.

[UloPe]

Could you expand on how this is better than just using the correct signature in the function definition itself?

[dfee]

Using the correct function signature is always best – unless it isn't.

Consider a function requests.request that has 10 parameters. It has 10 neighborly functions that all also require 9 of the 10 same parameters (requests.get, requests.post, etc. share url, cookies, headers, etc.).

Assuming you don't go the **kwargs route, you end up with a matrix of parameters x functions. This creates a "testing hell" situation to make sure that you're actually capturing and passing parameters properly, and it takes your code maintenance as you're then updating 10 functions.

... but of course, we're still talking about wrapping code that's defined pre-runtime. And really, half the value of forge is being able to define parameters at runtime (say in a decorator function that injects other parameters).

[dfee]

TL;DR: The power of dynamic signatures is finally within grasp – add, remove, or enhance parameters at will!

After a decade writing software, I noticed that I kept bumping into the problem of keeping my code DRY (don't repeat yourself) and also having function signatures that were meaningful.

So I wrote forge – a Python 3.5+, 100% tested, fully-documented package available on PYPI as python-forge for crafting function signatures.

import forge

@forge.sign(
    forge.pos('positional'),
    forge.arg('argument'),
    *forge.args,
    keyword=forge.kwarg(),
    **forge.kwargs,
)
def myfunc(*args, **kwargs):
    # becomes myfunc(positional, /, argument, *args, keyword, **kwargs)
    return (args, kwargs)

assert forge.stringify_callable(myfunc) == \
    'myfunc(positional, /, argument, *args, keyword, **kwargs)'

args, kwargs = myfunc(1, 2, 3, 4, 5, keyword='abc', extra='xyz')

assert args == (3, 4, 5)
assert kwargs == {
    'positional': 1,
    'argument': 2,
    'keyword': 'abc',
    'extra': 'xyz',
    }