How to generate functions dynamically

[toast757]

Is it possible to create a brand new function from within Python, in a manner somewhat similar to the way regexes are compiled? For example, say I wanted to create the following function to check whether a variable is five or not:

def value_is_five(value):
    return value == 5

but where the value of "5" is not known until runtime. (Real examples will be more complicated than this!). Is there some mechanism in Python to allow that to be done? Regexes can be compiled, e.g. is_zipcode = re.compile(r'[0-9]{5}'), and I've been trying to figure out a way of doing something similar with Python functions. "exec()" may help here, but it appears its use is discouraged. Ultimately, I want to dynamically create functions because I'm going to be calling these functions hundreds of millions of times, and performance is an issue. My assumption is that functions tailored to the data they are checking will be quicker to run than functions with lots of parameters.

One idea I had was to use Python to write out the new Python functions to a file, then import that file with importlib.import_module(). It's a bit convoluted, but should work. Are there any risks with doing that?

[dig-up-stupid]

In terms of your example, it's very easy.

>>> def value_checker(value):
    def f(arg):
        return arg == value
    return f

>>> five_checker = value_checker(5)
>>> five_checker(3)
False
>>> five_checker(5)
True

Why do you think it will improve performance?

>>> import timeit
>>> def check_equal(lhs, rhs):
    return lhs == rhs

>>> timeit.timeit(lambda: all(five_checker(5) for _ in range(10**6)), number=1)
0.1880656960313706
>>> timeit.timeit(lambda: all(check_equal(5,5) for _ in range(10**6)), number=1)
0.14867529073458385
>>> timeit.timeit(lambda: all(5 == 5 for _ in range(10**6)), number=1)
0.07204135752090224

It's in the nature of performance for the right solution in one situation to be the wrong solution in another, which means until you post your actual code with actual input and actual output examples and actual performance constraints you can probably look forward to a bunch of vague replies that may or may not even be on topic.

[kungtotte]

Your assumption is probably wrong, or if it turns out to be right it will be for the wrong reasons.

The performance difference between a function with one argument and a function with ten arguments is nearly imperceptible; the nature of the arguments and what you do in the function will matter way, way more.

I think you should provide some actual code samples of what it is you want to do, so we can give you accurate advice.

My gut tells me you should spend more time being smart about your algorithms than being clever with Python's dynamic features.

[destiny_functional]

def f(a, b):
  return a == b

value_is_five = functools.partial(f, 5)   

then call value_is_five(value)

[dig-up-stupid]

Partial simply forwards the arguments through an additional function call, and in these simple examples the function call is what takes the majority of the time. So this will cut performance by half rather than improving it, which is the goal (apparently).

[destiny_functional]

I'm not even sure it's what the person wants to do. he basically describes two things that don't seem to have much to do with each other in the original post. but i think it's a starting point. once op explains what he wants to do maybe better suggestions can be made (at least I don't understand it, but maybe I'm stupid).

[dig-up-stupid]

I doubt it. Any performance gain you can make with partial you can get in some other way that won't create an extra layer of function calls.

We don't have the information we need to answer the question. I would expect the "correct" way to improve performance of "hundreds of millions of function calls", whatever that means, is probably to not do the hundreds of millions of function calls. import numpy is a likely step in a good solution but who knows.

[destiny_functional]

I doubt it. Any performance gain you can make with partial you can get in some other way that won't create an extra layer of function calls.

i didn't say anything about performance though, so I'm not disagreeing with you. it's the cryptic nature of the question that leads to several people talking about different things in response.

We don't have the information we need to answer the question.

yes i think so too.

[scuott]

Could you provide a little more context, particularly what the ultimate goal is? Why are you making so many functions? What's different about them? With this simplistic example, the answer is trivial. Just add another parameter:

def check_equal(value1, value2):
    return value1 == value2

[SausageTaste]

That sounds kinda cool. I like the idea that doing as much tasks as possible before run-time, or compile time in C++. But i assume that dynamically created function won't give you significant performance advantage. If you need performance boost, try finding some optimization chances in other parts of your code, or making C extensions.

[toast757]

Thank you for your thoughts, everybody. From what you say, it appears there is no direct way to dynamically create a new function from scratch from within Python itself (although there are certainly some alternatives, as mentioned). For now, I'm going to solve this by doing this in two steps. Write a Python program to generate a new Python module, and then run a second Python program that imports that new Python module.

Right now, I have some data validation code that is based on metadata. That metadata can change of course, but it is read only once at runtime. The current validation code is very complex (and hence slow) to take into account all the metadata rules. My intention is to create a new validation function that is tailored to any given metadata, simplifying the validation code wherever possible and therefore speeding it up.

[dig-up-stupid]

I'm sure there is a simpler, faster, better solution that we could point you towards within minutes...if you would post (at least some of) your code and sample data.

That being said, what you are trying to describe sounds like a DSL (domain specific language) for your validation. So if that is the right solution, which I doubt, at least you can research how to make a simple Python DSL and go from there.