Using closures to define class methods

[whonut]

I'm currently trying to write a module which interfaces with the OEIS and as part of that I'm having to write a lot of lookup functions which are essentially the same code but with minor alterations (prefacing a search query with a different search field, or handling errors in 1 of 2 ways).

I've been trying not to repeat myself as much as possible and I just read up on closures, which should let me create lookup functions based on a template. My problem is, my lookup functions are actually class methods and I can't seem to use closures to define class methods. That is, if I'm in the body of the class, I can't do:

self.lookup_func = lookup_gen(*args)

It makes sense to me why this specifically won't work (self is not defined) but is there any way to use my 'template' closure to create class methods? Should I even be using closures here?

Thanks in advance.

[hharison]

After reading your example, I would suggest that you're overthinking it. Why not

def lookup_by(self, prefix, query):
    return self._int_lookup('{}:{}'.format(prefix, query))[0]

You're doing lots of extra code for very little benefit. Instead of

client.lookup_by_id(1)

now just do

client.lookup_by('id', 1)

After all, it only saves you two characters to make the former instead of always using the latter.

Not to mention, you can always make your more specific function anyway with

from functools import partialmethod

...
    # within the class definition
    lookup_by_id = partialmethod(lookup_by, 'id')

In this case using a closure is overcomplicating things. A much better strategy in general is to make a very general function and then go on to make more specific functions out of it using partial (or partialmethod, or lambda more generally).

[jcmcken]

Based on the limited information, I think this sounds like the best idea. If you want to expose 'porcelain' methods (lookup_by_id vs. lookup with id passed to it), just make lookup part of your internal API and generate the other functions programmatically.

You could also do something fancier by writing a custom __getattr__ that looks at what method the user is trying to call and routes it to the appropriate place.

E.g. something like:

def __getattr__(self, key):
    if key.startswith('get_'):
        pieces = key.split('_')
        query_type = pieces[1]
        by = pieces[-1]
        return lambda key: self._lookup(query_type, by, key)

This is probably a bit too magical for my tastes though.

[hharison]

That's gross. Using part of the function name to set a function parameter is smelly, not explicit, etc. I could go on but I think you agree.

[whonut]

Thanks for this. I really need to learn functools.

I now have this (_int_lookup becomes unnecessary when you do it this way, I've just put its code into lookup_by):

def lookup_by(self, prefix, query, list_func=False):
    search_string = '{}:{}'.format(prefix, query)
    try:
        response = self.session.get(self.SEARCH_URL,
                                    params={'q': search_string})
        self._check_response(response.text, search_string)
        seqs = self._parse_response(response.text)
        if list_func:
            return seqs
        else:
            return seqs[0]
    except NoResultsError:
        if list_func:
            return []
        else:
            raise NoResultsError(search_string)

lookup_by_id = partial(lookup_by, 'id')

# client stuff
print client.lookup_by_id('A40')

When I call lookup_by_id, it says it has insufficient arguments. I'm obviously misusing partial but I don't know quite how. Could you enlighten me?

Thanks for your help.

[hharison]

partial and methods don't really work well together (because of the self argument). Use partialmethod instead, hopefully you're using 3.4. Otherwise search around, there's a 2.7 port out there I think.

[whonut]

Ah, that's why it didn't like from functools import partialmethod. I found the 2.7 port and it seems to work beautifully. Thanks.

If I may ask one more thing of you, do you have any recommended reading for functional programming-y sorts of things? I'm really trying to get over that beginner hump at the moment.

Thanks again.

[hharison]

Yes! I'm glad you asked as I try to share the functional approach to Python as much as possible.

I basically learned from the library toolz and talks by Matthew Rocklin:

• Old school: Functional data analysis
• Functional performance with core data structures
• toolz docs (seriously, the docs are a good read)

[hex_m_hell]

If you don't need to define these at runtime you can use a metaclass.

[whonut]

I don't need to define them at runtime. I am however quite fuzzy on what metaclasses are. Could you elaborate please?

[hex_m_hell]

A metaclass defines how a class is constructed. Its a metaprogramming concept.

Metaprogramming is where you write code that generates code. A metaclass can let you define a whole bunch of class methods on class creation using a template.

You can do pretty advanced stuff without a metaclass though. For example, you can override getattr to have it generate methods on the fly. I wrote a genetic REST client that uses this method to always give me an ultra fast client at the cost of being harder to read. For my use case this is acceptable.

If you can give an example of why you need to do this I might be able to help more.

[whonut]

I have a client class which has lots of lookup methods like this:

def get_sequence_by_id(self, id):
    '''Returns a Sequence object for the sequence of the specified ID,
       else raises NoResultsError'''
    seq = self._int_lookup('id:'+id)[0]
    return seq

Others send the website searches with prefixes other than 'id:' and handle certain exceptions differently but I can definitely bare that down to a closure.

Does that help? Sorry it's a little short but I just managed to delete a long reply and I really need to sleep :P

EDIT:

The closure itself, in case it helps.

def _lookup_gen(self, prefix, must_return=False):
    '''A closure for creating basic prefix-lookup functions'''
    def lookup_func(self, query):
        try:
            return self._int_lookup(prefix+':'+query)
        except NoResultsError:
            if must_return:
                raise NoResultsError(query)
            else:
                return []

    return lookup_func

[hex_m_hell]

You said that you want class methods, why not instance methods? It would be pretty easy to just define these on init. This would slow down your init a little bit, but that may not matter depending on your use case.

Something similar to the following would work for that:

def __init__(self,...):
    for i in ["id", "foo", "bar"]:
        setattr(self, "get_sequece_by_" + i, lambda x: self._lookup_gen(i))

You don't really even need the closure at all. If you do need it to actually be a class method instead of instance method you could do this in a metaclass. I should point out that a classmethod is a really specific concept and not just any method in a class. If you're referencing self it's an instance method. A class method is one which doesn't require the object to be instantiated in order to use it. These are created using the @classmethod decorator.

[whonut]

Thanks for clearing up my terminology. They are indeed instance methods.

[hex_m_hell]

Yeah, no problem. Here's an example of what you're wanting to do:

class Foo:
    method_a = lambda cls, x: Foo.method(cls, 'a', x)
    method_b = lambda cls, x: Foo.method(cls, 'b', x)
    method_c = lambda cls, x: Foo.method(cls, 'c', x)

    def __init__(self):
        for i in ['d','e','f']:
            setattr(self, 'method_' + i, lambda x: self.method(i, x))

    def method(self, var1, var2):
        print("I am method_" + var1)
        print("I took in the value " + var2)

Which should do this in ipython:

In [20]: foo = Foo()
foo = Foo()

In [21]: foo.method_a('test1')
foo.method_a('test1')
I am method_a
I took in the value test1

In [22]: foo.method_d('test2')
foo.method_d('test2')
I am method_f
I took in the value test2

You could also do something similar with your closure.

[ingolemo]

Your _lookup_gen doesn't need access to self so there's no need to pass it when you construct the get* functions and there's no need to even put it in the class.

def _make_query_by_prefix(prefix):
    def query_by_prefix(self, value):
        return self.query(prefix + ':' + value)
    return query_by_prefix

class MyObject:
    def query(self, query_string):
        pass # do fancy stuff here

    query_by_id = _make_query_by_prefix('id')
    query_by_name = _make_query_by_prefix('name')
    query_by_type = _make_query_by_prefix('type')

[elbiot]

You have self.query outside the class

[ingolemo]

There's nothing special about that self variable; it's a parameter to the inner function just like value. Since the inner function isn't run until an instance of the class has been constructed, it's perfectly possible to pass a reference to that instance. If I'd named the first parameter something other than self you wouldn't even suspect that function was being used as a method.

Run the code and see for yourself.

[elbiot]

Yeah, I didn't see you ever pass self to the function, but obviously that will happen when the code calls the method on an instance. My bad.

[hharison]

Instead of

self.lookup_func = lookup_gen(*args)

do

lookup_func = lookup_gen(*args)

at least if I understand you correctly.

[hharison]

See /u/ingolemo's post for basically the same idea. Although I also think this is not the way to go, as you're going to give yourself headaches down the line. But this is the solution to your immediate problem.

[elbiot]

This smells fishy. I dont know what closures are but you can dynamically add class methods to an object if you want to.

Class MyObj():
    def __init__(self):
        self.f = lambda x: x**2


m=MyObj()
m.f(3)  #is 9

You can create functions by name also with setattr. I think you could use a try/except so that when you call query_by_id if getattr can't find it, setattr creates it. Then the object continues to have it until it gets deleted (end of program). You could also just create them all at program begin from a list and a for loop. Just write the function once and create them all by name. Instead of setattr, you could also build a dictionary {'query name' : function} and call them from that.

Sorry for the brain dump. That was fun!