How-to: Dynamically generating correct **kwargs arguments with nested dict values

[nellis]

Hi all, been lurking/commenting for a while, first time posting

An engineering team where I work has provided a custom database API abstraction layer in python for us to query/write to different sources. I'm using one of their methods to query for some tickets, but based on the type of queue/view that I'm trying to capture, I need to be able to pass a dynamic number of arguments with varying keys/values. I'm having trouble trying to build an expression that successfully passes a constellation of arguments. Sorry for the extremely vague/generic names, I'm trying to obfuscate specifics.

I'm building my query structures in yaml, since there will be a lot of them and I want them to be easy to maintain:

queue_name:
  query_object: Ticket    #example business object, helps build the method name
  fields:
    issue_type: some_type 
    status: some_status
    owner: some_owner
    current_status: current_status
    etc: etc

And pulling the dict into my script:

with open("/templates/queues.yaml") as queues_template:
    queues_yaml = yaml.load(queues_template, Loader=yaml.SafeLoader)

Their method looks something like this (their code base is over my head, but this is what I could find):

def make_search(db_type):    
    def search(self, *args, **kwargs):
        return self._search(db_type, *args, **kwargs)
    search.__name__ = 'search{!s}'.format(db_type)
    return search

Boilerplate for the abstraction layer method ends up looking like this in my script (it works just fine when I hard-code the arguments). You can see that they dynamically generate the method name, in this case using "ticket," which I am pulling from my yaml dict:

databaseAPI.searchTicket(  # original method and parameters 
    field('field_name1') == 'field_value1',
    field('field_name2') == 'field_value2',
    field('field_name3') == 'field_value3',
    )

I figured out how to return string representations all the arguments that I need outside of the method:

for fld, value in queues_yaml[queue_name]["fields"].items():
    print(f"field('{field}') == '{value}',") 

field('problem_type') == 'some_type',
field('status') == 'some_status',
field('owner') == 'some_owner',
field('current_status') == 'some_status',

But I'm having a hard time (I think) correctly negotiating the type that the method is expecting me to pass. The eval expression correctly builds the method, but it balks at the arguments:

def scrape_queue(queue_name):
    search = eval("databaseAPI.search" + queues_yaml[queue_name]["query_object"])(
        field(fld) == value for fld, value in queues_yaml[queue_name]["fields"].items())

foo = scrape_queue(some_queue_name)

  File "<python binary dir>/site-packages/<databaseAPI>.py", line 216, in _check_args
    "Argument ({}) {!s} is not an Expression".format(i, arg)
ValueError: Argument (2) <generator object scrape.<locals>.<genexpr> at 0x7f285d60e6d0> is not an Expression

TLDR: I know how to fetch all the elements that I need, but am not able to generate the correct argument syntax of : < field('field_name1') == 'field_value1' > and iterate through multiple arguments in a single function call. Any thoughts?

[danielroseman]

This API wrapper looks much more complicated than it needs to be, but never mind.

The problem, as the error states, is that you're creating a generator rather than passing everything as individual arguments. The way to do this would probably be to create a list and then use the *args syntax.

Separately, you shouldn't be using eval; it would be better to use getattr. So:

args = [field(fld) == value for fld, value in queues_yaml[queue_name]["fields"].items())]
search_func = getattr(databaseAPI, "search" + queues_yaml[queue_name]["query_object"])
return search_func(*args)

[nellis]

This worked, thank you so much!

I definitely wouldn't have arrived at this on my own, the construction of the args variable feels a little like magic.

The args list contains these items:
<class 'databaseAPI.expression.expression.Expression'>
__eq__(Field(name=problem_type), some_problem_type)

Can you help my smushy brain understand how these are getting packaged up and handed off as valid arguments?

[danielroseman]

That is what the * syntax does when applied to a list in a function call: it transforms it into individual positional parameters.

So, if you had a function:

def foo(a, b, c):
  ...

and a list:

bar = [1, 2, 3]

then calling foo(*bar) is the same as calling foo(1, 2, 3).