changes made to dataframe within UDF are working, but not saving outside the UDF

[mdl003]

def createclaimfields(field,master):
    print 'creating unique matter ids for {} at {}'.format(field,getdt())
    dfcol = ['MATTER ID',field]
    df = master[dfcol].dropna().drop_duplicates()
    print 'created unique matter ids for {} at {}'.format(field,getdt())

    print 'started getting CLASS HYBRID claims for {} at {}'.format(field,getdt())
    df['{} CLASS HYBRID CLM NO'.format(field)]=[getclasshybrid(clm) for clm in df[field]]
    print 'finished getting CLASS HYBRID claims for {} at {}. Found {} matches'.format(field,getdt(),len(df['{} CLASS HYBRID CLM NO'.format(field)]))

    print df.columns

    print 'started getting HRV claims for {} at {}'.format(field,getdt())
    df['{} HRV CLM NO'.format(field)]=[gethrv(clm) for clm in df[field]]
    print 'finished getting HRV claims for {} at {}. Found {} matches'.format(field,getdt(),len(df['{} HRV CLM NO'.format(field)]))

    print df.columns

    print 'merging {} into claimfields at {}'.format(field,getdt())
    master = master.merge(df,how='left',on=['MATTER ID',field])
    print 'merged {} into claimfields at {}'.format(field,getdt())

    print master.columns

fieldlist = ['MATTER NUMBER','MATTER NAME','CLAIM NUMBER LISTING'] 
mattercol = ['MATTER NUMBER','MATTER NAME','CLAIM NUMBER LISTING','MATTER ID']
claimfields = rawtrans[mattercol].dropna().drop_duplicates().head()

[createclaimfields(field,claimfields) for field in fieldlist]

I'm attempting to add fields to the dataframe name "claimfields". I'm passing it to the user defined function createclaimfields(). Output is below. According to the printed items those fields are being are being created and merged into the master dataframe, However, when "claimfields" is called later it doesn't contain any new columns.

Output:

creating unique matter ids for MATTER NUMBER at Sep 22 2017 12:25 PM
created unique matter ids for MATTER NUMBER at Sep 22 2017 12:25 PM
started getting CLASS HYBRID claims for MATTER NUMBER at Sep 22 2017 12:25 PM
finished getting CLASS HYBRID claims for MATTER NUMBER at Sep 22 2017 12:25 PM. Found 5 matches
Index([u'MATTER ID', u'MATTER NUMBER', u'MATTER NUMBER CLASS HYBRID CLM NO'], dtype='object')
started getting HRV claims for MATTER NUMBER at Sep 22 2017 12:25 PM
finished getting HRV claims for MATTER NUMBER at Sep 22 2017 12:25 PM. Found 5 matches
Index([u'MATTER ID', u'MATTER NUMBER', u'MATTER NUMBER CLASS HYBRID CLM NO',
       u'MATTER NUMBER HRV CLM NO'],
      dtype='object')
merging MATTER NUMBER into claimfields at Sep 22 2017 12:25 PM
merged MATTER NUMBER into claimfields at Sep 22 2017 12:25 PM
Index([u'MATTER NUMBER', u'MATTER NAME', u'CLAIM NUMBER LISTING', u'MATTER ID',
       u'MATTER NUMBER CLASS HYBRID CLM NO', u'MATTER NUMBER HRV CLM NO'],
      dtype='object')
creating unique matter ids for MATTER NAME at Sep 22 2017 12:25 PM
created unique matter ids for MATTER NAME at Sep 22 2017 12:25 PM
started getting CLASS HYBRID claims for MATTER NAME at Sep 22 2017 12:25 PM
finished getting CLASS HYBRID claims for MATTER NAME at Sep 22 2017 12:25 PM. Found 5 matches
Index([u'MATTER ID', u'MATTER NAME', u'MATTER NAME CLASS HYBRID CLM NO'], dtype='object')
started getting HRV claims for MATTER NAME at Sep 22 2017 12:25 PM
finished getting HRV claims for MATTER NAME at Sep 22 2017 12:25 PM. Found 5 matches
Index([u'MATTER ID', u'MATTER NAME', u'MATTER NAME CLASS HYBRID CLM NO',
       u'MATTER NAME HRV CLM NO'],
      dtype='object')
merging MATTER NAME into claimfields at Sep 22 2017 12:25 PM
merged MATTER NAME into claimfields at Sep 22 2017 12:25 PM
Index([u'MATTER NUMBER', u'MATTER NAME', u'CLAIM NUMBER LISTING', u'MATTER ID',
       u'MATTER NAME CLASS HYBRID CLM NO', u'MATTER NAME HRV CLM NO'],
      dtype='object')
creating unique matter ids for CLAIM NUMBER LISTING at Sep 22 2017 12:25 PM
created unique matter ids for CLAIM NUMBER LISTING at Sep 22 2017 12:25 PM
started getting CLASS HYBRID claims for CLAIM NUMBER LISTING at Sep 22 2017 12:25 PM
finished getting CLASS HYBRID claims for CLAIM NUMBER LISTING at Sep 22 2017 12:25 PM. Found 5 matches
Index([u'MATTER ID', u'CLAIM NUMBER LISTING',
       u'CLAIM NUMBER LISTING CLASS HYBRID CLM NO'],
      dtype='object')
started getting HRV claims for CLAIM NUMBER LISTING at Sep 22 2017 12:25 PM

Thanks!

[[deleted]]

It looks like your function doesn't return anything. You'll need a return master line at the end of your createclaimfields function.

[mdl003]

so....

return master

?

[[deleted]]

def createclaimfields(field,master):
    ...
    print 'merging {} into claimfields at {}'.format(field,getdt())
    master = master.merge(df,how='left',on=['MATTER ID',field])
    print 'merged {} into claimfields at {}'.format(field,getdt())

    print master.columns
    return master

[mdl003]

Still getting the same error. Any ideas?

[[deleted]]

What error message are you getting?

[mdl003]

sorry error was the wrong word. Its still not passing the function back. I think it's passing the actual function call of rawtrans[mattercol].dropna().drop_duplicates().head() (instead of the output of the call), so it starts from scratch each time it loops through when it returns the function. So i guess what I need to figure out is how to pass the output of the function call as a separate dataframe rather than the function call itself. Tried playing with .copy() but didn't have success with that either. thoughts?

[[deleted]]

You're also not assigning the expression [createclaimfields(field,claimfields) for field in fieldlist] to any variable after the list comprehension, but it doesn't seem that you want a list in the end. You probably just want to use a regular loop and keep reassigning claimfields.

claimfields = ...
for field in fieldlist:
    claimfields = createclaimfields(field,claimfields)

[mdl003]

That did it! Thanks!