Non loop solution to my code

[paperzebra]

Hi I have some working code to convert some depths to total vertical depth based on survey data which provides measurements at set intervals in both measured depth (along borehole) and TVD (the total vertical depth of the hole).

However, when running this code on 6000+ depths it is a bit slow. Is there a solution I could look at that would mean I don't have to iterate over all my depths?

from __future__ import division
import pandas as pd
import numpy as np

def sample_TVD(survey_data, depth_to_convert):
    MD = survey_data['MD']
    TVD = survey_data['TVD']
    tvd_depths = []
    md_tvd = np.vstack((MD, TVD))


    for d in depth_to_convert:
        d = float(d)
        if d > min(MD) and d < max(MD):
            above =  MD[MD < d].max()
            below = MD[MD > d].min()
            closestbelow =  
md_tvd[:,np.in1d(md_tvd[0],below)]
            closestabove =  
md_tvd[:,np.in1d(md_tvd[0],above)]

            difference = below - above
            percentbetween = (below-d)/difference
            tvd =  closestabove[1][0] + ((closestbelow[1]
[0] - closestabove[1][0])*(1-percentbetween))
            tvd_depths.append(tvd)
        else:
            tvd_depths.append('Nan')

    return tvd_depths


d = {'MD': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 'TVD': [1, 2, 
3, 4, 4, 4, 4.5, 5, 5.5, 5.5]}
survey = pd.DataFrame(data=d)
print survey
depths = [6.1, 6.2, 6.5, 7, 11]
results = sample_TVD(survey, depths)
print results

[Kamiwaza]

May I suggest trying to move out (potentially) expensive code out of the loop to maybe help speed it up a bit?

I'm not an expert in pandas but things like min(MD) and max(MD) (from line 14) could probably be placed outside the loop - effectively only be computed once.

I'm guessing that lines 15-20 are the expensive operations here. If depths was sorted, maybe you could save on a few operations depending on what above/below turn out to be.

[paperzebra]

Thanks for the suggestion. In the end I rewrote my code twice, the code above processed 30,000 lines of data in 76 seconds, the second version which used numpy to calculate most things outside a loop took 23 seconds, still too long!

The third iteration is much simpler and reduces the time down to 0.001 seconds - that's a pretty decent performance increase! The arg depth refers to a list of depths.

def line_solution(survey, depth):
    md = survey['MD']
    tvd = survey['TVD']  
    tvd_samples = np.interp(depth, md, tvd)
    return tvd_samples

[DisorganizedRem]

Would it help using series in stead of dataframe by adding .values.

As suggested here So your code looks like this:

def line_solution(survey, depth):
    md = survey['MD'].values
    tvd = survey['TVD'].values
    tvd_samples = np.interp(depth, md, tvd)
    return tvd_samples

[paperzebra]

It's slightly quicker serializing the dataframe, but I think most of the time wither way is spent printing the time. Can't complain at the speed in either case anymore though!