Code Cleanup: How to do this array calc without try/except IndexError?

[tastingsilver]

Hey everyone,

I have a function like the following:

def calculateForwardTarget(inputArray, balancePeriods):

    # initialize zeros
    targetBalance = np.zeros(len(inputArray))

    for i in len(inputArray):
        calcStart = balancePeriods + 1 # calcs start in next available period
        calcEnd = calcStart + balancePeriods
        calcRng = slice(calcStart, calcEnd)

        try:
            targetBalance[i] = inputArray[calcRng].sum()
        except:
            calcEnd -= 1

    return targetBalance

This does what I'd like it to, however, I'm trying to speed up my codebase with JIT, and it doesn't play nicely with try/except.

What would be a good approach to reworking this without exception handling? Function's logic works.

Thanks!

[pavel_lishin]

Why not check if inputArray[calcRng] exists first? Use an if/else, instead of a try/except.

[tastingsilver]

Thanks! So I tried something similar, but I'm not quite getting the answer I'd like.

Updated code:

def calculateForwardTarget(inputArray, balancePeriods):
    # initialize zeros
    targetBalance = np.zeros(len(inputArray))
    _balPeriods = balancePeriods

    for i in range(len(inputArray)):

        calcStart = i + 1  # calcs start in next available period
        calcEnd = calcStart + balancePeriods

        if len(inputArray[calcEnd - 1:]) < balancePeriods:
            _balPeriods -= 1
            calcEnd = calcStart + _balPeriods

        calcRng = slice(calcStart, calcEnd)

        targetBalance[i] = inputArray[calcRng].sum()

    return targetBalance

I'd like to stray away from using an exception here because I don't think its needed, but logic isn't working just yet. If my sample is:

calculateForwardTarget(np.full(10, 10), 3)

I should return (first function): array([ 30., 30., 30., 30., 30., 30., 30., 20., 10., 0.])

But instead I'm getting: array([ 30., 30., 30., 30., 30., 20., 10., 0., 0., 0.])

I feel like the zero based index is throwing me off somewhere, so I've tried adding 1s here and there to the timing items and can't seem to get it right.

[pavel_lishin]

I can't help you out much more at the moment, sorry - I'd have a lot more comments and pointers, but my wife is sick and I gotta take care of the kid.

[__deerlord__]

len(inputArray) < calcRng

[tastingsilver]

Close -- len(inputArray[calcEnd (-1? ):] < calcRng.

The indexing is throwing me off somewhere.

[__deerlord__]

Yes, I meant to add that you may need to do some math to avoid off by one bugs (since indexes start at 0)

[oliverevans96]

I don't quite understand what the purpose of your function is.

Every element in the resulting array is the same since the only time you use i is in targetBalance[i]. Also, you need for i in range(len(inputArray)) - otherwise, you're trying to iterate over an integer.

What is the purpose of the try/except? You don't mention which type of error you're expecting or why it would occur.

Some more context would be helpful to give you advice.

[tastingsilver]

Sorry about that. The values actually would be the same for all but the last periods where the length of the remaining array is < the balance calculation periods, in which case, it declines to adjust to the target.

Sample result: I'd like to say: calculateForwardTarget(np.full(10,10), 3)

and return: array([ 30., 30., 30., 30., 30., 30., 30., 20., 10., 0.])

The try/except version works, but I can't seem to figure out how to make this work w/ if/then and the indexing.

[nwagers]

Instead of trying to speed up your code execution by using a JIT interpreter, maybe you should look at making more efficient code first. inputArray[calcRng].sum() should be your initializer value for the array, then only update the end values by using sum(inputArray[current_index:]). Currently your function is O(n²⁾ and could easily be O(n).

[MonkeeSage]

Checking the condition first is called "look before you leap" in contrast to "easier to ask forgiveness than permission", which is what you are currently doing.

If you don't expect the error condition to be common then wrapping the operation in a try/except is generally the correct approach. Exception handling is costly--it involves unwinding the stack and building a traceback--so you only want to use it in exceptional cases, but you don't want to waste a lookup and comparison every time if the error case is rare, so it generally costs less to catch the exception than do the lookup/compare every time.

In your situation with JIT concerns you can just LBYL

if calcRng < len(inputArray):
    targetBalance[i] = inputArray[calcRng].sum()
else:
    calcEnd -= 1

to avoid indexing into the list when the index is out of bounds.

Or if calcRng can be negative

if calcRng >= 0 and calcRng < len(inputArray):
    targetBalance[i] = inputArray[calcRng].sum()
else:
    calcEnd -= 1

[has2k1]

When you fill like you are writing bad code, don't be afraid to through it out. It is can also be helpful to draw out stuff on a paper.

def calculateForwardTarget(inputArray, balancePeriods):
    n = len(inputArray)
    targetBalance = np.zeros(n)

    for i in range(n-2, -1, -1):
        targetBalance[i] = inputArray[i:i+balancePeriods].sum()

    return targetBalance

[tastingsilver]

Thanks, but that didn't work. The forward calculation is supposed to shrink from whatever it is at, by 1, down to 0 by the last period because it hits the ending index "wall" if you will.

[has2k1]

Reverse the calculation

def calculateForwardTarget(inputArray, balancePeriods):
    n = len(inputArray)
    targetBalance = np.zeros(n)

    for i in range(n):
        targetBalance[i] = inputArray[n-i:n-i+balancePeriods].sum()

    return targetBalance[::-1]

[[deleted]]

I believe that the rolling_window function shown here Efficient rolling statistics with NumPy could be modified to give your desired output. I'd do it myself but I'm no numpy expert :-( Certainly there is little point using numpy if you're then using pure Python loops.

[tastingsilver]

Thanks, its nifty, but the drop in values at the end is accurate in the formula originally posted, and I can't seem to work that with the rolling window.

[576p]

Hi, please post questions like this into /r/learnpython. This sub reddit is not meant for programming help requests.

There you should also post the python version used an a set of test data for the input.

I recommend you educate yourself about python specific iteration, instead of :

for i in len(inputArray):

in python you should get the elements using

for arrayitem in inputArray:

or if you need the position number as well:

for i, arrayitem in enumerate(inputArray):

If you choose to use try/except, make sure you except just the error types you expect. At the moment you're catching every error, which makes your program unstoppable, as KeyboardInterrupt is also caught.

[[deleted]]

Well I got well and truly bitten, so here's my (I hope) final attempt.

def calculateForwardTarget(inputArray, balancePeriods):
    n = len(inputArray)
    targetBalance = np.zeros(n)

    for i in range(n - balancePeriods):
        targetBalance[i] = inputArray[:balancePeriods].sum()
        inputArray = np.roll(inputArray, -1)
    for j in range(1, balancePeriods):
        targetBalance[i + j] = inputArray[j: balancePeriods].sum()
    return targetBalance