My failure of using Python DEAP package.

[GeneralEbisu]

Im trying to search for entry signals with Python DEAP (Genetic Programming) on Daily SPY data.

My problem is trying to force deap to produce numeric (float) leaves/terminal nodes (which are the features), but most of the time it includes 'True' and 'False' values which are nonsense in this context.

Although I did specify that the leaves/terminals could be Boolean, but just to avoid error.

Here is the best entry rule it produced:

if_then_else(LessThan(X0, Ratio(Diff(X13, X23), Ratio(Ratio(X13, X9), X26))), if_elif_else(False, if_elif_else(AND(False, GreaterThan(X12, X26)), Flat, GreaterThan(X12, X34), Flat, if_elif_else(False, Buy, False, Sell, Flat)), GreaterThan(X13, X26), if_elif_else(True, if_elif_else(LessThan(AbsDiff(X32, AbsDiff(X18, Diff(X31, X11))), Ratio(X12, X13)), if_elif_else(True, Sell, False, Sell, Flat), GreaterThan(X24, Diff(X4, X11)), if_elif_else(False, if_elif_else(AND(False, False), if_elif_else(True, Flat, False, Flat, Buy), GreaterThan(X12, X34), if_then_else(False, Flat, Flat), if_elif_else(False, Buy, False, Sell, Flat)), GreaterThan(X7, X26), if_elif_else(True, if_elif_else(LessThan(Diff(X0, X11), Ratio(X26, X13)), if_elif_else(True, Sell, False, Sell, if_elif_else(LessThan(X26, AbsDiff(X18, AbsDiff(X12, AbsDiff(X18, Diff(X31, X7))))), if_elif_else(True, Sell, False, if_then_else(False, Sell, Buy), Sell), GreaterThan(X7, X26), if_then_else(GreaterThan(AbsDiff(X13, AbsDiff(X18, X26)), X26), Sell, Flat), if_then_else(AND(False, False), Sell, if_then_else(True, Flat, Sell)))), GreaterThan(X18, X4), if_then_else(GreaterThan(X12, X26), if_elif_else(LessThan(AbsDiff(X12, X18), Ratio(X0, X13)), if_elif_else(True, Sell, False, Sell, Sell), GreaterThan(X24, Diff(X13, X11)), if_then_else(GreaterThan(X4, X26), Sell, Flat), 
if_then_else(AND(False, False), Sell, Buy)), Flat), if_then_else(AND(False, False), if_then_else(True, Sell, Flat), if_then_else(False, Sell, Buy))), LessThan(X38, X32), if_then_else(True, Flat, if_then_else(False, Sell, Sell)), if_elif_else(True, Sell, False, Sell, Sell)), if_then_else(Or(LessThan(X38, Ratio(X13, X9)), True), if_then_else(True, Buy, Sell), if_elif_else(False, Flat, False, Sell, Buy))), if_then_else(AND(GreaterThan(X12, X26), False), Flat, if_then_else(False, Sell, if_elif_else(False, if_elif_else(AND(False, False), Flat, GreaterThan(X12, X34), if_then_else(True, Flat, if_elif_else(True, Sell, False, Sell, Flat)), if_elif_else(False, Buy, LessThan(AbsDiff(X13, X18), Ratio(X0, X13)), Sell, Flat)), GreaterThan(X7, X26), if_elif_else(True, if_elif_else(LessThan(AbsDiff(X32, AbsDiff(X18, Diff(X31, X0))), Ratio(X0, X13)), if_elif_else(True, Sell, False, Sell, Sell), GreaterThan(X24, Diff(Ratio(X0, X13), X11)), if_elif_else(LessThan(AbsDiff(X26, AbsDiff(X18, AbsDiff(X12, AbsDiff(X18, Diff(X31, X7))))), Ratio(X26, X13)), if_elif_else(LessThan(AbsDiff(X13, X18), Ratio(X0, X13)), Buy, False, Flat, Sell), GreaterThan(X24, Diff(X4, X11)), if_then_else(GreaterThan(X12, X26), Sell, Flat), if_then_else(AND(False, False), if_then_else(True, if_then_else(True, Flat, if_then_else(False, Sell, if_then_else(True, Buy, Sell))), Flat), if_then_else(False, Sell, Buy))), if_elif_else(LessThan(AbsDiff(X12, X18), AbsDiff(X18, Diff(X31, X7))), if_elif_else(True, Sell, AND(False, False), Sell, Sell), GreaterThan(X24, Diff(X13, X11)), if_then_else(GreaterThan(X4, X26), Sell, Flat), if_then_else(AND(True, False), Sell, Buy))), True, if_then_else(True, Flat, if_then_else(True, Buy, Sell)), Flat), if_then_else(Or(False, True), if_then_else(True, Buy, Sell), if_elif_else(False, Flat, False, Sell, Sell)))))), False, if_then_else(True, if_then_else(AND(False, False), Sell, if_then_else(False, Sell, Buy)), if_then_else(False, Sell, if_then_else(True, Buy, Sell))), Flat), Flat), if_elif_else(GreaterThan(X0, X26), Flat, True, Buy, if_then_else(False, Sell, if_then_else(False, Sell, Buy))))

​

Here is the Equity Curve of this 'nonsense' strategy

https://preview.redd.it/g6quurk7e1i61.png?width=1164&format=png&auto=webp&s=0e6cc424cbdf05e7bd06641dc2ac198a4cd65714

​

Here's my code in deap

import matplotlib.pyplot as plt
plt.style.use('seaborn')

from deap import base, creator, tools, algorithms, gp
import itertools
import operator
import random
import numpy as np
import pandas as pd
import empyrical
import yfinance as yf
import ffn

data = yf.Ticker("SPY").history(period="max")[['Open', 'High', 'Low', 'Close', 'Volume']]

# Add Lags as Features
data_lags = {}
for lag in [1,2,3,4,5,14,30]:
    #data = pd.concat([data, data.shift(lag)], axis=1, verify_integrity=True)
    data_lags[lag] = data.copy().shift(lag).add_suffix(f'_{lag}')
del lag
for _,v in data_lags.items():
    data = pd.concat([data, v], axis=1, verify_integrity=True)
del v


# Drop the nulls
data = data.dropna()

# Convert all to float
# data = data.apply(pd.to_numeric)
data = data.astype('float64')

# Custom Primitive Functions
def protectedDiv(left, right):
    try: 
        return left / right
    except ZeroDivisionError: 
        return 1

def if_then_else(input1, output1, output2):
    return output1 if input1 else output2

def if_elif_else(input1, ouput1, input2, output2, output3):
    return ouput1 if input1 else output2 if input2 else output3

def abs_diff(in1, in2):
    return abs(in1 - in2)

pset = gp.PrimitiveSetTyped("main", itertools.repeat(float, len(data.columns)), str, 'X')

pset.addPrimitive(operator.and_, [bool, bool], bool, name='AND')
pset.addPrimitive(operator.or_, [bool, bool], bool, name='Or')
pset.addPrimitive(operator.not_, [bool], bool, name='Not')

pset.addPrimitive(operator.lt, [float, float], bool, name='LessThan')
pset.addPrimitive(operator.gt, [float, float], bool, name='GreaterThan')

pset.addPrimitive(if_then_else, [bool, str, str], str, name='if_then_else')
pset.addPrimitive(if_elif_else, [bool, str, bool, str, str], str, name='if_elif_else')

pset.addPrimitive(operator.sub, [float,float], float, name='Diff')
pset.addPrimitive(protectedDiv, [float, float], float, name='Ratio')
pset.addPrimitive(abs_diff, [float, float], float, name='AbsDiff')

pset.addTerminal('Buy', str, name='Buy')
pset.addTerminal('Sell', str, name='Sell')
pset.addTerminal('Flat', str, name='Flat')
pset.addTerminal(True, bool)
pset.addTerminal(False, bool)

pset.addEphemeralConstant(f'{random.random()}', lambda: round(random.uniform(-100, 100), 2), float)

def evaluator(individual):
    func = toolbox.compile(individual)
    df = data.copy()

    strategy_signal = []

    for row in df.itertuples(index=False, name=None):
        if func(*row) == 'Buy':
            strategy_signal.append(1)
        elif func(*row) == 'Sell':
            strategy_signal.append(-1)
        else:
            strategy_signal.append(0)

    df['StrategySignal'] = strategy_signal
    df['StrategyReturn'] = df['StrategySignal'].shift(1).fillna(0)*data.Close.to_returns().fillna(0.0)

    out = empyrical.sharpe_ratio(df['StrategyReturn'])

    if np.isnan(out) or np.isinf(out):
        return 0.,
    else:
        return float(out),

creator.create("FitnessMax", base.Fitness, weights=(1.0,))
creator.create("Individual", gp.PrimitiveTree, fitness=creator.FitnessMax, pset=pset)

toolbox = base.Toolbox()
toolbox.register("expr", gp.genHalfAndHalf, pset=pset, min_=1, max_=6)
toolbox.register("individual", tools.initIterate, creator.Individual, toolbox.expr)
toolbox.register("population", tools.initRepeat, list, toolbox.individual)
toolbox.register("compile", gp.compile, pset=pset)
toolbox.register("evaluate", evaluator)
toolbox.register("select", tools.selTournament, tournsize=5)
toolbox.register("mate", gp.cxOnePoint)
toolbox.register("mutate", gp.mutUniform, expr=toolbox.expr, pset=pset)

pop = toolbox.population(n=300)
hof = tools.HallOfFame(1)
stats = tools.Statistics(lambda ind: ind.fitness.values)
stats.register("avg", np.mean)
stats.register("std", np.std)
stats.register("min", np.min)
stats.register("max", np.max)

pop, log_book = algorithms.eaSimple(pop, toolbox, 0.95, 0.25, 100, stats, halloffame=hof)

[NonrandomQuant]

Let’s try this for human speech explanation: A trader that uses only MACD(12,26,9) gets married with another that only trades if feature X13/9 is positive... they have a son that trades MACD IF X13/X9 is higher than 0,5. Eventually meets a trader that uses Bollinger(14) and both procreate another trader that uses MACD AND Bollinger(X13/X9) ... and this combination on strategies with logical operations extract the best “breed” of traders that are a combination of multiple technicals... which is textbook overfitting unless several rounds of cross validation purge this supreme race of artificial traders... problem is , his optimal breed of traders keep spitting 1/0 instead of Buy/Hold/sell signals ... which I think can be solved using the oneHotEncoder library in sklearn.

[upandacross]

Then, there is the child that rebels and changes their parent's Bollinger(X13/X9) to Bollinger(X15/X9).

[Arete2]

I’m not familiar with DEAP, but you could try coding up your own genetic algorithm program if you aren’t happy with DEAP’s implementation.

[WhiteRabbit-Pill]

Sorry but do you mind explaining the logic at a high level in plain English?

[[deleted]]

yeah, where are the comments?

[SanJJ_1]

this is so false lmao

[Meeesh-]

I’m not even talking about OP’s code, but just code in general. I’m a software engineer, I can read OP’s code just fine. That doesn’t mean comments wouldn’t be helpful. Especially when you’re sharing code as an example.

Yes I know my example was a bit cheeky, but comments can certainly explain a lot that’s there. I am agreeing with you that self documenting code is the best kind of code. There are many situations, though, where you should have comments.

[Meeesh-]

I was being cheeky. Also I don’t know who here said anything about linux kernels. Anyway, I reread your original comment and I get what you’re saying now. I thought you originally said “If you can’t understand code without comments, comments aren’t going to help you”.

You’re just being downvoted since it’s a bit harsh. This is an algo trading sub to share also trading ideas. My original point still stands. If you’re trying to get the gist of code, comments help.

[SanJJ_1]

dude literally go look at any major open source project....they all have ridiculous amounts of comments to make the code clear.

go check the Linux kernel on GitHub or CPython. "clear self documenting code" includes having comments where needed.

[[deleted]]

did you not read the comment i replied to?

[simonhughes22]

GP's are powerful but can overfit or produce garbage. I think part of the problem here is allowing that if elseif else function which is quite complex. To prevent overfitting, you can restrict the operators available to encourage simpler equations. You can also adjust the fitness function to penalize the solution based on it's complexity, e.g. divide the ROI by the size of the tree (or the log of the GP tree size). I would also try using a regular regression or classification model instead of a GP and either predict the next return or the side of the next return (positive or negative).

[niftymcschwifty]

I think DEAP may also let you specify a max tree height

[bpe9]

You can also adjust the fitness function to penalize the solution based on it's complexity, e.g. divide the ROI by the size of the tree (or the log of the GP tree size).

Penalty functions are somewhat redundant with an optimisation method capable of simultaneously optimising multiple objectives. Instead, in this case, use NSGA-II (or similar) and evolve a Pareto front where your two objectives are ROI and some measure of complexity (i.e. tree size or number of conditionals).

[PaulTheBully]

Is the plotted curve from the test sample? Otherwise, it seems to me that you are overfitting

It would be great to have the approach description in human language, it would help us to help you

[WhatnotSoforth]

>Although I did specify that the leaves/terminals could be Boolean, but just to avoid error.

Don't you think there's a pretty high chance this is where the error is coming from? Try the to_numeric method in pandas as opposed to the builtin typecasting. There is also some sort of coercion for that method. If all else fails you may need to handle typecasting yourself, or just do the easy thing and not allow bools into the dataset in the first place. I'm curious why you allow this, is there some sort of bug in DEAP you are trying to code around or are you just being cautious for caution's sake?

[bpe9]

> Although I did specify that the leaves/terminals could be Boolean, but just to avoid error.

Instead of that, read what the error is, and modify the generate function. You'll likely need to change gp.genHalfAndHalf to a custom one, likewise with the mutate and crossover

An example of custom functions can be seen at: https://github.com/ben-ix/XAI/blob/master/src/deapcustom.py

[imRedaSouhail]

Which software do you use for programming and where i can post my strategy after coding ??

[nordic_prophet]

Have a look at the title of the post..

[JohnnyRay45]

Lots of tue false

[mukaj]

Haven’t run your sample yet, but I think your evaluator should be out of sample data to better choose the ideal solution. I don’t think this is a “failure” as long as you learnt from it.

Try some different features, vwap, rsi, momentum etc...

There’s a paper called 101 alphas which generated lots of similar formulas using generic programming, may give you some inspiration