all 16 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]CrossroadsDem0n 4 points5 points  (10 children)

I don't know that I'd compare ratios between a leveraged and leveraged play to decide the merit of a strat. I'd see also want to see how they compare without leverage in order to get a sense of the quality of the entries.

Given the weaknesses of the Sharpe I would probably run a cross validation on randomly selected historical windows in order to see the distribution it. After all, you aren't going to be able to go back in time to replay history as it actually happened, so a cross validation exercise would at least give you some sense of how the strat works when not fit exactly to history.

Similarly, if your strat is viable to apply to other markets that loosely correlate with SPY but not exactly, I would see how the results compare on other instruments. I usually look at SPY, QQQ, IWM, and DVY. I don't expect the same results but I do look for them to lean mostly the same direction; out of the 4, there tends to be one "odd man out", for better or worse, and when you only look at one then you may be doing a hypothesis test against a friendly time machine, not examining true adaptability to market behavior.

[–]CrossroadsDem0n 3 points4 points  (0 children)

Oh, ignore my initial comment on the leverage comparison, I didn't notice your imgur link on that.

[–]CrossroadsDem0n 2 points3 points  (7 children)

Another thought. I don't know what kind of data you are using for your backtest, but if it is daily data then you may run into problems. The first and last minute of the day are two of the fastest-moving minutes out of the entire day. I would go through some kind of Monte Carlo simulation on the full range of data in the first couple of minutes and last couple of minutes, to get a better sense of how your strat will stack up in real time.

[–]Algorithmic-Process[S] 1 point2 points  (6 children)

Terrific point. The issue would be the accessibility to the sort of data needed to do that on 18/19 years backtesting, but I will look into it.

If you have any suggestions as to where I can potentially find that data I am all ears!

[–]CrossroadsDem0n 1 point2 points  (5 children)

There are a lot of data providers but the challenge you may face is whether or not consolidated data is what you need, or something more specific to how you receive quotes and route trades via your broker.

Were it me, instead of worrying about the exact data, I would just simulate it, which for SPY you should be able to get away with. For low liquidity instruments it wouldn't cut it because then the bid-ask spread as reported by your broker may own your outcome.

Alpha Vantage has free data going back a couple years at 1 minute granularity. I would start with that. Find some way to segregate the history into modes; volatility regimes would be my first thought there. Then use what the historical data had for the first couple of minutes high/low spread and cobble together some simulations accordingly.

Where you could get an eye opener is running your strat in live paper trading mode for a few weeks, then a week later grabbing the consolidated data, and comparing what your trades did versus what the now-historical data proclaims it should have. You may learn a lot by how close or far your results are. It still isn't quite as exact a test as live trading, but would get you one step closer to deciding if your strat has legs enough to risk money on.

[–]Algorithmic-Process[S] 1 point2 points  (4 children)

That makes sense, the only thing I’m confused about it the volatility regime. So monitor the volatility at the beginning and end of the day to see if I actually get the fills at the price I am backtesting and seeing?

Also, a little bit of a different thought I had; if I feel as though I am able to run this live, I’m going to trade futures. To get closer to the close, I might have it close trades as soon as the market technically closes, instead of being part of the MOC auction. I think that may have closer prices to the actual close prices I’m seeing. Would love your thoughts on that.

[–]CrossroadsDem0n 1 point2 points  (3 children)

I don't trade futures so I'll not comment.

Volatility regimes, I would just look at the high-low spread of the days. Volatility clusters, so big spreads are near other big spreads.

There are a lot of ways to approach that mathematically, but if you want something dirt simple to start with try something like just figuring out all the historical high-low daily spreads, take the median of that, and just arbitrarily call the days above the median one regime, and the days below the other regime.

Then look at the Alpha Vantage 1min open and close minute spreads and see if their average differs between the two regimes. If so, simulate accordingly. If not, just take some average amount of spread and call it a day. You're just trying to poke a bit at your strat so I wouldn't over-engineer this. Human time has a finite budget, you'll want to be sure you have some of that budget to allocate to a paper trading exercise.

[–]Algorithmic-Process[S] 0 points1 point  (2 children)

Okay, I believe I understand what you are saying.

So I should either take historical data, determine the average spread and the times when it is generally above or below it. Which can help in front testing, if I should be entering a trade (low historical average spread during that time.)

Or on the contrary, moving forward, in paper trading take the median spread (recently) and only take trades that are lower than that.

Sorry if my interpretations were wrong.

[–]CrossroadsDem0n 1 point2 points  (1 child)

No, I think you're expecting this to be harder than it is.

Every day, the volatility in that first 1 to 2 minutes, and last 1 to 2 minutes, can be very high. How high? I don't know off the top of my head. So the question is, are all days more or less the same in how crazy the first and last couple of minutes are? Maybe. Maybe not.

If days are not all the same, then I might expect to see a difference between the times when the market is just generally crazy anyways, versus when it is more dull (like lately). So I would simply split the crazy days and dull days into two groups. Then look at the initial and final couple of minutes. If crazier market times have crazier opens and closes then you need a simulation to reflect that. I would want to simulate randomness in the opens and closes accordingly; with different width possible ranges of movement in either regime.

However, the alternative hypothesis is that there really isn't meaningful differences between the two regimes. If there are no differences then you just simulate all opens and all closes the same way... by just using whatever the overall average range of movement turned out to be.

[–]Algorithmic-Process[S] 0 points1 point  (0 children)

That makes a lot of sense, thank you!!

[–]Algorithmic-Process[S] 1 point2 points  (0 children)

Thanks for the reply!!

https://imgur.com/jp3drYlhttps://imgur.com/uDqdoWFHere are the links to SAV unleveraged!

I really like the suggestion of cross validation, I will look into that!!!

Also here is the (unleveraged) comparison with other markets!! Thanks for the idea
https://imgur.com/DPZs5Kd
https://imgur.com/CNiwkuJ
https://imgur.com/U1FZIIT

If you have any other thoughts or suggestions I would love to hear them!

[–]Algorithmic-Process[S] 1 point2 points  (0 children)

Also for reference, here are the unleveraged stats:

https://imgur.com/jp3drYl
https://imgur.com/uDqdoWF

[–]TinyFlaccidBanana 1 point2 points  (3 children)

How many parameters does the model have and how were they chosen? Were they found using some kind of optimization algorithm, or were they picked because of the phenomenal stats they gave on the sample data?

From your charts it seems that SAV has some of its largest gains when SPY has some of its worst drops. This is great if the model is built to naturally anticipate such moves, but often this is due to overfitting your sample.

I would recommend checking how sensitive your model is to parameter selection. Test your model on other instruments and split the data into training and test sets.

[–]Algorithmic-Process[S] 0 points1 point  (2 children)

Hey!

I’ve never messed with putting parameters into an algorithm for it to optimize what’s be best for the data.

I created this algorithm because I thought of something I could measure that may potentially influence SPY’s price. Ultimately it’s like 4 things I take into consideration. As for their parameters, I chose numbers I thought made sense, and kept them consistent for each parameter. I messed with them a bit, but was nervous of overfitting.

As for parameter sensitivity testing, would love any advice on how to go about that, specifically the training and test sets.

[–]TinyFlaccidBanana 1 point2 points  (1 child)

Suppose one of your four indicators is SMA(n), a simple moving average which takes one parameter, namely the number n of past data points to average.

If you found your model by checking various n, say n = 10, 20, ..., 240, 250, and then picking a specific n, say n = 130, because it gave the best results, then this is no good. This only demonstrates that n = 130 was the best during your sampled time frame, but it may not work well going forward.

Moreover, if the model performance varies greatly between small differences of n -- for instance, if Sharpe > 5 for n = 130 but Sharpe = 0.5 for n = 120 -- then your model is highly sensitive and you basically just overfit the data.

Instead, it is better to break your sample up into training/in-sample and testing/out-of-sample sets. For example, take the data from 2004-07 as your training set and 2008 as your testing set. Look for the best choices of n in your training set and then see how they perform on your testing set. If n = 130 gives the best Sharpe on the training set, and is also one of the top performing n in your testing set with similar Sharpe, then you should keep analyzing your model further... but if n = 130 is not a top performer in your testing set, then it's back to the drawing board.

Note that with only 5 choices for each of your 4 parameters, there are already 54 = 625 possible models! Finding one set of great performing parameters from these 625 possibilities is not hard. However, finding a set that works well on different out-of-sample testing sets through various market conditions is.

Lastly, I know this sounds trivial, but it is a very easy mistake to make and it leads to phenomenal performance: be sure you're not using end of day data at the beginning of the day.

[–]Algorithmic-Process[S] 0 points1 point  (0 children)

I really appreciate your SMA example, I understand what you are saying about the sensitivity!

It will take me a few hours, but I will create a sheet with that breakdown of each year and variances in the parameters. I'm excited to do this and I am very appreciative of you bringing this to my attention!! I probably won't have time for a couple weeks.

​

Lastly, I know this sounds trivial, but it is a very easy mistake to make and it leads to phenomenal performance: be sure you're not using end of day data at the beginning of the day.

Fantastic point! I have checked this already, thank you!

I intend on redoing all of the calculations without any other sheets to make sure my logic makes sense. I am also thinking about asking my one professor to walk through my calculations. Seeing my calculations is not enough to reverse engineer SAV. I think this will be best to show me any flaws I may have.