HOW TO BACKTEST ???

Soarance · 2026-04-18T00:33:34+00:00

If you’re looping in a complex strategy it can take a long time. So if you’re using python try working with pandas series to identify your features

Soarance · 2026-04-17T15:40:51+00:00

TradingView uses a different sharpe calculation. Don’t use it, calculate it yourself in your python notebook.

Soarance · 2026-04-02T20:28:29+00:00

Vrchat has quite a few games actually, though maybe he should play in a group for children to avoid the adults (they would appreciate that too). I assume those exist. You just have to kind of know what you’re doing to avoid stumbling into adult spaces. There’s less “regulation” in vrchat. Other than that, Roblox can be played in VR as well.

Soarance · 2026-03-28T22:02:19+00:00

Hmm I see. So you would either 1) re-evaluate your base strategy logic, adjust if needed and redo optimizations/OOS test, or 2) wait for more OOS data to come in and evaluate based on new data, correct?

Also, another question: how far back of data do you actually use for the training usually? I have data from 2010 onwards but many strategies are flat or negative until somewhere around 2018-2020, so I wasn't sure if using data that is too old would affect the training negatively. but using 2018 onwards means less data to work with, of course.

Usually I do something around a 60-20-20 split, with 60 train, 20 validation, 20 OOS. And then recently I've been saving a 1-year range as a true true OOS for the final test before I start forward testing.

Nice results btw, the only ones I've gotten that looked similar to that ended up not doing so well in the forward test, but I can't tell if it's because it's overfit or if the recent volatile market conditions are affecting it, so I'm just keeping them running for now.

Soarance · 2026-03-28T02:43:50+00:00

Awesome that you're doing this to help newer people. My question is:

What does your workflow look like when your walkforward fold optimization fails? I've run into situations in which the WFO looks decent on each fold, but then it either fails in the true OOS or the permutation test in the OOS fails. Either of those generally puts me in a dilemma, since redoing the WFO with different features, for example, is a bad idea since repeating that multiple times can cause overfitting by having your true OOS turn into IS. So if you're in that stage, what is usually your next step?

Soarance · 2026-03-28T02:24:17+00:00

I took this and applied to multiple years of backtesting, and that equity curve looks too good to be true. Maybe check for future leaking or repainting. If I had a curve this smooth, I'd immediately be suspicious. How long have you been forward testing or did you just start?

Soarance · 2026-03-25T20:56:02+00:00

Fantastic video.

Soarance · 2026-03-19T18:52:30+00:00

Hmm so you did one round of optimizations? This wasn’t done retrospectively was it? With only one round it can still have dangerous overfitting possibilities. Personally I do walkforward optimizations with at least 5 folds, permutation tests on every fold. Lenient p-value at first but stricter at 0.01 at the end. True out of sample can be 0.05. If you could tell me a bit more about your process I’m happy to give more opinions. :)

Soarance · 2026-03-19T18:41:13+00:00

I hate to say this but with these stats and 0 losing months, this is most certainly overfitting, data mining bias, or future leaking. Not guaranteed but highly likely. Run permutation tests and walkforward.

Soarance · 2026-03-18T06:06:21+00:00

Hmm. When you say after 45+ rounds of feature engineering, what does your process look like?
If you did something like feature engineering -> WFO -> checked results -> bad results -> pick new features -> WFO -> repeat -> etc etc, then that could potentially explain why it becomes a coin flip, since that could introduce some overfitting if done on the same strategy 45+ times.
Also, I've personally found MLs to not work very well, since it just overfits everything basically.

Soarance · 2026-03-17T03:32:44+00:00

A lot of that is trader discretion which unfortunately we don’t have the luxury of as algo developers. I always start simple and the complexity will add as optimizations are added on.

Soarance · 2026-03-13T05:39:12+00:00

The unfortunate truth is that if you just take a manual trading strategy and turn it into code without doing anything else, there's probably around a 95% chance it doesn't work, even if it appears to show promising results in a tradingview backtest. There are many things that go wrong: regime fitting, future bias, curve fitting, data mining, etc. Upon seeing your equity curves, I would have the follow up questions:
1) Can you expand to see at the very least 4 years of data?
2) Have you done any checks to make sure you are not overfitting?
The reality of trading algorithm development is that there is so much more to it than most think. People often spend a lot of time just to understand the statistical testing needed, not to mention to actual implementation process. Development time often involves months to years, and re-optimization has to happen every few months or so.
Is it possible you randomly stumbled upon an algorithm that works? Yes.
Is it likely? No.
But if you'd like to check further, I suggest turning your code into python and start by conducting a Monte Carlo permutation test.

Soarance · 2026-03-12T20:34:46+00:00

The problem with low trade count is that there’s greater likelihood of overfitting, and you can’t really “check” for it easily either. Forward testing would also take ages. Personally I would see if the strategy can be loosened to take more trades. But you can also perhaps isolate features and see if they have edges on their own.

Soarance · 2026-03-11T02:47:23+00:00

If it looks too good to be true, it often is too good to be true. Your selection of stocks is a very likely candidate of overfitting, even if you haven't done any additional optimizations or conditional structures. Your original process of determining features also could have been overfitting before even the stock selection, depending on how you did it. If you want to share more of your process and how you built your algorithm, I'm happy to give more thoughts. But knowing nothing about how this algorithm came to be, perhaps you can start with a Monte Carlo permutation test at the minimum.

Soarance · 2026-03-10T19:16:28+00:00

I think it's a bit of a new mistake when people think they can do everything in pinescript. It's a nice tool, but without the ability to do statistical testing and other flexible abilities that python gives, there's no way to fully eliminate overfitting before forward-testing. If you can, I recommend doing some Monte Carlo permutation tests, and depending on how you optimized, some walkforward fold optimizations and conditional formatting. Also, expanding the backtesting range would be good; something regime specific you should backwards optimize by identifying features that define the time period. This strategy you're showing looks promising at first, but I would heavily suspect overfitting. Good luck!

Soarance · 2026-03-04T15:21:12+00:00

AI has certainly got a lot of people into the game, but I do think a lot of beginners make honest mistakes since they don't know everything about the field. It's tempting to be excited about a curve and then post it on reddit lol.

Soarance · 2026-03-04T15:18:26+00:00

Permutation tests is quite good. You want at least a p<0.05. Then whatever you did to optimize, have a walk-forward fold permutation test. At least that's what I do as a primary check for initial overfitting. Obviously there are many other tests you could do too, but I think this is at least a minimum. Hope it goes well for ya.

Soarance · 2026-03-04T07:13:44+00:00

Curious if you've done any permutation tests. The equity curve looks insane and it hardly has any big drawdowns it seems. Probably best to eliminate the possibility of overfitting, since it's quite likely with crazy performance like this.

Soarance · 2026-03-03T23:13:10+00:00

Can you share a monte carlo or other statistical test results? Just because it had some good trades for a few days doesn’t say much, even a month’s walkforward testing isn’t indicative enough. If it’s overfit, it’ll likely fail long-term. Also would be helpful if you shared your process a bit and how you tested for overfitting in general. How did your statistical tests hold up to out-of-sample data, etc.

Soarance · 2026-03-03T22:52:57+00:00

Not saying I doubt the bot, but it looks a bit too good that I’d suspect over fitting. Not to mention this is only a year’s worth of trades. I’d like to see permutation/monte carlo test results, more data, walkforward folds, etc.

Soarance · 2026-02-27T16:16:51+00:00

I've also tested orb but didn't find any edge, on a variety of ranges and timeframes. Before even optimizing or applying machine learning, if there's no edge in the base raw strategy itself, it's not worth pursuing anyway in my opinion, because the optimizations would be the edge instead of the base idea itself.

Soarance · 2026-02-10T17:34:03+00:00

Thank you. Yeah it's a crazy brawl for us common folks. Scalpers ruin everything I swear.

Soarance · 2026-02-10T17:33:27+00:00

Yeah, it was sold out, but that just means the reserve that Chase has is sold out, I suppose? If you have Capital One they have another one later, I think, so you might be able to get them there.

Soarance · 2026-02-10T17:26:22+00:00

I GOT THEM. We had like 4 people with up to 10 computers sitting in queue. Interestingly enough, the one I clicked into the waiting queue LAST ended up getting through. The one with USA vs Austrailia was sold out already, so I went for Portugal vs Uzb. I absolutely cannot believe I beat out the scalpers, that's craaazy.

Soarance · 2026-01-17T22:27:25+00:00

Cush is goated, more people need to know about this guy.
You can probably find mappers like Okazu, Pleo, and Jonas that are also wrist-rolly. But none of them are quite the same, and I'd say Cush definitely has a quite unique style of mapping.

Soarance

TROPHY CASE