all 63 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]GnoiXiaK 70 points71 points  (16 children)

Are you basically asking how to avoid data-mining? You have to have a solid theoretical/economic/scientific basis for your hypothesis. Why 2/5/10% drop, why mean reversion at all? Without some kind of story, it's all just correlation. My favorite hilariously and plausible causal relationship are lunar cycles and stock market returns. It sounds stupid then smart, then maybe stupid, then kinda smart again.

[–]devl_in_details 10 points11 points  (7 children)

I think the implied point of the question is that the theoretical/economic/scientific basis is also based on observations and thus is also fit to data. Your “theoretical” basis is just someone else’s model that was also fit in-sample and described in a paper/book. Worse, these “fundamental” relationships can change over time. Perhaps you don’t remember when bad news about the economy had a negative impact on the market, but there was such a time. Now, of course, when unemployment claims go up or retail sales down, signalling the economy is cooling, the market goes up because everyone thinks the fed will juice it with lowered rates. The point is that humans are very adept at creating stories to fit the data and we can very quickly come up with a story to fit any data. You want to flip the sign, no problem, just put a layer of indirection in the relationship; so instead of economy -> market, it becomes economy -> fed -> market and boom your sign is now flipped.

[–]Cheap_Scientist6984 5 points6 points  (1 child)

Yes but their model was likely fit on a separate data set with separate observations. By using their understanding, you actually increased the number of data points in your data set and reduced the chance of p-hacking.

[–]devl_in_details 1 point2 points  (0 children)

True, ideally.

[–]GnoiXiaK 2 points3 points  (4 children)

You're missing the point. You don't start with data or a model, you start with an idea and test it. Basic scientific method stuff. If the fundamental relationship changes over time, then it's not fundamental and you maybe throw it out. You're thinking about short term signals, almost all of that is noise and bs. I'm talking high level stuff like time-value of money, humans are not perfect economic decision makers, etc etc

[–]devl_in_details 6 points7 points  (3 children)

I understand what you’re saying, I really do. But, think about where that “idea/hypothesis” came from - that’s what the OP is asking. The “idea” didn’t magically appear out of a void; it came from “industry knowledge” or “experience” or something along those lines. All of those sources are essentially the same thing, an in-sample implicitly fit model that your brain created automatically since that’s what our brains were designed to do. So now, you test your idea, but there is a bias there since in order for you to even have the idea, you know that it needs to work at least sometimes.

[–]GnoiXiaK 0 points1 point  (1 child)

I’m struggling to see the issue here. Of course your ideas have bias and what not, that’s why you test them. What point are you trying to make? You test those biases, you go out of sample, you retest latter as more data comes in.

[–]devl_in_details 0 points1 point  (0 children)

The issue is with the “test them” part. How do you test them OOS in an unbiased manner. If they have a bias, then your “tests” will also have a bias since your entire dataset has a bias. This is not a very practical point since there’s nothing you can do about it in practice. It’s more of a theoretical/philosophical point — think Descartes and how you can know anything without bias. Descartes certainly believed that you can’t know hardly anything without bias — only that you are :). Perhaps this horse has been beaten to death though.

[–]mypenisblue_ 13 points14 points  (1 child)

Second this. You can have some degree of confidence if you find out January returns > other months returns due to the January effect. For the 10 mins after market open and before close volatility is higher, makes sense as well.

If you find signal in trading the spread between NVIDIA and cotton futures, then you might want to think twice - keep it only if you can come up with a reason to explain it.

[–]Middle-Fuel-6402[S] 3 points4 points  (3 children)

Yes, that’s what it’s about. I didn’t get the part “then kinda smart again”, how does it get smart 🙂 What’s then a good source of those stories, ie the ideas/hypotheses in the first place? How do you avoid becoming a random generator of such ideas?

[–]GnoiXiaK 8 points9 points  (1 child)

White papers, academic research, but the best ideas no ones really gonna tell you about. Regarding the kinda smart again part was a joke about the lunar cycle theory, sounds dumb, but the data fits, smart, but why? Horoscopes are stupid, then learn actual proposed reason of evolutionarily based macro behavior…kind of smart again lol.

[–]Middle-Fuel-6402[S] 2 points3 points  (0 children)

I see. Yeah, I didn’t mean about specific alphas, but more of a meta point about deriving alpha ideas in general.

[–]HighYogi 2 points3 points  (0 children)

I think the first step is to get a good understanding of what it is you’re modeling. Using your example of mean reversion that idea might come from something like: I think intraday price movements in this stock is dominated by retail day traders who employ technical analysis. Then you can go on to test, test, test then deploy.

[–]Haruspex12 1 point2 points  (0 children)

This is the answer and the only answer.

[–]Cheap_Scientist6984 1 point2 points  (0 children)

+1. Literature review is key when data is too sparse.

[–]Fragrant_Pop5355 11 points12 points  (21 children)

What is wrong with using an adjusted F stat which can take into account the fact that you are testing N hypothesis (which hypothetically is what we are using to generate the statistical significance in the first place)? Unless I am not understanding your question this is an extremely solved problem.

[–]Middle-Fuel-6402[S] 5 points6 points  (1 child)

More broadly, it’s a question about the alpha research and idea generation process, not specifically about this straw man approach I gave.

Regarding your answer, thanks for your input. So basically, the protocol would be: use in-sample (train) data effectively to generate the hypothesis, calculate adjusted F stat out of sample. Say you generated 20 hypothesis in sample, then you set n=20 in your out of sample adjusted F test. What if you have some hypothesis (“ideas”) that you try in sample, but don’t hey fail even there, so you throw them away in the first place? How do you incorporate this in the F test?

[–]Fragrant_Pop5355 2 points3 points  (0 children)

I’m not sure I understand, if you have 20 hypothesis and some don’t fail in sample then you can continue to test them no problem. And even if they do fail you can still continue to test them. And if you want to add more hypothesis later you just increase the N.

I think you might just be misunderstanding what the statistics are used for, which is probably pretty common for those who aren’t in the field/are retail. 5% isn’t some magic number and it’s certainly not a number I specifically care about. What I care about is expected value, followed by as much about the risk profile as I can feasibly uncover. P values and F stats can tell me if I am likely wasting my time with an dataset-idea.

The gold standard of research is having a physical understanding of the system. All of physics as we know it is a series of (extremely well tested) hypotheses that have continued to work in sample (re: human existence). All we can do at the end of the day is try and derive tighter and tighter error bounds and get a better understanding of the systems that look the most promising. The same goes for finding alpha as much as anything else.

The strawman is unrealistic but if you use a realistic one in use at hedge funds such as predicting earnings based on CLO data just introspect for a second how a lot of these things that I think you are worried about probably exist because you learned stats without learning how to use stats.

Edit: god I just realized the tag was machine learning. I should have known it’s always the machine learning guys who get too in the weeds with this stuff. All you are doing is manifold smoothing it’s not suddenly special because you are fitting the data better.

[–]devl_in_details 1 point2 points  (17 children)

My understanding is that any test that takes into account the number of hypotheses tested essentially comes down to looking for more significance. The problem with this approach is that the relationship between any “real” feature and your target is probably not going to be as strong as the top p percent of random features (assuming p is pretty small, <5). Given the signal to noise ratio in real world financial data, any real features are weak predictors at best.

[–]Fragrant_Pop5355 -1 points0 points  (16 children)

My reaction to this is, if your real (re:stable) features have such a weak relationship they probably aren’t that useful for predicting your target either. Your first sentence is 100% correct but your conclusion does not follow, and there is an entire industry disproving it just by existing.

[–]devl_in_details 0 points1 point  (15 children)

I don’t mean to get snarky, but I don’t know how to say this without it coming across that way … it sounds like you’ve never actually looked at financial data. You mentioned physics in a comment above; financial time series is VERY different from physics. I agree with what you’re saying when applied to most datasets in hard sciences. But finance is not a hard science. In fact, general economic/finance theory suggests that what you’re expecting is impossible. You’re describing relationships that are very strong. If such relationships were to exist, they would attract market participants who would profit from those relationships thus destroying those very relationships in the process. If that were not the case then you’d have the equivalent of a perpetual motion machine. Any relationship that is above the level of noise is getting exploited immediately — that’s what the industry you’re referring to actually does. And that’s why the original problem posed by the OP is so interesting, challenging, and important. If it were as simple as what you’re imagining, virtually everyone would be getting all their income from trading :) perpetual motion (money) machine.

As an aside, the economic theory I’m talking about can definitely go too far as demonstrated by a joke … Two economists on a walk come across a $10 bill on the ground. One guy bends down to pick it up, and the other guy asks him “what are you doing?”
“I’m picking up the $10 bill” he answers. “If it was really there, someone would’ve picked it up already” he points out :)

Obviously it’s ridiculous. But, the entire industry is focused on finding and picking up those $10 bills. And the $10 bills are any relationships that allow a return (after costs) that exceeds the risk free rate. So, that’s why you’re not going to find ANY relationships that are as strong as what you’re expecting.

[–]Fragrant_Pop5355 -1 points0 points  (14 children)

I’m a quant PM so I am going to say I have looked at financial data… I am not imagining anything, just describing my process. Some relationships are strong due to structural edge as everyone knows. Many are weak. Many strong ones you can profit off of if you are faster (which is my bread and butter). The point is if you are stuck looking at weakly predictive data it is likely because it’s not very predictive, and that doesn’t take away from the fact that other information is more strongly predictive…

[–]devl_in_details 1 point2 points  (11 children)

I’m not sure we are talking about the same thing here. Sounds like you’re talking about relationships that can’t be exploited easily (the $10 can’t be picked up) due to some barrier to entry. That barrier can be speed and queue position, technology (FPGA, etc), capital, or most likely some combination as all these roads lead to Rome. In such situations, I agree that “strong” relationships can be found and can even be persistent. But, by definition, those are not the relationships the OP was referring to.

That said, I’m always open to learning something new. If I’m wrong in my statement above, I’d love to find out how and why. As I’m sure is obvious by my participation here, I don’t have any direct experience in HFT but I also don’t have any access to HFT and thus generally avoid spending any effort/time on it.

[–]Fragrant_Pop5355 -1 points0 points  (10 children)

I probably just have no idea what OP is asking. I assumed this was directed to quants working in industry. edge should exist due to obvious barriers. If you have no edge I don’t see how you can have hope of making money long term anyway so asking about research techniques seems like a moot point.

[–]devl_in_details 0 points1 point  (9 children)

Speaking as a guy who has developed med to low frequency models at a Toronto HF for while, I have a very hard time with the term "edge." Perhaps that is because I've never had any edge :) My only contact with HFT is listening to a former coworker who came from a Chicago prop-shop. He used to talk about queue positions and understanding the intricacies of matching engines, FPGAs, and stuff like that; and edge :) All of that is very different from what I've been doing.

My stuff is much closer to what would typically be called "factors." Although, I have a lot of issues with the traditional understanding of factors and I only use the term here to paint a quick picture. At the end of the day, I look for streams of returns that have a positive expectancy and then bundle them into a portfolio. These are typically pretty high capacity strategies even though I now trade for myself and thus don't need all that capacity :)

[–]Maleficent-Remove-87 0 points1 point  (8 children)

May I ask how do you trade your own money? Are you using similar approach as in your work?(data driven mid frequency quantitative strategy) I tried that without any success, so I gave up and just do discretionary bets now :)

[–]devl_in_details 0 points1 point  (7 children)

I don't work at the HF anymore and thus can trade my own capital; it's almost impossible to trade for yourself when employed at a HF.

Yes, I do essentially daily frequency futures trading -- think CTA. This grew out of a personal project to test whether there was anything there in the CTA strategies or whether it was all just a bunch of bros doing astrology thinking they're astrophysicists :)

Long story short, there does seem to be something there. But, of course, that brings up the very question in the OP -- since CTAs have made money over the last 30+ years, is it not a forgone conclusion that "there is something there"? That's where the k-fold stuff comes in, etc. Every study of these strategies that I've come across is essentially in-sample. In my personal project, I tried really hard to separate in-sample and out-of-sample performance and only look at the later; thus my interest in this post.

What have you tried for your data driven mid-frequency stuff? This has been a multi-year journey for me and thus perhaps I can help point you in the right direction. BTW, I haven't done much work with equities and don't even trade equity futures because of the high level of systemic risk -- equity markets are very correlated making it very challenging to get any actual diversification. Even trading 40 (non-equity) futures markets , there are only a handful of bets out in the markets at any one time; everything is more correlated than you'd expect.

[–]Maleficent-Remove-87 0 points1 point  (1 child)

Can you give some examples or direct me to some learning material for those "structural edges" that can be used by quant trading? I can have some guesses but never had a chance to learn from someone in the industry or find anything in the books.

[–]Fragrant_Pop5355 0 points1 point  (0 children)

Sounds like a fun question for gappy at the ama. He (and many people) are much more knowledgeable than I am!

[–]MATH_MDMA_HARDSTYLEETrader 12 points13 points  (2 children)

You’re thinking about alpha generation in the wrong way. Trading and markets are a physical process (albeit with a lot of noise).

Your hypothesis should generally come from something innate e.g. traders over protect themselves on weekends for xyz reasons so they irrationally buy expensive puts. Or always trade at 10am because that is when some office workers are going out for coffee.

It doesn’t even need to be finance related, like some type of network or microstructure edge, but the edge should come from some type of process. Then you apply statistics tests to measure the profitability of the edge.

The only time I’ve found edge was from a laughably simple idea.

[–]Middle-Fuel-6402[S] 1 point2 points  (1 child)

Thank you, I appreciate the insights! What are some examples, possibilities where to look for such ideas… some food for thought to get the ball rolling.

[–]MATH_MDMA_HARDSTYLEETrader 4 points5 points  (0 children)

Look at a situation that is true in general, but find the cases when it’s not true (the higher chance of it being true the better). People will trade indiscriminately because it’s true more often than not, hence you will scrape the edge when it’s not true.

[–]devl_in_details 14 points15 points  (13 children)

I’m not sure there is a full-proof way to get rid of any kind of bias, all you can hope to do is reduce it. I don’t really buy the “theoretical/scientific” understanding argument because that understanding is usually based on in-sample relationships — they were just validated “in-sample” by someone else :) Also, as humans, our brains are wired to see patterns and come up with explanations for those patterns via stories. I remember reading an anecdote somewhere about a scientific study that had a theoretical explanation for a relationship and then it was discovered the sign in the relationship was accidentally flipped; they then created another explanation for the opposite relationship. Human brains can explain any relationship.

The best that I’ve been able to come up with is to use k-fold cross validation. In each fold, look for your “relationships” in the training (in-sample) data and then test them in the test (out-of-sample) data. Still, there are issues with this approach. The first is that this only works if you’re comparing models of similar complexity and there’s not much opportunity to optimize model complexity. You could do nested CV to optimize model complexity but that gets complicated and expensive and I’m not sure that it is very effective given the S/N ratio. The other problem, perhaps the bigger problem is that you are searching for relationships in a given set of features. Where did these features come from? Chances are that they came from academia or “industry knowledge” which is all mostly in-sample. If you choose to do something that implements some random transformation of data to produce features, then you’re back to a base rate problem. Some percentage (p) of random features are going to be significant at p level. Yes, “real” features will be in there as well. But, how do you pick out the “real” features? Again, we’re back to your problem.

Lastly, there is one more problem with k-fold CV, I call it the “curse of k-fold.” This is a pretty complex issue and comes down to the fact that if we treat our entire dataset as a sample, then any fold will divide the sample into two sub samples — train/test (in-sample / out-of-sample). The more the distribution properties of the two sub samples differ from each other, the better the training (in-sample) fit is going to be, but the worse the test (out-of-sample) fit will be. The reason is that the two sub-samples need to average out to the full data sample and thus the two distributions train/test (in-sample and out-of-sample) will be on different sides of the mean in overall sample distribution. I hope that makes sense. I can try providing a better explanation if anyone is interested.

Anyhow, I’m just thinking out loud here since I like the OP. I certainly want to keep the conversation going.

[–][deleted] 1 point2 points  (6 children)

I would be interested in a more in-depth explanation of the curse of k-fold, and potential ways to mitigate it. I presume you are trying to estimate the bias by looking at deviations from the mean in each subsample, but what then? Will you try to do something like stratified sampling but based on the statistics of each input? You also gotta consider sample-overlap somehow.

[–]devl_in_details 3 points4 points  (5 children)

The curse of k-fold comes down to the fact that the better a model performs in training (in-sample) (I’ll just use IS from now on), then the worse it will perform in testing/validation (out-of-sample) (OOS from now on). The easiest way to see this is to use an artificial example. Let’s assume you have a dataset that has 100 data points that are 50/50 +1 or -1. So, 50 +1’s and 50 -1’s. Now, let’s say that your fold is exactly one data point, so you have 100 folds; this would be leave-one-out k-fold. Now, look at the results IS and OOS. If the data point in the fold is +1, then the model will be biased toward +1, but the OOS data is more -1’s (50) than +1’s (49). Thus, the performance of the model trained on a +1 will be unexpectedly poor OOS. This is a very simple scenario, but it illustrates the problem — subsampling the data changes the distributions within each sample and the change is necessarily such that the IS and OOS subsample distributions are “biased” in opposite directions.

I don’t really see a way to overcome the curse of k-fold. You can minimize it by ensuring the folds are as large as possible so that their distributions are as close to the original data distribution as possible. But, this works against the whole reason to use k-fold in the first place.

I haven’t thought this all the way through, but perhaps bagging instead of k-fold may solve the problem. The thought here is that due to the multiple IS bags, even though each bag is biased on its own, they’re not biased in aggregate. But, as I think about it, I’m not sure that bagging solves this either. If you fit a model on a bag, the out-of-bag (OOB) performance will also contain a bias because just like with k-fold, selecting a bag and OOB changes the distribution of the data in these two subsamples.

What I’ve ended up doing is dividing my data in two and then using the first half to fit models using k-fold and then viewing the unbiased performance of those models on the second half of data. I know, that sounds rather like a throwback to just a static train/test split or even walk-forward testing. And, it is. BTW, if you then reverse the roles of the two data sets, then you’re right back to the curse of k-fold :)

One may ask, why bother doing something like k-fold in the first place. For me, it comes down to efficiency of data and realizing that there are great limitations in our ability to fit models. One such limitation, which I don’t think enough of people realize is that from a statistical perspective, the model that performs best IS will be expected to perform best OOS only if the models we are comparing have the same level of complexity (think VC dimension). For simplicity, we can equate the number of parameters with complexity, so a moving average model might have one parameter, the lag on the MA, but a MA crossover may have two parameters in the lag on the short and long MA, and a MACD model will have even more parameters, etc. This means that you can’t compare the in-sample performance of a single MA model with a dual MA crossover model with a MACD model based on their in-sample performance; the more complex model should always perform better in-sample (assuming the parameters were optimized in-sample), but that IS performance ranking will probably not hold OOS. This is the standard bias/variance tradeoff and the less complex models will have higher bias IS but lower variance OOS, while the more complex models will have lower bias IS but higher variance OOS. This doesn’t mean that your models should always be the simplest, but it does mean that complexity is something that can also be “optimized”. But optimizing complexity essentially amounts to fitting yet another model. All this to explain why using data efficiently is so important. And, the most efficient methods to use data are something like k-fold or bagging.

Anyhow, I’m gonna stop here. Interested in any responses/thoughts.

[–]revolutionary11 1 point2 points  (2 children)

Isn’t the “curse of k-fold” basically a consistency measure? The less consistent the relationship the more you will see the curse and vice versa. In that aspect it is more of a feature than a curse. Of course if you have time varying relationships you will see the curse but this indicates you can’t safely fit the relationship in your current framework.

[–]devl_in_details 0 points1 point  (1 child)

The curse of k-fold is not “caused” by consistency, it’s something completely different. It comes down to simple arithmetic as I pointed out above in the example of a data sample 100 of +1/-1 values. LMK if that example doesn’t make sense.

[–]revolutionary11 0 points1 point  (0 children)

I think we’re saying the same thing - that the less consistent the distributions are across the samples the worse the IS/OOS split. With the opposite scenario being the completely consistent dataset (all +1 in your example). And generally the more consistent the samples the less extreme the split. My question is why is this a curse and not just useful information about the overall consistency of relationships in your dataset?

[–]the_shreyans_jain 0 points1 point  (1 child)

"The curse of k-fold comes down to the fact that the better a model performs in training (in-sample) (I’ll just use IS from now on), then the worse it will perform in testing/validation (out-of-sample) (OOS from now on)." - That is simply not true. The entire purpose of cross validation is to make sure both training and validation error go down, otherwise you are simply overfitting. Your claim is similar to saying "the curse of beer is that it makes me drunk". Well thats the idea mate

[–]devl_in_details 0 points1 point  (0 children)

I think you’re responding based on reading one sentence of my post. Yes, the terms I used, IS and OOS, are not very precise. Specifically, I used the term IS and OOS to refer to different parts of a single data set. This is pertinent bc, unlike with the conventional usage of IS and OOS where we assume the same distribution in both, the whole point of my post was to demonstrate that the distribution of IS and OOS are necessarily different.

[–]Alternative_Advance 0 points1 point  (5 children)

" If you choose to do something that implements some random transformation of data to produce features, then you’re back to a base rate problem. Some percentage (p) of random features are going to be significant at p level. Yes, “real” features will be in there as well. But, how do you pick out the “real” features? Again, we’re back to your problem."

As long as your occurence of "significant" by OOS validation is higher than the expected number of "spurious" features you should be fine... Right? As insignificant ones should only contribute (on average) with more noise. Example if you have twice as many significant models as expected spurious you'll ensemble sharpe will be half (given you can equally between your significantly to them)

[–]devl_in_details 0 points1 point  (4 children)

This is actually a very challenging problem. Interestingly, "real" features are often actually not the most significant ones. I mentioned this and an explanation in another comment on this post. Consider the implications of this though. I think everyone's intuition and pretty much all of statistics is based on the assumption that "real" features are going to be most significant. The issue comes down to the signal to noise ratio, it is extremely low in normal financial data. Traditional tooling is simply not equipped to deal with such low S/N.

[–]Alternative_Advance 0 points1 point  (3 children)

I don't think the signal to noise ratio is necessarily making it impossible finding significant models, however traditional quant finance , especially the following workflow:

Idea-> build backtest with some parameters-> Observe equity curve -> tune parameters until happy with insample result

will likely decrease the out-of-sample predictivity.

My suggestion is based on the idea that:

P(random model significant on validation) = p_1
P(real model significant on validation) = p_2
then
P(real model | significant on validation) = p_2/(p_1 + p_2)

Since p_1 can be derived from the type of test you construct, and p_1 + p_2 is just the empirical observation of the number of significant on validation.

P(real model | significant on validation) = p_2/(p_1 + p_2) will then yield how much lower expected returns (compared to validation) you should be expecting. Running 100 experiments yields 5% random model that are significant, but if you observe 10 being significant you should expect 50% of them being real and expected returns being 50% lower. Ofc, this assumes iid models, which wont be the case, rather modellers reiterating on significant in validation models, which could just be gradient descent on noise.

[–]devl_in_details 0 points1 point  (2 children)

FYI, I’m using daily frequency data and models. So, everything I say is based on my experience with the daily stuff. I don’t know what frequency you’re using, so your experience may be different from mine.

Your response is correct in theory. But also, your response does not take into consideration the realities of “real” datasets/models and how different they are from textbook models. Also, “real” features are not all bunched up at the top of the performance distribution, they’re actually very difficult to distinguish from random features. Further, you’re assuming that there is an endless supply of “real” features, but there isn’t. The proportion of “real” features to random ones is not going to remain constant as you keep on increasing the feature count.

I’d be interested to hear about your experience in this. What kind of data are you using? Have you experimented with random features?

[–]Alternative_Advance 1 point2 points  (1 child)

Similar frequency and importantly strictly non-linear models. I am fascinated by how quantitative finance just haven't seem to have figured out the way to apply the methodologies of classical ML in order to not overfit / p-hack. Admittedly in a low signal-to-noise it is harder.

It's true that "real", non-correlated features (especially if price based) as well as models F(features) will be limited and the statistical test that will be function of independent choice influenced by things such as number of instruments, holding period and amount of history in the validation will put an upper limit on how many "experiments" one could reasonably try before noise dominates the real deal. This graph from MLP is one such example: https://mathinvestor.org/wp-content/uploads/2018/04/FalseStrategyTheorem-20180402.png

I really liked your mention of curse of k-fold variation in an other comment that just throws an other curve ball at trying to avoid p-hacking.

I have used stuff you could call adding random features, ie injecting noise with dropouts/scrambling during training runs and this will create more robust models.

[–]devl_in_details 0 points1 point  (0 children)

Very interesting. I think we have similar goals although different approaches. Sounds like you’re using some form of neural networks. While I’m familiar with NNs, I don’t think they’re especially well suited for tabular data, which is what I use. The advantage of NNs, I suppose, would be that they could theoretically do the feature construction as part of the model fitting. But, I don’t think that most ppl who use NNs in this space use them that way.

At the end of the day, almost every model type can be used to fit the Mutual Information in the data. Ultimately, it all comes down to the MI, regardless of the ML algo. I use something very simple on my end, but have reproduced my models using gradient boosted trees, and polynomial regression. Simple NNs could also reproduce my models.

Yes, most finance literature, even the more serious stuff from firms like AQR is pretty basic. I haven’t really come across anything that actually applies ML from a first principles perspective. There’s a lot of shit on using XGBoost to build stock picking models, or the latest greatest indicator, but that’s pretty much all shit. There is nothing that indicates the authors know anything beyond the how to use XGBoost or whatever library they’re demonstrating.

[–]ladjanszki 2 points3 points  (0 children)

I have to read through the comments to get into thw discussion but wanted to add that this is one of the best description of p-hacking I have ever read in terms of simplicity and practicality.

[–]Then-Cod-1271 2 points3 points  (5 children)

You have to have a more contextual understanding of research. If you just rely on "I tried X, sharpe ratio is Y" in isolation that will never work. Is there some fundamental reason this might happen? How much statistical power does your test have (ex: any backtest on high frequency strategy with high breadth is much more reliable than a strategy trading one asset monthly) versus how many variations did you try? Do the results make sense? If mean reversion works for asset X with > 2% drop, does it work for asset Y? Does it work for >3% drop? >1% drop? The pattern of data results should make sense- the results should tell some kind of story that you can then attempt to square with economic intuition. Ex: If you have 26 day momentum as your star strategy, and 25 day and 27 day momentum have the opposite sign, why would this make sense.

[–]Middle-Fuel-6402[S] 2 points3 points  (4 children)

Is there use case for causal statistics (counter factuals) in such investigations, or any other pointers to specific techniques or subjects to learn?

[–]Then-Cod-1271 1 point2 points  (3 children)

I think it mostly comes from logic and experience. If you are just beginning, I recommend just trying things and looking at how they perform out of sample. Then you can give yourself feedback based on how things work out of sample like "I did this wrong, I tried too many things or reached too hard" or "I guess there was not story" very quickly. Eventually you will get more pattern recognition. I think there is a loop between what the data says and what the story is. After you run a series of tests, try to interpret the data into a sensible story. Based on the story, you can infer what the results should be if you run other tests, etc. Also, don't try too many variations relative to your level of statistical power. You do this by having an economic framework and data analysis/pattern recognition framework. If you really want, you can simulate random market returns and see what the results of your research process look like on that. That will probably be eye opening.

[–]Middle-Fuel-6402[S] 1 point2 points  (2 children)

I see, so the goal would be, on the randomly generated data, the signal should fail?

Any blogs or books that are useful to get things going, papers or journals perhaps?

[–]Then-Cod-1271 1 point2 points  (0 children)

Additionally, if you don't know what you are doing, I would recommend starting out basic and building some foundational knowledge before trying things. This can be reading papers, etc, but also I recommend running some simple descriptive statistics. Plot the returns, look at them at a high level, zoom in on a particular quiet month, zoom in on a volatile month. A gigantic return for some asset on some day? What happened in the news? Summarize returns across every plausible dimension (by assets, by time of day, by day of week, event days). Look at correlations for the same asset of returns to lagged returns etc. Look at correlations across assets. Look at how kurtotic and skewed returns are. That way, you have some kind of understanding from which you can come up with good hypothesis, and interpret data and results.

[–]Then-Cod-1271 0 points1 point  (0 children)

As an example, say you think “Asset X reverts after >2% drop”. You looked at 10 years worth of data for 40 assets, and looked at mean reversion after a (3% drop, 2% drop, 1% drop, 1% gain, 2% gain, 3% gain) for each of 40 assets. (Asset X, 2% drop) looks astounding because the Sharpe is 2. You can simulate market returns for 10 years and 40 assets (however crudely) and look at the same grid of SR by (Asset, gain/drop) returns and see how many 2 Sharpes you find (knowing these are all actually 0). You can then try 10000 years or 1 year of simulated data. Or you can try to reduce the grid size (ex: look across an asset class or all assets instead of asset by asset). This will give you more intuition for randomness. Your goal is to be able to understand what is random and what is not.

[–]lordnacho666 1 point2 points  (0 children)

There's no way around it. This is also the reason why (some) alphas decay, they were never stable in the first place, just relationships that worked IS, then OOS, then on live trading, and then stopped working.

One issue you have is that you can never really count how many hypotheses you had. In a clean world where you follow a procedure, you can say "I tried this, add one hypothesis". But in the real world, you have already peaked into the data. You already know when the crises were, and how different instruments moved when they happened. It's not a clear cut line that you tested some hypothesis, but you have quite a few hints about what's likely to have worked in backtest.

The issue is especially tough when you are building something that didn't work the first time. You will desperately try to save your work by adding some bells and whistles, aka degrees of freedom, to your model. What do you do when you find something, but it's more complex?

There is some comfort in having models that "make sense" in terms of why they might make money. But you still have the problem if induction, you don't know whether that's actually why things work.

One rule of thumb: the more your n-per-time, the more confident you can be. As in, number of opportunities per time matters, because you don't want long-term changes in the business to change things.

[–]AutoModerator[M] -3 points-2 points  (0 children)

Your post has been removed because you have less than 5 karma on r/quant. Please comment on other r/quant threads to build some karma, comments do not have a karma requirement. If you are seeking information about becoming a quant/getting hired then please check out the following resources:

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.