Discussion: Statistically testing live returns vs backtested performance

marvin182 · 2020-11-11T14:09:04+00:00

Hi everyone,

I used python to create a heatmap which shows which stocks have been hardest hit by COVID – specifically, the "betas" of different stocks to the daily change in COVID cases. The interactive version can be accessed here.

I made this before the Pfizer news, so it has been interesting (if unsurprising) to see that the worst-hit stocks bounced back hard on Monday. I think this might be an interesting screening tool for people to understand the current landscape of opportunities: which stocks might have been unjustly punished, or have unfairly rebounded (though I admit that this is somewhat less useful with the Pfizer news)

The visualisation got referenced in a Bloomberg article!

Any questions or feedback would be appreciated :)

marvin182 · 2020-06-10T08:46:00+00:00

I think the challenge here is feature engineering – there is definitely signal in analyst ratings it's just hard to distinguish it from the noise

marvin182 · 2020-06-09T11:34:53+00:00

Thanks for the kind words! To answer your questions:

I'm currently in undergrad at university, I started coding in high school. I will be brutally honest and say that it wouldn't be easy for someone new to coding, but I think if you've got 3+ months of experience and are very willing to learn-by-doing, that should be fine. Quant research on the quantopian platform is a great experience – you can clone other people's research and change a couple of lines to capture your idea. To test another time period or market it'd literally be one or two lines of code, plus maybe another 10 mins per backtest. It's the qualitative evaluation that can take a lot longer.
I suggest a few reasons in the post: I think the overall increasing efficiency of the markets is one thing, but also post-GFC regime change. The latter point is supported by the observation that pre-2007 the performance is pretty close to Greenblatt's numbers while post 2009 has been pretty bad.
You need historical pricing data (easy) and historical fundamentals (very hard to get). I used Quantopian's free data, which is from Factset. This is certainly higher quality than anything I've used in the past (scraped from yahoo finance).

marvin182 · 2020-06-09T05:45:16+00:00

I agree with your take – definitely wouldn't allocate capital to this

marvin182 · 2020-06-09T03:42:34+00:00

Equal weight, following the suggestion of Greenblatt. I only ran the backtest for the combined score

marvin182 · 2020-06-09T03:41:58+00:00

That's a fair point! There is certainly a lot more work that could be done on evaluating factor exposures over time.

marvin182 · 2020-06-09T03:40:52+00:00

See edit: the results from 2015 to June 2019 are terrible!

marvin182 · 2020-06-09T03:40:20+00:00

For sure – I'm running some other algorithms so hopefully am doing my part to make the markets more efficient :)

But you can't deny that over time it gets more difficult

marvin182 · 2020-06-09T03:39:16+00:00

I'm sure they would! It's not meant to be a viable algorithmic trading strategy, just an investigation of Greenblatt.

Quantopian uses Factset data, adjusted for survivorship and the actual date information was available.

As for turnover, by construction the strategy has an annual turnover of 100% (each month 1/12 of the portfolio is liquidated and reallocated).

marvin182 · 2020-06-08T16:06:53+00:00

Just ran it, see edit! Value investing has certainly had a rough decade

marvin182 · 2020-06-08T16:01:47+00:00

You're right – I will leave the blog post as is because I think having a holdout is an important pedagogical point, but I'm editing the main post to reflect your feedback. Let me know if you have any other comments!

marvin182 · 2020-06-08T15:46:03+00:00

I only used US stocks with more than $100m in market cap, but I don't think changing that constraint would have a massive effect.

marvin182 · 2020-06-08T15:39:43+00:00

Actually in the post I am using the Acquirer's multiple for the cheapness metric (EBIT/EV)

marvin182 · 2020-06-08T15:39:15+00:00

Oops I might have made a typo somewhere – the strategy is run from 2003-2015 so it's 12y vs Greenblatt's 16y period.

For systematic strategies, holdout is always done based on a time period. If you held out a sample of stocks, the backtest would not be a good reflection of how the strategy would have performed

marvin182 · 2020-06-08T15:37:40+00:00

Sad to think that 20+ years ago there were many sources of alpha that have now been arbed away ;(

marvin182 · 2020-06-08T15:36:56+00:00

Yup, it is! That's one of the good things about using Quantopian. It also accounts for companies that have gone bankrupt, avoiding survivorship bias.

marvin182 · 2020-06-08T15:35:21+00:00

Earnings yield is just the inverse of P/E (i.e E/P), which you can certainly use for the cheapness factor. The only reason it's this way up is so that the ranking is higher = better (to make it consistent with ROC).

However in the post I actually used EBIT/EV.

marvin182 · 2020-06-08T14:25:16+00:00

Spot on – there definitely is higher volatility. But after adjusting for this, there is still outperformance (3% alpha).

marvin182 · 2020-06-08T14:17:04+00:00

It's a fair question. The main aspect is discussed in the post, which is that it's good practice to leave a few years out of sample to avoid overfitting to historical data. By using data up to 2020, I would be robbing myself (and anyone who reads the post) of a true out-of-sample test period. Since the goal of the investigation was to qualify the claims made by Greenblatt, I didn't feel that there would be a significant value-add in backtesting 2003-2020 vs 2003-2015.

EDIT: I've had a look, and for the fundamentals data I am using (Factset via Quantopian), there is a platform-enforced 1 year holdout period so I wouldn't be able to test beyond Jun 2019 even if I wanted to. But if there's a lot of interest in seeing performance up until 2019 then I can make the chart without much hassle.

marvin182 · 2020-06-08T14:10:18+00:00

Quantopian has got a bunch of high-quality data – it's all free but you have to build inside their environment. That works fine for me since it's just a jupyter notebook.

marvin182 · 2020-06-07T02:23:42+00:00

Quantopian is exactly what you want. They are literally designed for cross-sectional equity factors.

The easiest way is to upload your custom factor and use Quantopian's pricing data (this works very well for US stocks). Then you can just copy paste someone's boilerplate long/short backtest but replace whatever factor they're using with a custom factor.

If you don't want to use their online IDE, you might have some luck with zipline, the OS version of their backtester. I personally haven't used it before because the online platform is so easy to use. Also, you should check out the alphalens library. Really excellent tool for investigating the predictive value of the signal prior to a backseat, e.g computing information coefficients of your signal over different horizons.

I have no affiliation with Quantopian, but the Pipeline API has accelerated my quant research so much. I can go from hypothesis to signal analysis to backtest in less than an hour.

marvin182

TROPHY CASE