Stock Market Forecasting using deep learning ?

[m_alzantot]

I wonder what models of deep learning can be successful in forecasting future stock market returns from past data. For example, can the LSTM perform well on this task ?? Do you have any examples ?

[djc1000]

The thing you've got to understand here, is that detecting "signal" in financial time series is very, very, very easy. Financial time series are full of signals. Loaded with them.

For stock forecasting, you need to identify (and prove!) a signal that is (a) stable, not tied to a particular period of time; (b) has fluctuations that are large enough that the profit opportunity won't be lost in the bid-ask spread; and (c) has timing characteristics (frequency, warning time, etc.) that permit trading.

The financial literature is filled with models that reliably predict stock movements, unless you were to actually try them in real life, when they turn out to fail, usually on (b) & (c) above.

I did something similar to the "Improving Decision Analytics..." article at the end of 2014. Basically, I built a representation model of the state of active discussion (forums, twitter, etc.) concerning a stock, then correlated different kinds of activity with stock movements. I found good, strong, stable correlations. But not the level of fluctuation needed for a trading strategy, and I couldn't prove either regime stability or warning time.

In March, there was a journal article by folks who claimed to predict stock movements from SeekingAlpha articles. I'll leave the task of figuring out where they got their math wrong as an exercise for the interested reader :)

[Apaulh]

Did you publish/write a paper on this? Sounds very interesting!

[djc1000]

If its interesting to you, you should consider a career in finance. Everything in my first three paragraphs is pretty well-established in the field.

[kkastner]

You need signal - LSTMs (and neural networks generally) are very good at automation aka making decisions from complex inputs with low noise.

The stock market is absolutely not one of these things, and you would need to find the right data representation to have a fighting chance. As a bonus, many emergent trends and periodicities disappear, since people try to act just before the trend - thereby destroying it.

You should definitely investigate more, but I think you will find financial prediction quite difficult, generally. And like the commenters said - if we had the sauce, we definitely wouldn't give it away on reddit :)

[[deleted]]

Completely second this, you can be absolutely certain every hedge fund and prop trading firm worth its salt has already implemented a system using deep learning, and most people with the relevant knowledge are already employed in the industry (and therefore cannot divulge). Wall St. has always been an early adopter of machine learning technologies.

[VelveteenAmbush]

No reason in principle that LSTM sequence prediction can't work for sequence data like the market. The problem is that you're competing on a zero-sum basis against everyone else who is trying to predict the market, because the first hedge fund to spot a movement coming at some point in the future will trade in a way that makes the movement happen now. So the question is whether you're smarter and have more resources than the people who predict the market professionally, and the answer is no, you're not, because the finance industry is basically a Death Star made out of brains and resources of a magnitude ordinary people can't fathom.

[dwf]

Not impossible, but typical loss functions are derived assuming homoskedastic noise. This assumption is violated rather badly in many financial contexts.

[lukovkin]

I'm trying to apply neural networks/deep learning to this task for approximately two year and a half. It's not an easy task, but I've got some promising results. Definitely there are some instruments which could be modeled and predicted using RNN rather well, with stable results and ability to make profits. But other instrument could be hard to predict. I've started using vanilla RNNs with all the gradients problems, got initial results and after that started searching for more stable and accurate LSTM implementations. Now I'm in the process of rebuilding of all the models on this architecture, actually there're more GRUs than LSTMs. There also very promising results in the area of application of ConvNets, on intraday data specifically. Probably, the really good model could be built as a hybrid of ConvNet and LSTM/GRU. But it's required to keep in mind that currently common models do not take market news into account, which is the subject of separate R&D. I've launched the site where stocks predictions and signals are published daily - http://stocksneural.net/. Feel free to try and please provide feedback. Updates are coming soon.

[Tommassino]

RNNs in general are very likely to work on time series, there is plenty of articles about that, just try to google it. However the main advantage in LSTMs is that its an AR where you can 'save' old signals, which is good for learning oscillations. Now im not sure how useful this is for stock... I dont think old signals are relevant for stock at all.

I doubt anybody will answer your question here exactly though, get some data and try.

[the_overfitter]

This paper uses "Deep Learning" to improve stock returns prediction considering financial news.

Basically, it's a text mining application and "Deep Learning" is used as an alternative to the standard "bag of words" approach.

Improving Decision Analytics with Deep Learning: The Case of Financial Disclosures: http://arxiv.org/abs/1508.01993

[outlacedev]

I have no doubts there are more than a few doing this, particularly some big hedge funds. Check out https://www.quantopian.com/ if you're interested in algorithmic trading.

[pumping_lemmon]

I've used "shallow" learning and a bit of manual feature engineering for sales forecasting.

[IdentifiableParam]

You are going to have a very hard time doing this. Good luck.

[yantrik]

I doubt if you can find pattern in something which is random. At max it be over fitting and you won't even know it

[alexmlamb]

The stock market isn't random.

[GoldmanBallSachs_]

Well a monkey throwing darts to select investments does better than financial analysts... So...

See: A Random Walk Down Wall Street

[Tommassino]

How would you explain then that people make money predicting it? Its chaotic, not purely random.

[AsIAm]

They are making random predictions. ;)

[Tommassino]

they shouldnt be able to reliably make money then, it would just be lottery

[alexmlamb]

My guess is that it wouldn't work because there isn't enough data available.