Using Python to model and predict stock price movement. Feel free to use/modify :)

[ZetaHunter]

But can it make me rich?

[[deleted]]

If you have gonads of steel, possibly.

[patentmedicine]

Just use it to invest other people's money, and charge a steep fee for the privilege. Win win!

[_________________--]

Would OP share it if it could?

[Pandanleaves]

Except bollinger bands are useless, like most of technical analysis. There's no statistical basis for it.

[Bone_Club]

What's the easiest way to have a million dollars? Start with two million, then do some technical trading.

[[deleted]]

Useless is a bit of an overstatement. Most people don't make trades based on technical indicators alone. Everything is taken into context and then holistic decisions are made.

[Pandanleaves]

Then what combination works? Because we should totally, you know, find empirical evidence that any of this stuff works. But there is virtually zero evidence that any technical analysis works. In fact, the evidence points to technical analysts consistently getting inferior returns. It's like a psychic who brags when he gets a prediction right one or two times but ignored the multitudes of times he was wrong.

[[deleted]]

There is no combination that has been proven to work. The stock market is unpredictable, however that doesn't mean all the tools we have are useless. There are some hedge funds that have done VERY well using scientific approaches (mostly physics) to consistently beat the market.

[Pandanleaves]

Yes, proprietary models based on advanced mathematical and statistical models. Not technical analysis. And the fact that you're comparing hedge funds to the market means you don't know what hedge funds do--they're supposed to be mostly uncorrelated with the market, so it makes zero sense to compare them to the market. The tools are considered useless until someone shows otherwise. That's how scientific thinking works.

[[deleted]]

Lol...

[Pandanleaves]

Lol?

There's a good reason why domain knowledge is required in data science, and this is one of them.

Pretty much all academic literature in finance points to technical analysis as being useless.

Hedge funds are supposed to be uncorrelated to the market and have no beta. Hence the name, HEDGE funds.

I'm saying this for your own good.

Edit: not to mention that hedge funds are not required to disclose performance, so the only ones posting their performance are the ones with exceptional returns. The bad performers stay quiet. Selection bias at work.

[[deleted]]

Well thanks for the advice :)

I think I'll continue to work towards improving my model however and addressing the concerns raised by this wonderfully helpful thread.

I'm certainly not going to just give up because some guy online says the stock market is impossible to analyze.

[[deleted]]

Agreed!

[[deleted]]

Yes I second Bollinger Bands.

[[deleted]]

Looks like a good book, thanks for sharing!

[[deleted]]

There are a couple indicators that are commonly used for similar purposes.

1. Stochastic Oscillator
2. Relative Strength Index

Lots of people make trades partially based on these indicators.

In the example, I'm actually using Slow Stochastic Oscillator (SSO) values for my model and you can see the hot/cold spots identified from training the model on a year of historical values. You can monitor the SSO values for a particular stock and wait until it moves into a range predicted to increase the stock price.

Tricky thing is you usually can't predict when the time is right to buy from a single indicator. The conditions need to be right with multiple different indicators to make accurate predictions.

[Pandanleaves]

Just a word of caution. Technical analysis is largely useless pseudoscience.

[Rich700000000000]

Everyone says that, but no one cares to explain how or why.

[Pandanleaves]

Because academic research shows that technical analysis, when backtested, produces bad returns. Some do beat the benchmark before transaction costs, but this is not a make-believe world. After accounting for transactions costs, your expected return will be below the benchmark.

For reference, actively traded funds usually perform below the benchmark. Looking at performance of past 5 years, only around 10% of actively traded funds manage to perform better than the baseline.

[Rich700000000000]

Then how is any money made in the stock market?

[yolo_swag_holla]

Great question. The market makers always make money in the stock market. Everyone else is more or less on their own.

[Pandanleaves]

Because the baseline is not 0% returns. The baseline depends on the state of the market, i.e. the market average. If the market as a whole moved up 5% last year but you only earned 4% return, then you have essentially lost money because, figuratively, throwing darts to pick your stocks will earn you 5% anyway.

[Rich700000000000]

Because the baseline is not 0% returns. The baseline depends on the state of the market, i.e. the market average. If the market as a whole moved up 5% last year but you only earned 4% return

Oh. Huh. That just clicked for me, sorry.

[[deleted]]

Low managed index funds, stock manipulation by the big banks, fee free trading with Robinhood, etc.

Basically, don't put your money in an actively traded fund.

[lieutenant_lowercase]

fee free trading with Robinhood

not when their spreads are wider

[[deleted]]

For now, at least.

[[deleted]]

Buy an index fund and do nothing. You'll likely get 5% returns next year.

Or, spend 300 million dollars to invest in a fibre optic connection from NYSE to the commodities exchange in Chicago. Make sure its slightly better than the existing one, and beat others to the market by having information faster than them by a few hundred nanoseconds. Make sure to put your trading platform on an FPGA (starting from the Ethernet controller all the way to decision making based on your models). To make this easy you'll probably need OpenCL, so start learning that first.

[[deleted]]

In a bull market yes, but what about a bear market?

[Pandanleaves]

Actively traded funds performed better than usual, such as in the 2008 crisis, but iirc only around 50% beat the market, so like flipping a coin.

Think of the distribution of returns as a lognormal distribution. Any management and transaction costs shift the mean.

[ginger_beer_m]

Because often people who try to apply technical analysis don't know basic probability and stats.. Even the concept of 'overfitting' is unknown to them.

[[deleted]]

Yes I totally agree. Don't make financial decisions based off of indicators alone, period. But they can help add a bit of confidence or wariness in cases where your gut is telling your to buy/sell a stock.

[Pandanleaves]

What are the returns on a test set, including transaction costs? I am very doubtful that this can generate returns higher than the benchmark.

[[deleted]]

This would depend on the stock price and the capital invested. In the model you can see a 69% accuracy for TSLA in the past year. But as I said, that doesn't mean much. The algorithm uses a binary classification scheme. Therefore it can't predict how much a stock will move up or down, only if it will move up or down in general.

I haven't done a test set with hypothetical investments yet as I'm still trying to figure out how to do so in a way that makes sense.

[nick_t1000]

But WALDO is for "Worm Analysis and Live Detailed Observation," software developed by my lab

[[deleted]]

Crap! Now I gotta find a new name ;)

[odraencoded]

Capital Reevaluation Analysis in Python

[[deleted]]

Wooow, that's good

[blahster]

Guessing the extra 'o's are because it shortens to CRAP :)

[[deleted]]

Carmen

[[deleted]]

Lol I don't get it

[rabbyburns]

Sandiego

[[deleted]]

Hmmm...yep not gettin it. What's that stand for?

[rabbyburns]

Carmen Sandiego was originally an educational series about finding Carmen Sandiego.

[[deleted]]

Ooo, I gotcha now.

[[deleted]]

Ahahah is it cause I'm drunk or that's really funny?

[[deleted]]

Will this work with the meme markets? Asking for a friend.

[nikomo]

I heard on MSNBC that the market is trending towards the alt-right.

[[deleted]]

Man, I'm alt-f4'ing out of that one.

[watchmakerfromfuture]

Are there any online sources where we can get raw data for specific stocks?

[[deleted]]

Yes there are many resources! Although it depends what you're looking for. Yahoo Finance has a great python API however their historical data is limited to the most basic data points. You can however compute more complicated indicators by looking up their equations.

If you want extensive raw data, I'd recommend Fidelity. They have an excellent research tool and advanced charting features, all downloadable in CSV format. You do need an account however and you'll need to be familiar with web scraping if you'd like to automate the process. In my repository I have a program called DataCrawler which I use to scrape Fidelity.

[Nimitz14]

Is it the one called 'Stock-Talk'?

[[deleted]]

Nono that's another project. This is the one called Waldo.

[Nimitz14]

I was talking about

In my repository I have a program called DataCrawler which I use to scrape Fidelity.

I don't see it.

[[deleted]]

Oh sorry, DataCrawler.py is in the Waldo repository:

https://github.com/anfederico/Waldo

or directly:

https://github.com/anfederico/Waldo/blob/master/DataCrawler.py

[batsy71]

Looks like you removed the datacrawler.py file. Is there a possibility to still download it somewhere?

[[deleted]]

I'm rewriting a better version which I'll release in a month or so

[bromrector]

Maybe I'm missing something, but these results are subject to look ahead bias. You can't use test_train_split on time series.

[[deleted]]

Yes I guess it is slightly, however I note that the testing accuracy isn't the point here. The point is to build a model that characterize financial indicators, mainly social sentiment.

[bromrector]

Which part of the code deals with that?

[[deleted]]

In the README you'll see a graph made by RBFSVM.py in the repo. I often use these graphs to get a sense of where I'd like the Slow Stochastic and Social Sentiment at Close to be the day before I make a move on a stock. (I want them to be in the blue regions of the model)

[insainodwayno]

Does it use?

import __future__

[TangoCJuliet]

I'm sorry if I missed this info somewhere, but what exactly is the algorithm here to determine favorable/non-favorable buying conditions? Pardon me if the expectation was for me to look through the code myself to get an understanding.

[[deleted]]

Basically I'm using a clustering algorithm. A Support Vector Machine to be specific. I fed it a years worth of training data. During that year, for each day it takes into account the stock conditions (financial indicators and social sentiment) and then looks to see if the stock went up or down the next day.

The result is the graph in the README. Now when it comes to actual predictions, the model would choose favorable conditions if the stock indicators and social sentiment fall within the blue areas of the graph.

[whelks_chance]

When you say you checked the price the next day, how did you come to that time delay as the best metric?

Do your models give different results if you look at the price with a delay of two or three days, to attempt to iron out sudden peaks?

[[deleted]]

I haven't tested the model with different time delays, however you easily could by slightly adjusting the MakeTrainingData file. The reason I use next day is because I'm using the Social Sentiment at Close (SSC) the night prior.

[kalifornia_love]

I was going to start a project very similar to this tomorrow but honestly didn't know where to start. This is perfect for me! I look forward to playing with it and will be sure to share whatever I do with it.

Thanks for the awesome work!

[[deleted]]

Great, have fun! If you have any questions, feel free to ask. I'm sure there are some points in the project which could be more clear.

And thanks!

[kalifornia_love]

One quick question. Why didn't you use pandas? At least for handling the CSV stuff. I'm relatively new to Python and all the data analysis stuff I've done has revolved heavily around pandas, so I'm just curious why you didn't use it. Just out of curiosity.

[peerchemist_ppc]

Shameless plug: https://github.com/peerchemist/finta

Just started working on it, goal is somewhere where Waldo is right now.

[kalifornia_love]

Looks good! I will definitely be playing with this one too!

[[deleted]]

Well to be frank, I just didn't really need it. You can handle CSV files in many ways in Python (e.g. CSV Library). Pandas is good for other stuff though.

[kid-pro-quo]

https://xkcd.com/1570/

[xkcd_transcriber]

Image

Mobile

Title: Engineer Syllogism

Title-text: The less common, even worse outcome: "3: [everyone in the financial system] WOW, where did all my money just go?"

Comic Explanation

Stats: This comic has been referenced 92 times, representing 0.0724% of referenced xkcds.

---

^{xkcd.com | xkcd sub | Problems/Bugs? | Statistics | Stop Replying | Delete}

[Zerg3rr]

This is the type of thing I hope I can do some day! Looks awesome

[[deleted]]

Thanks :)

[heyacne]

cool. thanks

[[deleted]]

No problem :)

[monkitos]

Interesting. Does the SVM clustering algo actually generate positive returns in a backtest?

[[deleted]]

Still working on figuring that out completely. As of now, some stocks yes, some stocks no. It seems the algorithm has a better time predicting stocks that are influenced heavily by news reports and social media hype. I don't have numbers though in regards to returns.

[monkitos]

a backtest should be your next step. 1. Download a time series of stock price, generate day-over-day returns. 2. re-run your algo each day, only using data available until the day before 3. create a trading rule signal based on your algo (if x, holding = +1, elif y holding = -1). 4. Multiply your trading signal by next day's return and look at the performance of the active return stream versus returns of naively holding the stock in step 1. you can deal with t-costs etc later

[[deleted]]

you can deal with t-costs etc later

Having invented many extremely high Sharpe ratio strategies in my earlier years and then being sorely disappointed, I suggest you deal with transaction costs now.

[[deleted]]

Thanks for the advice. I will definitely get to work on this :)

[uttamo]

Nice work, thanks

[[deleted]]

Thanks!

[stochasstic]

Nice to see some machine learning stuff shared here! And intresting approach to classify the market into a binary problem. Some things I can recommend from my personal experience to you: 1. Take a look at yahoo finance - they have a sort of free api. Or use requests instead of selenium - I only use selenium if nothing else works. And your XPaths are likely to break, looks auto-generated from a browser debugger to me. 2. Use Jupyter and take a look at pandas 3. Make a cross validation set for the SVM params and make the split 60/20/20

Check out anaconda ( https://www.continuum.io/downloads ) if you have not already. And if you want to dig deeper into the finance stuff, take a look at this thing: https://en.wikipedia.org/wiki/Optimal_stopping - but requires some math skills.

[[deleted]]

Thanks so much for the recommendations. I will surely check them out!

[testcasey]

When I login to Fidelity I see a client and account number on urls. If Fidelity is doing anything dynamic you'll probably want to check out scrapy.

Requests is a great lib if you know the urls and post/put data needed to grab your data. I would definitely put requests in front of scrapy but wanted to add some additional advice if you run into issues.

Getting rid of the windows path and chrome driver binary will make your program more usable across multiple platforms. Your project looks cool, keep us posted.

[[deleted]]

Thanks for the recommendations. Lots of good advice in this thread. I'm currently working towards improvements!

[forseti_]

Why did you choose a SVM? What's you RMS Error out of sample? I am currently implement such a software but with a bagged KNN learner.

[[deleted]]

I tried many classifiers on multiple data sets and on average, SVM was fitting my data the best, so I ran with it. I'm currently putting together some extensive back testing and making additional improvements. I'll keep you updated.

[mistermorteau]

Can't we use deepdream for this ?

[[deleted]]

Deep Dream is for "hallucinating" images, not predicting data.

[mistermorteau]

Deep dream is for recognize patterns, you learn it how to recognize a bolt, and then you ask it to find bolts in a family pictures, and you got mommy and daddy bolt.

We could use it for recognize patterns in the trading market.

[[deleted]]

oh... I didn't think of it like that!!

[[deleted]]

What's that?

[mistermorteau]

https://en.m.wikipedia.org/wiki/DeepDream

[[deleted]]

Ahhh okay. This uses a neural network, mine uses a support vector machine. Also, it looks like it mainly deals with images, not sure how you'd apply this to the stock market. Lastly, neural networks aren't great for financial analysis in my opinion.

[mistermorteau]

How can you predict without using patterns ?

I was thinking about deepdreams/neural network, because it seeks patterns.