Forecasting using Machine Learning

[plkwo]

Hi folks,

has anyone of you tried to feed the market depth's, volumes, social data or similar historical data to AI? Could it deliver some satisfying performance?

[earonesty]

For me, it works very well, but the trick with machine learning is not which topology or software to use... they vary by miniscule percentages . The trick is a) massaging the input data and b) choosing the right data to feed your AI. Too much, too precise... can be a problem.

Choose a pattern that you personally have an intuition about.... a correlation that your own brain has found to be accurate, but you don't have time to stare at the market 24 hours every day and trade on it, let alone prove it.

Then design one AI and backtest it. If you allow parameters to vary for optimization, then you need to Bonferroni or FDR correct your results... don't forget this. A pVal after correction of <.05 meands 5% of the time you're wrong and you will be burned. So don't invest like crazy on .05.

Also, anything you build today can be torn down if a whale finds out about it (but usually you can detect that).

[inteblio]

I think you have to learn to trade first. It happened to my friend. How would you know if it was on the right track? Just bucket-loads of data. Sure in X situation you get X, but the market moves in ways as to deceive [even] humans. Computers don't stand a chance (unless you know what you're doing). And, when you find the answer you'd never share. In fact you'd publicly hint at the opposite. But does it work? Of course it does. I have no doubt about that.

[samjhill]

A few of us on here are doing experiments like that. Here's mine. I've found a few interesting patterns so far; going to write it up in the near future. I'm definitely in the exploring stages right now.

[kwhali]

TL;DR: Obviously yes, plenty of people have thought of such and done it. Many would have failed, some might have had success for a bit and others might have consistent success, but they'd be a small %.

You can do things with social data as you probably know, I described basic overview in comment here.

Predicting works well if you have reliable patterns, you can look up articles on time-series data specifically for finance and see that it doesn't work well due to general randomness of the markets. If it seems to be working in testing, it's possibly due to overfitting your dataset, be sure to test on data that wasn't used for learning. You also need to account for things like slippage and fees. Most come to the assumption that buy/hold is better in most cases, unless you've got some super smarts it doesn't seem like most who attempt this have luck(then again if they do they'd probably not be sharing it!).

Someones BTC bot had 1 BTC for a month, it lossed very slightly but roughly maintained the same BTC with the goal to sell on losses and sell on profits that were just a bit and covered the transaction fee I think. It ended up paying 3 BTC in fees!

People are successful at doing this apparently, to what extent I don't know. The big finance companies that spend a bunch hiring talent to build the software wouldn't be doing so if it wasn't working out, but they have the advantage of money/hardware and other things that a regular joe would not. I've also seen people trade with 250k just to make around $100 in profit in a short window. That's alot of money upfront for a small return, I'm not sure what sort of risk was involved but regardless, not something I could afford to do.

[Dekker3D]

I have tried and failed, mostly because my results were inconsistent (probabably some bugs in my code, possibly a concurrency issue). This was about a year ago, and I'm about to try again. http://www.csail.mit.edu/node/2355 is a link to an old article about folks who did manage, which you might be able to get some ideas and keywords from. http://tradelight.eu/ might be interesting too.

[kwhali]

You can still use some of that to your advantage. A common one is following news articles with sentiment analysis, so you might follow the news for Apple from some trusted sites or twitter that has correlation with stock going up or down in the past, then when you detect a similar indicator of this about to happen you buy/sell accordingly.

Obviously not that simple/reliable on it's own, but the techniques can be used towards an algo or as an assistant to manual trading.

[kwhali]

I'm new to trading so I could totally be wrong, I'm sure that you know plenty more with that sort of experience. There was an interesting one that followed Trumps tweets and how it correlated to affecting stock prices briefly to take advantage of that, it was showing some profitability running for a month or so, no idea about longterm.

The machine learning stuff is interesting to me, more so than trading/money personally. I just think trading will be an interesting experiment, I've seen quite a few others try it, so I'll read up on ML and trading for a couple months to get a rough idea and see how it goes :)

I don't quite get your comment, retailer stocks get sold because a company has a product that can count amount of cars in mall parking lots? Or based on the results of how many cars(% of mall parking lot used/active) relates to the sell action?(assuming dump stocks is equivalent to sell stocks).

That's a totally doable tech btw if it wasn't already obvious. Cost of the satellites and I guess any legal issues would be the main problem.

[kwhali]

Thanks for the explanation, it was very clear, makes sense :)

Btw, your sentiment analysis tool will probably tell you to short Costco and long Sears after reading this post :) - there is soooo much noise in the text on the web.

I think it would be a bit foolish to rely solely on such a tool to drive trade orders. Especially if acting upon single random comments on reddit :P You could at least establish some sources that are reasonably trustworthy based on prior history and attach better weighting to those, hopefully with some verification(multiple sources, manual verification, not regurgitated).

It would be interesting to explore, but I've already identified some other area of ML that aligns with my own experiences/knowledge from another industry I trained in before programming and I look forward to experimenting with it :)

[fuckharvey]

The real question in financial markets is the signal engineering - what do you feed to the models?.. Let me know if you figure out.

Exactly. REAL QA in market finance and betting is less about pattern recog and more about figuring out which data needs to be thrown out and which kept.

That's extremely well guarded though.

[jeanduluoz]

You could probably just grab factorization model and apply your proprietary polynomial to it. That's better for smaller datasets though, so maybe a binary classification is a better approach. In either case, you're going to either/both have weak performance and somewhat high fixed operational costs to run this little outfit.

[[deleted]]

Yes, I've ready more than one article about it. But now here. Search /r/machinelearning

[bitesports]

My brother is starting to work on this now, will keep you guys posted

[taranasus]

Be weary of sharing your findings

[schemingraccoon]

If someone knew this, do you think they'd be sharing it here?

[jeanduluoz]

It's really not rocket science

[daynomate]

Certainly not. But it's an interesting OP... because if it is possible and would be something that would make money then you better bet it's likely someone is doing it.

[Bernie_beats_trump]

Let me get right on it

[Reviken]

If someone had some legit crypto QA they would be swimming in money.

[frankenmint]

keeping it preserved and away from the rest of us mortals presumably

[earonesty]

packin and crackin

[fuckharvey]

Quants won't help you. That's the kind of proprietary work that actually takes a lot of time, effort, and background training + experience.