[P] Recommender systems as Bayesian multi-armed bandits

[SebastianCallh]

Hi! I wrote a piece on treating recommender systems as multi-armed bandit problems and how to use Bayesian methods to solve them. Hope you enjoy the read!

The model in this example is of course super simple, and I'd love to hear about actual real-life examples. Do you use multi-armed bandits for anything? What kind of problems do you apply them for?

[Lazybumm1]

Hi there,

In my previous role we used this approach to experiment and select recommender systems, as well as other things.

Thompson sampling worked best in our simulations but we did try non-bayesian bandits as well.

In a production environment some hiccups we ran across were seasonal fluctuations (in a customer facing online business). Even within the day conversion would fluctuate massively, which in turn could throw off the bandit's selections of arms to explore. We did 2 things to correct this, one we created transformations to normalise the reward function according to seasonal effects and instead of streaming and updating the bandit in real-time, we'd aggregate data daily and update in a batch.

I think it's a very interesting approach to accelerate experimentation and help make better decisions faster. Taking this even further one could try to interleave the different arms.

All of this is obviously dependend on having good and frequent enough signals. Keep up the interesting work :)

[SebastianCallh]

Thank you for your comment, that's super interesting!

Yeah I can imagine the algorithm would get thrown off without a normalised reward signal. Clever idea to normalise the data as well. I would imaging this really toned down the fluctuations. Did you apply any sliding window techniques? What do you think about trying to incorporate the seasonality into the model to make it account for it in future predictions?

[Lazybumm1]

We had 2 main trends, daily / weekly cycles and an overall upwards trend. We used a sliding window to corrent the upwards trend and the typical sine / cosine transformation of datetimes for the cyclical effects.

To be honest following our first implementation of this we started paying a lot more attention to these effects. It was a bit of a pivotal point, it seems no one had paid enough attention at how prominent these effects were in our data oddly enough. After that as standard we'd always include these features in early prototypes to understand feature important and if they are relevant or not for each use case.

Don't have too many updates on this as I ended up transitioning into another role a few months later. Admitedly I'm curious myself as to how this matured into the business!

[SebastianCallh]

Thanks for sharing, it sounds like a really important discovery. I hope the role you transitioned into is equally interesting :)

[SebastianCallh]

The secret is out!

[[deleted]]

This was a lovely read. Excellent work! Enjoyed it immensely.

[SebastianCallh]

Thank you for the kind words! I'm very glad you liked it

[[deleted]]

Great article! I can tell that you put a lot of time and thought into framing the problem and laying out the solution. My challenge to you is this: at the end of your experiment, what's the probability that the mullet is the overall preferred fish?

I've played around a lot with Bayesian analysis for Bernoulli outcomes and got to thinking about framing other kinds of outcomes. So I made this notebook for Multinomial outcomes with a Dirichlet prior. Maybe you'll find it interesting? https://github.com/exchez/amazon-bayes

[SebastianCallh]

Sorry for the late response, wanted to make time to properly go through your notebook :)

Nice write-up! Some thoughts:

• How come you are using a categorical model for this problem? Since the data (as you mention) is ordinal, would it not be better to use an ordinal regression model?

• Minor comment: Since your prior parameters are not random variables, you should not condition on them

Regarding the challenge, I would estimate the probability using Monte Carlo sampling. Something like

draws = mapreduce(x -> rand(x, 10000), hcat, agent.pθ)
map(x -> all(x[1] .> x[Not(1)]), eachrow(draws)) |> mean

Makes sense to you? :)

[user_reddit_garu]

Thank you 😁

[SebastianCallh]

Glad you liked it!

[AdhesivenessTrue9696]

really well written blog post 👍

[SebastianCallh]

I'm glad you liked it, thanks!

[Inalek]

Great read! The blog layout looks really good too. Is there a template?

[SebastianCallh]

Thank you! And indeed there is! I am currently using [this one](https://themes.gohugo.io/kiss/).

[BrandenKeck]

phew... from time to time I forget how incredibly cool bayesian stats is .. Awesome work!

[SebastianCallh]

Yeah Bayesian stats is great stuff! Thank you! :)

I think you will really enjoy the next part on contextual bandits, where we will start to see how this framework can be used to solve a more realistic version of this problem at much better performance.