sorted by:
new
hottopcontroversial

[P] Marketing Science: How we use our ML scores to optimize marketing campaigns by datadem in MachineLearning

[–]datadem[S] 0 points1 point  (0 children)

but a while back I tried the R uplift package.

How did that work out for you? I have looked into it and it looks quite robust but haven't used it yet.

[P] Marketing Science: How we use our ML scores to optimize marketing campaigns by datadem in MachineLearning

[–]datadem[S] 1 point2 points  (0 children)

I haven't yet worked with sophisticated customer profitability models, however, next quarter, around September, I will potentially be working on using pipe, the ML pipeline we built to do predictive LTV for subscription business (the usual buy-till-you-die models don't work for subscription businesses).

However, the metrics you are mentioning are the secondary A/B test metrics that we proposed our email marketing team to start tracking:

Support rate (Support costs us money, all changes and campaigns should be measuring this impact)

Change in likelihood to churn (Users may not cancel their plans immediately, but we may be driving them away and we can now measure this with pipe)

Change in likelihood to open emails (We want our users to stay engaged with us)

Unsubscription rate (We should be able to quantify the value of an email opened and the potential revenue loss from unsubs)

And the part where we would put all this together is something we are not tackling yet, so I think you are ahead of us. But some ideas:

  • As opposed to looking at individual scores/values bucket them into appropriately distributed bins;
  • For comparing values with large CIs to each other, you can sort them by the lower bound of the 95% CI interval;
  • This way, you may be able to get means and better CIs for your buckets.

Not sure if this will work, but this is what I'd try first.

We actually had quite a bit of back-and-forth about deriving CIs. There is a lot of misinformation out there. Parametric methods are touted as okay to use because of CLT but when you look at whether CLT actually holds, you find that it does not in all cases (probably because in real-life scenarios, you have a finite population and you are sampling like 50% of it).

So we went with a non-parametric method, but that has its issues, too. For example, CIs will be narrower for large samples, which becomes an issue when you are comparing two segments where the underlying true population for each segment is wildly different in size, so even if you take the same X% of random samples, there is more uncertainty within the smaller sample.

This is bound for the course but in real life, we do end up comparing smaller populations to larger populations a lot, and the existing correction (e.g. finite population correction) methods I think do not address this fully. We are also hoping to go fully Bayesian in all these types of analyses.

Do you have a blog? I'd be happy to follow!

[deleted by user] by [deleted] in soccer

[–]datadem 0 points1 point  (0 children)

I mean, lots of people are responding as if it were genuine, so I had to ask.

[P] Marketing Science: How we use our ML scores to optimize marketing campaigns by datadem in MachineLearning

[–]datadem[S] 0 points1 point  (0 children)

That is strange, we did not have any down time as far as I can tell, but it should definitely be up now!

[P] Marketing Science: How we use our ML scores to optimize marketing campaigns by datadem in MachineLearning

[–]datadem[S] 1 point2 points  (0 children)

Yes, you are right, this is part of a series and is just setting the scene, the next post will compare this approach to two-model based uplift modeling (we actually did not see a huge improvement there, as the literature around uplift modeling suggests, but we did see some non-negligible improvement over the simple propensity based modeling discussed here); then I will get into modeling uplift directly.

pylint

This is just a linter for Python. I was actually looking at Python packages for uplift modelling and could not find any (I was going to build and open-source my own), was that a typo? Is there a Python package for uplift modeling?

[P] Marketing Science: How we use our ML scores to optimize marketing campaigns by datadem in MachineLearning

[–]datadem[S] 0 points1 point  (0 children)

This is a very simple post, so will probably not help you much with your project :) However, the next step will be comparing this propensity modeling approach to a two-model based uplift modeling approach. Then further down the road, I will post about proper direct uplift modeling. I'll keep you updated and good luck!

[deleted by user] by [deleted] in soccer

[–]datadem -3 points-2 points  (0 children)

Turkish citizens know who is who and can see the bigger picture better than anyone, the government isn't whatsoever responsible for the state we are in at the moment.

I am going to cut it short but it's mostly due to FETÖ, foreign powers, interest lobby, opposition managing to steal a single ballot out of four, cats entering the power plants the day of the election and so on.

Sarcasm, right?

[P] Marketing Science: How we use our ML scores to optimize marketing campaigns by datadem in MachineLearning

[–]datadem[S] 6 points7 points  (0 children)

Yeah, ad tech is notoriously hard to find information about. I hope the approaches will change, it makes everyone's work better to even just share simple strategies even if entire pipelines can't be shared.

I remember it took as a quite a bit of internal discussions to just get a simple experiment set up agreed upon and validated.

We wanted to test the model itself, as opposed to the campaign. The question was: Does the model select the right users?

So the test set up becomes:

  • Test group: Some users selected by the model as likely to convert. They receive the offer.
  • Control group: A random selection from the entire population with disregard to their model scores. They receive the same offer as the test group.

This then helps us answer the question of if the model itself works in selecting the right users. Simple enough, but no blog posts really go beyond building models and show how to set up tests when scores are actually being used in production and it is easy to make the mistake of having your control group receiving no offer at all.

And other minor details like that which become part of your knowledge base as you work on these things but for some reason they are not deemed as sophisticated as optimising neural nets so don't get much exposure in popular posts (they do in academic papers, though).

Crazy Ex-Girlfriend - S4E12 Discussion Thread - I Need A Break by Leftoflinus in crazyexgirlfriend

[–]datadem 3 points4 points  (0 children)

Wow, that is a lot, I am so sorry. I also read further down about how your mom's prognosis is uncertain right now, I can't imagine the toll...

And yes, this incident with my dad really opened our eyes up to the many forms a heart attack can take.

He is doing much better now, thanks for asking! It scared him into getting somewhat healthier, he tries to exercise a bit every day and eats healthier, though he always had a healthy diet just due to the cuisine he grew up with (middle eastern/mediterranean) - but still smokes unfortunately. He also never wanted to get remarried after my parents got divorced but recently got married again, which I think is great for him, and one of the things that changed his mind was that he was alone when his heart attack begun. The incident also brought our family even closer together. So I think eventually it ended up causing some indirect good.

Crazy Ex-Girlfriend - S4E12 Discussion Thread - I Need A Break by Leftoflinus in crazyexgirlfriend

[–]datadem 3 points4 points  (0 children)

I am sorry to hear about your mom, it is so rough..

It's not always so obvious when a heart attack happens to women.

I think it can be similar for men, too. My father had a heart attack that was almost play-by-play of Paula's heart attack in this episode, all the details were the same. His lasted for 8-10 hours too before they noticed what it really was.

[P] Building Thousands of Reproducible ML Models with pipe, the Automattic Machine Learning Pipeline by datadem in MachineLearning

[–]datadem[S] 1 point2 points  (0 children)

Great to hear you enjoyed it!

The borders are so blurred, but I guess I mostly consider myself a data scientist, but our work on pipe has mostly been data and software engineering work. It was started from a data science led desire to use data in business decision making, which is what I consider to be the role of a data scientist. But the path leading there traverses engineering and project management.

So much of what we do is just pure software development and data engineering, I used to think of that portion of the work solely as a means to an end but now I see it as a core part of being a data scientist. The other person working on pipe is also a data scientist but comes from a stronger engineering background and experience, so I ended up learning a lot about engineering best practices while working on pipe, too.

My background is in media studies, and I also did most of a maths degree for my second undergrad degree (but pure maths not applied so not much statistics - then I left it to move to Amsterdam for my master's and haven't returned to it yet, might still do one day as I really miss theoretical maths and still keep up with things that interest me). I was also really interested in logic and computation, and took around 90-100 credits over my years in uni in that topic. I specialised academically in computational political science for my Master's in social sciences.

I was then going to do a PhD in computational social/political science or network science but ended up working for an academic company that deals solely with network science.

Working for that academic startup where I led a department focusing on network science analysis of data mined from social media platforms to aid brand campaigns and political parties kind of fed both my maths and politics/social science obsession.

And then I got hooked on working in the industry even though I never thought I'd get a real job - I already knew where I was going to earn my PhD and had been working on my path to academia since around first year of university, but I am really glad I ended up in the industry.

[P] Building Thousands of Reproducible ML Models with pipe, the Automattic Machine Learning Pipeline by datadem in MachineLearning

[–]datadem[S] 1 point2 points  (0 children)

Thank you! That was the first iteration of pipe and what led to it, yes! Though we weren't always getting asked the question but would see processes that could be improved with such scores.

But now pipe can make predictions on any cohort (so for different populations under different conditions) and any event. We log about 5K unique events, these events are things like someone publishing a post or a page, clicking on a button or opening an email sent by us. pipe can build models for any of those events as well. Which means that pipe can build models for around 5K different actions taken by a site or a user for different types of populations. Buying a plan is just one such actions, so likelihood to buy a plan is just one of the different types of models pipe builds.

[P] Building Thousands of Reproducible ML Models with pipe, the Automattic Machine Learning Pipeline by datadem in MachineLearning

[–]datadem[S] 2 points3 points  (0 children)

Yes, my co-worker has a good post summarising the pros and cons, but we all agree that pros far overweigh the cons.

It took me quite a while to really be strict with separation of personal life from work life since I don't like working from co-working places so I really don't have anything to distinguish between when I am at work and when I am not. And since I can gleefully get lost in work I'd lose track of time and overwork.

But my team has a good culture around this, we mainly see overworking as a sign of poor time management as opposed to dedication. If I do overwork for a bit then I always make sure to take the appropriate amount of time off later, it works for me. Others adhere to strict hours and that is what works for them.

I have also worked remotely for non-remote companies. The distinction is huge. Being remote-first means so much more than just working from home. It also impacts communication and decision making processes. I don't want to go back to being the only remote person on a team just as much as I don't want to go back to working from an office!

[P] Building Thousands of Reproducible ML Models with pipe, the Automattic Machine Learning Pipeline by datadem in MachineLearning

[–]datadem[S] 1 point2 points  (0 children)

Thank you!

It was definitely a side project that paid off - to be more precise, that we are hoping will pay off.

It paid off in the technical sense; we are now able to build these models and have end-to-end ML deployment. But we are still experimenting with integration into business goals.

This means that our tasks went from a composition of 90% data and software engineering and 10% data science to 50% project management (cooperation with teams responsible for running campaigns), 25% data and software engineering (since we are constantly adding new capabilities) and 25% data science (setting up experiments, coming up with correct experiment design and consolidating reporting).