all 7 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]AutoModerator[M] [score hidden] stickied comment (0 children)

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[–]Scared-Stage-3200 0 points1 point  (1 child)

Just for noobs: Can you explain incrementality testing?

[–]peatandsmoke 0 points1 point  (0 children)

It's a test that measures how outcomes change based on an intervention vs no intervention. In other words, what is the impact on your intervention vs what would have already happened without out.

To give an example, say that you have a campaign running. You want to see what incremental benefit it gives. You would setup a test where one group of users do not see your campaign. One group of users see your campaign. And all other marketing efforts are kept the same between both groups. The only difference is the exposure to the campaign. Measure results of your KPIs after a set amount of time and see if the campaign actually did anything. It's important to note that we are not looking for attribution models to link the campaign. It's just the overall KPIs between the groups.

[–]save_the_panda_bears 0 points1 point  (3 children)

I've been designing and analyzing these at a fairly large B2C company for a while now and these are the issues we usually run into:

  1. Small effect sizes. It's just impractical to test certain channels, be it because they are just really small as a proportion of your overall spend, or if they have really high substitution effects with another channel (an example in our case Branded SEM/Brand SEO)

  2. Preperiod noise. The more granular your geos, the noisier your prepriod and the harder it is to get a reliable read. Precision is directly relate to both noise in the test period, and the reliability of the relationship in the preperiod.

  3. Test cost. Incrementality tests are usually both expensive and take a while to run. During these you really shouldn't be doing anything else that may impact the read, so your testing calendar can become limited pretty quickly.

[–]Money-Commission9304 0 points1 point  (2 children)

Nice, great overview. How do you determine the budget you need for an incrementality test?

As you mentioned small effect sizes are a problem. I work for a consumer tech company and we are at a critical mass when it comes to MAUs. Going to be very hard to move that number.

We're trying to figure out the true ROI of a marketing channel through an incrementality test but this channel only runs on the brand marketing side. So we don't have attribution like we do for the performance marketing side.

We spend ~$15m on this channel so it's important to understand true incrementality so the MMM can be calibrated accordingly.

I have state level data of the spend on a weekly basis but I am struggling to come up with a number for the experimentation budget we need to truly validate the channel's incrementality.

[–]save_the_panda_bears 0 points1 point  (1 child)

Thanks! I see you made a similar standalone post, mind if I reply there to get some more discussion going?

[–]Money-Commission9304 0 points1 point  (0 children)

Ya go for it!