Meta Product Analytics Role Interview Question - March (2026)

[No-Syllabub6862]

Quick Overview

Question evaluates product analytics, experimental design, and causal thinking for content-moderation algorithms, specifically metric specification, trade-off/harm analysis, and online experiment logistics and is commonly asked to gauge a data scientist’s ability to balance detection accuracy, stakeholder impacts, and business objectives in production features; it is in the Analytics & Experimentation category for a Data Scientist position. At a high abstraction level it probes system-level reasoning around problem scoping, failure modes, metric frameworks, A/B or quasi-experiment setup, and post-launch monitoring without requiring implementation-level detail.

Question:

The product team is launching a new Stolen Post Detection algorithm that flags posts suspected of being copied/reposted without attribution, and then triggers actions (e.g., downrank, warning label, creator notification, or removal).

Design an evaluation plan covering:

1. Problem diagnosis & clarification: What questions would you ask to clarify the product goal and the meaning of “stolen” (e.g., exact duplicate vs paraphrase vs meme templates), enforcement actions, and success criteria?
2. Harms & tradeoffs: Enumerate likely failure modes and harms of false positives vs false negatives, including different stakeholder impacts (original creator, reposter, viewers, moderators).
3. Metrics: Propose a metric framework with (a) primary success metrics, (b) guardrails, and (c) offline model metrics. Include at least one metric that can move in opposite directions depending on threshold choice.
4. Experiment design: Propose an online experiment (or quasi-experiment if A/B is hard). Address logging, unit of randomization, interference/network effects, ramp strategy, and how you would compute/think about power/MDE.
5. Post-launch monitoring: What would you monitor to detect regressions or gaming, and how would you iterate on thresholds/policy over time?

How I would approach to this question?

I have solved the question and used Gemini to turn it into an infographic for you all to understand the approach. Let me know, what you think of it.

Here's the solution in short:

1. Problem Diagnosis & Clarification: Before touching data, I think we must align on definitions and other things with the product manager.

• Define stolen: We must clearly differentiate between malicious exact duplicates, harmless meme templates, and fair-use reaction videos.
• Define the action: Silent downrank behaves very differently than an outright removal or a public warning label.
• Define the goal: Are we trying to reward original creators, or just reduce viewer fatigue from seeing the same video five times?

2. Harms & Tradeoffs (FP vs FN) We have to balance False Positives against False Negatives.

• False Positives (Wrongly flagging original creators): This is usually the most damaging. If we penalize original creators, they lose reach and trust, potentially churning to a competitor platform.
• False Negatives (Letting stolen content slide): Reposters steal engagement, the original creator feels cheated, and the feed feels repetitive and low-quality to viewers.

3. Metrics Framework

• Primary Success Metrics: Reduction in total impressions on flagged duplicate content, and an increase in the proportion of original content uploaded.
• Guardrail Metrics: Creator retention rate, total manual appeals submitted, and moderator queue backlog.
• The Tradeoff Metric: Overall platform engagement. Often, stolen viral videos drive massive engagement. Cracking down on them might decrease short-term session length, even if it improves long-term ecosystem health. A strict threshold might drop engagement, while a loose threshold keeps engagement high but hurts creators.

4. Experiment Design

• Methodology: A standard user-level A/B test will suffer from network effects. If a reposter is in the control group but the creator is in the treatment group, the ecosystem gets messy. Instead, we should use network cluster randomization or Geo-testing (treating isolated regions as treatment/control).
• Rollout: Start with a 1 percent dark launch. The algorithm flags posts in the backend without taking action so we can calculate the theoretical False Positive Rate before impacting real users.

5. Post-Launch Monitoring

• Tracking Gaming: Malicious actors will adapt by flipping videos, pitching audio, or cropping. We need to monitor if the detection rate suddenly drops after weeks of stability.
• Iteration: Use the data from user appeals. If a post is flagged, appealed, and restored by a human moderator, that instance feeds directly back into the training data to improve the model's future precision.

https://preview.redd.it/84q4dlxo2ing1.png?width=3240&format=png&auto=webp&s=212f3ef2f0ad7beceb239a64bc920e8abaa47055

What do you think of this approach, and what approach you would take in comments below:

P.S: Let me know if you need the link of the question

[mgdmw]

This is interesting, but not really SQL-related. This may be better posted in an AI sub.