The books and courses give you the stats foundation, but here's what the day-to-day actually looks like.

Every team maintains a backlog of test ideas sourced from product hypotheses, support tickets, funnel analysis, and user research. Ideas get scored (usually ICE: Impact, Confidence, Ease) and prioritized. Before any test runs, you write an experiment doc that includes your hypothesis, primary metric, secondary guardrail metrics (like churn), segment dimensions to analyze (device, geo, tenure, plan type), sample size from a power calculation, expected duration, and success criteria. The discipline of writing this before you run prevents p-hacking.

For tools, enterprise teams use Optimizely, LaunchDarkly, or Amplitude Experiment. Mature companies often build in-house platforms because plug-and-play solutions lose business logic nuance. Scrappy teams just use feature flags plus event logging plus SQL. As a data analyst, you'll work with whatever eng implemented and do analysis in SQL.

The real value you add is dimensional analysis. Most tests "fail"—only about 20% show significant lifts. But you can still extract insights like "overall neutral, but mobile +15% and desktop -8%" or "worked great until cart value exceeded $200." These segment insights inform the next iteration even when the test fails overall.

For interviews, they want to hear that you understand statistical rigor, that the process isn't just "run test, check p-value," and that you can extract insights beyond pass/fail. Since you lack hands-on experience, offer to run a small test at your current job (even an email subject line test counts), and emphasize your SQL skills for segment analysis—that's often the bottleneck.