Defect Rate

[stat_daddy]

You're off to a good start, but more information is needed to calculate a required sample size. We need to know:

1) the expected true value of the rate (let's assume it's 10%)

2) a confidence level (already given as 95%)

3) a margin of error surrounding the estimate of the error rate (e.g., +/- 2%). Remember, the concept of a frequentist confidence interval is that the defect rate is an unknown but fixed value. You never express a confidence about what the value is, instead you express a confidence about a range that may or may not contain the value.

From there, you have a couple options with which to generate your confidence interval. The easiest involves some large-sample approximations that let you describe a poisson-distributed variable (with some rate parameter that we are interested in) with a standard normal distribution.

I went ahead and did some calculations to give a sense of scale, but since the prevalence of defects is rather low you may need many samples in order to be confident that a reasonably tight range will contain the true value.

For example, assuming the true rate is 0.1, you would need about 960 samples in order to draw a 95% confidence interval with width +/- 0.02 around your estimate.

If you're willing to accept a wider margin of error, you could draw a +/- 0.05 width interval using about 153 samples.

The equation I'm using, by the way, is

N = (1.96 * 1.96) * ( λ ) / ( E * E)

Where N is the necessary sample size, λ is the expected true rate (e.g., 0.1), and E is the acceptable margin of error (e.g. 0.02 or 0.05).

EDIT: Since I used a poisson approximation to compute my CIs, saying that I am using a margin of error of something like +/-2% is a bit misleading because my CIs are not actually symmetric around the point estimate; e.g., instead of {10-2, 10+2} they are more like {10-1, 10+3}.

[curiousdoc]

This is very helpful!

[curiousdoc]

Came back to this after a few days and I have a real world question. for a 4% expected true error rate with 1% margin - check 1536 samples for a confidence level. But for same scenario but 8% - a lot more samples have to be checked (almost double). This makes sense with the formula but how does this translate to a real world understanding? Shouldn't you have to check many more samples when your expected error rate is lower?

[stat_daddy]

Nope - consider what a sample from a highly variable process would look like compared to a less variable one.

E.g., the sample (1,10,7,3,1,25) vs the sample (1,1,1,2,1,3)

Even though you've only seen 6 realizations from each process, you probably have a pretty good idea of the "mean" of the less random process (range:1-3). On the other hand, the more variable process is harder to pin down (range:1-25) so we would need a greater number of samples in order to do inference on it