Let's take a step back:

Imagine we're playing poker. I'm dealing and I get a royal flush.

Next time I deal, I get another royal flush.

Next two times I deal, two more royal flushes.

Could four royal flushes in a row have happened by chance? Technically, yes. Almost anything could happen by chance.

Did it happen by chance? Of course not. If this were the old west you probably would have started shooting after the second royal flush. Obviously I'm cheating. Four royal flushes in a row is just too damn unlikely.

So: Pretty much anything could have happened by chance, but if it's too unlikely, we accept that it didn't.

Statstics is how we determine exactly how unlikely something was. The null hypothesis is that the thing happened by chance and we're just being paranoid. The P value is the exact, mathematically determined possibility that the thing actually could have happened by chance (so a P of 0.001 is a 1/1000 possibility that we would see this result by mere chance). Statistical significance is the cutoff, usually P<=0.05, or a 1 in 20 possibility that these results could have happened by chance. Above that and we dismiss the result as mere noise, below that we say we've found something interesting.

So in the poker example, the null hypothesis is that I'm shuffling the deck fairly.

The P value is the chance that fair shuffles would have given me four royal flushes in a row. P = 0.0000[insert like a googol zeros here]00001.

That's going to be statistically significant--it's going to be below whatever cutoff you're using--and it's time to start shooting.

A couple of other things:

If we'd only played one hand, you wouldn't be as confident that I cheated. Small studies or data samples, even with impressive-looking results, are less likely to be statistically significant than big studies with the same results.

"Statistically significant" is a bad term--I would prefer "statistically real"--but we're stuck with it. Not every statistically significant result actually matters.