One of the delightful insights in Good's "Good Thinking" is why collecting data until you achieve significance is fundamentally doomed to fail. I'm pretty sure this isn't a novel perspective, but it's the first I had seen it explained so clearly.
Under a point null hypothesis tail probabilities past some threshold (i.e. p-values) are uniformly distributed no matter the model. Consequently the p-values corresponding to a sequence of increasing measurements will be uniformly distributed marginally at each iteration.
At the same time because the data are growing at each iteration the p-values will be correlated with previous p-values. In other words the sequence of p-values forms a Markov chain (of some order) whose stationary distribution is that uniform distribution.
As you collect more data from the null hypothesis the p-value Markov chain will explore the entire interval [0, 1] and eventually surpass _any_ significance threshold. It's an experimental analogy of the Gambler's ruin -- play long enough and you're guaranteed to lose.
I'm sure I'm missing all kinds of technical details (what's the order of the Markov chain, under what conditions is it ergodic, etc) that people who like talking about Martingales can figure out, but I really like the conceptual picture regardless.
