,
17 tweets,
4 min read
Read on Twitter
Hypothesis testing involves estimating the probability that an observed result would have occurred by chance if the null hypothesis were true.
There are limitations to hypothesis testing:
1- Viewing outcomes as dichotomous (yes/no) rather than viewing them as a continuum
2- Choosing a single cut point for “rejecting” the null hypothesis when that choice is often arbitrary
1- Viewing outcomes as dichotomous (yes/no) rather than viewing them as a continuum
2- Choosing a single cut point for “rejecting” the null hypothesis when that choice is often arbitrary
CIs help answer the question “what is the plausible range within which the actual difference between an experimental and control group lies?“
Using CIs also avoids the yes/no dichotomy of hypothesis testing.
Using CIs also avoids the yes/no dichotomy of hypothesis testing.
The larger the range, the less precision there is, and the less useful the data is.
So how do you get a narrow CI (more precise estimate of true value)?
So how do you get a narrow CI (more precise estimate of true value)?
Intuitively, more confidence can be placed in trials with a larger sample size. But in addition to sample size, event rate is essential in achieving a narrow CI.
Large sample size may mean large number of events (e.g. death or MI) but not always..
Large sample size may mean large number of events (e.g. death or MI) but not always..
Consider 2 studies thats showed a RRR of 50% of an experimental agent vs control.
In study 1, 100 patients were assigned to each group.
In study 2, 1000 patients were assigned to each group
In study 1, 100 patients were assigned to each group.
In study 2, 1000 patients were assigned to each group
When you look at the number of events, you find that study 1 had 25 events in the experimental group and 50 events in control group.
Study 2 had 5 events in experimental group and 10 events in control group.
The RRR is 50% in both, but the CI may be narrower in study 1
Study 2 had 5 events in experimental group and 10 events in control group.
The RRR is 50% in both, but the CI may be narrower in study 1
Along the same lines, studies with few events may produce a CI that indicate a statistically significant result if the absolute difference in events is large, but a small change in events in either group can change that significant result.
Too few events can be misleading.
Too few events can be misleading.
Next, how do you determine if sample size is large enough to estimate effect?
Suppose a trial that is read as “positive” has an absolute risk difference of 1% in favor of the investigational arm. But the CI ranges from 0.9% to 1.9%.
In such a trial, in order to determine if the sample was large enough, look at the lower boundary of the CI (0.9%) - the smallest plausible treatment effect. If that lower boundary is greater than the smallest effect that is considered important, then the sample size is adequate.
On the other hand, in a “negative” trial, look at the upper boundary of the CI - the largest plausible treatment effect.
If that upper boundary is less than the smallest effect that is considered important, then the sample size is adequate.
If that upper boundary is less than the smallest effect that is considered important, then the sample size is adequate.
That means that you cannot conclude lack of benefit in a negative trial. Negative trials fail to demonstrate benefit due to inadequate sample size.
This logic depends on specifying a threshold benefit below which patients will not choose the intervention due to toxicity/burden.
To complete the picture read this bmj.com/content/311/70…
I’ll copy the folks that know much more than me about this @raj_mehta @bogdienache @venkmurthy
And of course, @f2harrell.
And of course, @f2harrell.
