Publication Bias
The File Drawer Problem
The File Drawer Problem
“For every significant study published, there are 10 failed studies sitting in a file drawer.” – Rosenthal (1979)
If we only meta-analyze published studies (which tend to be significant), we overestimate the true effect size.
1. Visualizing Bias: The Funnel Plot
A Funnel Plot displays Effect Size (X-axis) vs. Precision (Y-axis).
- Symmetry: Indicates no bias (small studies scatter randomly L/R).
- Asymmetry (Missing Corner): Indicates bias (small, non-significant studies are missing).
Interactive Funnel Plot
We will simulate a “Biased” literature where small, non-significant studies were “censored” (never published).
2. Statistical Tests: Egger’s Regression
We can formally test for asymmetry using a regression.
\[ EffectSize = \beta_0 + \beta_1(StandardError) \]
If \(\beta_1\) is significant, small studies have different effects than large ones -> Bias.
3. The Solution: Trim-and-Fill
“The trim and fill method estimates the number of missing null studies from meta-analysis.”
- Logic: It iteratively removes (trims) the most extreme small studies from the positive side of the funnel plot, re-estimates the effect size, and then adds back (fills) the original studies plus their mirror images on the negative side.
- Result: A symmetric funnel plot and an adjusted (often lower) Effect Size estimate.
4. Fail Safe N
Also called the File Drawer Analysis (Rosenthal, 1979).
Definition: “The number of null studies that have to be added for the overall effect to be reduced to Non-Significant.”
- Interpretation: If Fail Safe N is large (e.g., > 100), the results are robust. If small (e.g., < 10), the results are fragile.