The Meta-Analytic Protocol
Defining the Scope
1. The Research Question (PICO)
A meta-analysis must be precise. We use the PICO framework: * Population: Who are the participants? (e.g., Adults with MDD) * Intervention: What is the treatment? (e.g., CBT) * Comparator: What is the control? (e.g., Wait-list, TAU, Placebo) * Outcome: What is being measured? (e.g., Depression Symptomology)
Critique: “Garbage In, Garbage Out.” Response: Strict inclusion criteria act as the filter.
2. Inclusion/Exclusion Criteria
We must define the “Universe of Studies”.
| Criteria Category | Examples of Inclusion | Examples of Exclusion |
|---|---|---|
| Participants | Age > 18; Primary dx of Depression | Bipolar; Psychosis; Substance Use |
| Intervention | Manualized CBT; > 8 sessions | Bibliotherapy; Unstructured care |
| Design | RCT (Randomized Controlled Trial) | Pre-Post only; Quasi-experimental |
| Outcome | BDI; HAM-D; QIDS (Continuous scales) | Dichotomous “Success/Fail” only |
| Language | English, German, Spanish | Untranslatable |
| Timeframe | 1980 - Present | Pre-1980 (Outdated methods) |
3. Literature Search Strategy
Where do we find the studies? 1. Databases: PubMed, PsycINFO, Embase, Cochrane Central. 2. Grey Literature: Dissertations (ProQuest), Conference Abstracts. (Critical to avoid Publication Bias!) 3. Forward/Backward Chaining: Checking references of included studies.
The Importance of Grey Literature
Publication bias impacts validity. We must search for “Grey Literature”:
- Dissertations/Theses: often contain null results (ProQuest).
- Conference Abstracts: Early data not yet published.
- Government/Technical Reports: Non-academic sources.
- Personal Communication: Emailing authors for missing data.
4. The Coding Manual
Coding Reliability
If multiple coders are used, we must assess Inter-Rater Reliability (IRR).
- Cohen’s Kappa (\(\kappa\)): For categorical variables (e.g., Diagnosis type).
- Intraclass Correlation (ICC): For continuous variables (e.g., Effect sizes).
- Standard: Double-code at least 20% of studies. \(\kappa > 0.80\) is excellent.
Once studies are retrieved, we “extract” the data. This requires a Coding Dictionary.
Descriptors (Moderators)
- Study Level: Year, Country, Funding Source.
- Sample Level: Mean Age, % Female, Ethnicity, Comorbidity.
- Treatment Level: Duration (weeks), Format (Group/Individual), Provider (PhD/MD).
- Quality: Risk of Bias (Randomization, Blinding, Attrition).
Effect Size Data
For every outcome, we need: 1. Sample Size: \(N_{tx}\), \(N_{ctrl}\). 2. Means: \(M_{tx}\), \(M_{ctrl}\). 3. SDs: \(S_{tx}\), \(S_{ctrl}\). 4. Test Statistics: \(t\), \(F\), \(\chi^2\), exact \(p\)-values.
Rule: Extract everything. If precise stats are missing, extract \(p\)-values or \(N\) to impute later using
compute.es.