Meta-analysis is the statistical combination of effect sizes from separate studies that address the same question, so that a pooled estimate, its precision and the spread of effects across studies can be examined together. In educational leadership it is used to answer questions that no single survey can settle: how strongly principals’ leadership is associated with teacher commitment or trust, whether instructional leadership matters more for pupil outcomes than transformational leadership, or whether a relationship holds across school levels and countries. A meta-analysis is usually one part of a systematic review. The review is the whole process of framing a question, searching, screening and appraising studies by explicit methods; the meta-analysis is the quantitative synthesis that may, or may not, follow. PRISMA 2020 treats them this way, and its guidance applies to reviews with and without a statistical synthesis (Page et al., 2021).
When to use it
Meta-analysis makes sense when enough quantitative studies exist on a well-defined relationship, when they report statistics that can be turned into a common effect size, and when the studies are similar enough in constructs and designs that averaging them answers a meaningful question. It is a poor choice when studies measure quite different things under the same label, when only a handful exist, or when the evidence is mainly qualitative. In those cases a systematic review with narrative synthesis is more honest.
Most leadership meta-analyses pool correlational, single-source survey data. The pooled value then describes how strongly teachers’ perceptions of their principal are associated with other perceptions, not the causal effect of leadership. A well-known exception in design, though not in the causal limits, is Robinson, Lloyd and Rowe (2008), who meta-analysed 27 published studies linking leadership to student outcomes. In their comparison of leadership types, based on 22 of those studies, the mean effect size for instructional leadership (0.42) was three to four times that for transformational leadership (0.11), and in a second analysis of 12 studies the dimension “promoting and participating in teacher learning and development” showed the largest mean effect size (0.84, from six studies). The authors themselves cautioned that the instructional leadership estimates varied widely and that most transformational leadership studies measured social rather than academic outcomes.
Key concepts
| Concept | What it means | Practical note |
|---|---|---|
| Effect size | A common metric for the strength of a relationship or difference | Correlations (r) dominate leadership research; group comparisons use d or g |
| Hedges’ g | Standardised mean difference with a small-sample correction | Cohen’s d slightly overstates effects in small samples; g corrects this |
| Fisher’s z | Transformation of r used for pooling | Back-transform pooled z to r before interpreting |
| Fixed-effect model | Assumes one true effect shared by all studies | Rarely plausible across schools, regions and instruments |
| Random-effects model | Assumes true effects vary across studies | The usual default in leadership research |
| Q statistic | Test of whether observed variation exceeds sampling error | Low power with few studies, very sensitive with many |
| I² | Share of observed variation attributable to true heterogeneity rather than chance | A proportion, not an absolute measure of spread |
| Prediction interval | Range in which the true effect of a new comparable study is likely to fall | Answers the question I² is often wrongly used for |
Step by step
Register a protocol and frame the question
Write down the population (for example teachers in primary and secondary schools), the predictor (a named leadership model), the outcomes, eligible designs and languages before you search. PRISMA 2020 asks authors to say where a protocol can be accessed and to explain any changes made to it (Page et al., 2021).
Search systematically
International sources usually include ERIC, Web of Science, Scopus and ProQuest Dissertations and Theses. Turkish sources matter because much of the country’s leadership research appears in graduate theses and national journals. The Council of Higher Education (YÖK) National Thesis Center lists theses from Turkish universities; full texts can be downloaded only for theses whose access has been permitted, so some records give little more than an abstract. TR Dizin, run by TÜBİTAK ULAKBİM, is a national index of scientific journals selected against published evaluation criteria, with records searchable from 2000 onwards. DergiPark, also an ULAKBİM service, hosts many Turkish journals but is a publishing platform rather than an index. Search in both Turkish and English (for example “dağıtımcı liderlik” alongside “distributed leadership”) and report the full strategy for every source, as PRISMA 2020 item 7 recommends.
Screen against inclusion criteria
Typical criteria cover the construct and the instrument, the sample (teachers, principals, students), the design, the statistics needed to compute an effect size, publication type and language. Two people should screen and code independently, with agreement reported. Decide in advance what to do when a thesis and a later article report the same sample: include it once.
Compute effect sizes
For correlations, convert r to Fisher’s z, pool, and convert back. For group comparisons, use Hedges’ g. When studies report different metrics, standard formulas convert between them; with equal groups, d is approximately 2r divided by the square root of (1 − r²). Borenstein et al. (2021) give the formulas and the assumptions behind them. Conversions are approximations, so record which effect sizes were converted and check whether results change without them.
Choose the model
The fixed-effect model estimates a single common effect; the random-effects model estimates the mean of a distribution of true effects. Studies of school leadership differ in country, school level, instruments and sampling, so random effects is usually the defensible choice, and the decision should rest on that reasoning rather than on a significant Q test (Borenstein et al., 2021).
Examine heterogeneity
Report Q, τ² (the between-study variance), I² and a prediction interval. Higgins et al. (2003) tentatively labelled I² values of 25%, 50% and 75% as low, moderate and high, while warning that such labels do not suit every situation. Borenstein et al. (2017) go further: I² tells you what proportion of observed variance is real, not how widely true effects range, so an I² of 80% can describe effects that are all moderately positive or effects that change sign. The prediction interval answers that second question directly.
Test moderators
Subgroup analysis (for example primary versus secondary schools) and meta-regression (for example publication year) can explain heterogeneity. Specify moderators in advance, keep their number modest relative to the number of studies, and treat findings as observational: studies differ on many characteristics at once.
Assess publication bias, with caution
Funnel plots, Egger’s regression test (Egger et al., 1997) and trim-and-fill (Duval & Tweedie, 2000) are standard. Each has limits. Funnel asymmetry can arise from true heterogeneity, poor study quality or chance, not only from unpublished studies, and asymmetry tests have too little power with fewer than about 10 studies (Sterne et al., 2011). Trim-and-fill performs poorly when between-study heterogeneity is substantial (Peters et al., 2007), so its adjusted estimate is better reported as a sensitivity analysis than as a corrected result. Including unpublished theses, which Turkish databases make relatively easy, is itself a partial safeguard.
Handle dependent effect sizes
Leadership studies often report several correlations from one sample, such as leadership with each sub-dimension of commitment. Treating these as independent overstates precision. Options include averaging within studies, multilevel (three-level) models that separate within- and between-study variance, and robust variance estimation (Hedges et al., 2010). Pustejovsky and Tipton (2022) describe working models that combine the two approaches.
Software
Comprehensive Meta-Analysis (CMA) is commercial software from Biostat, widely used in Turkish theses. The free metafor package for R (Viechtbauer, 2010) fits fixed- and random-effects models and meta-regression, draws forest and funnel plots, and runs publication bias tests; its rma.mv function fits multilevel models, and the clubSandwich package adds small-sample-corrected robust standard errors. JASP offers a menu-driven meta-analysis module that works as a graphical interface to metafor, including multilevel models, funnel plots, funnel-asymmetry tests and trim-and-fill.
What to report
- The PRISMA 2020 flow diagram and checklist, protocol or registration details, and full search strategies for each database, including Turkish sources.
- Inclusion criteria, coding procedures and inter-coder agreement.
- The effect size metric, any conversions, and how dependent effects were handled.
- The model, the τ² estimator, the pooled estimate with its 95% confidence interval, and a prediction interval.
- Q, τ² and I², with a forest plot.
- Moderator analyses, flagged as planned or exploratory.
- Publication bias analyses, with their limitations, and sensitivity analyses.
- Software and version.
Common mistakes
- Choosing the fixed-effect model because Q was non-significant, or switching models after seeing results.
- Reporting pooled Fisher’s z values as though they were correlations. A pooled z of 0.90 corresponds to an r of about .72.
- Reading I² as the size of heterogeneity and omitting the prediction interval.
- Counting a thesis and its derived article as two studies, or entering several correlations from one sample as independent.
- Reporting Rosenthal’s fail-safe N as evidence that bias is absent; Borenstein et al. (2021) explain why it answers the wrong question.
- Describing correlational pooled estimates as “effects” of leadership without discussing common-method variance.
Examples from Turkish research
Turkey has produced a large number of education meta-analyses, helped by an extensive, searchable thesis archive and by the volume of correlational survey research on school leadership. Many of these studies used CMA with random-effects models and drew heavily on graduate theses, which makes duplicate samples and same-source bias recurring issues to check.
- Karadağ, Bektaş, Çoğaltay and Yalçın (2015), in Asia Pacific Education Review, meta-analysed 57 articles and dissertations (28,964 participants) and, under a random-effects model, reported a medium-sized pooled effect size for the relationship between educational leadership and student achievement, largest for distributed and transformational leadership.
- Çoğaltay and Karadağ (2016), in Educational Sciences: Theory & Practice, pooled 77 Turkish studies published between 2000 and 2013, drawn from YÖK, ULAKBİM and Google Scholar, and found positive correlations between educational leadership and eight organisational variables, strongest for organisational justice (r = .75) and trust (r = .73) and weakest for citizenship (r = .35).
- A 2025 doctoral thesis at İstanbul Sabahattin Zaim University pooled 158 effect sizes from 113 studies on distributed leadership; see the research digest. Its results are reported as Fisher’s z values, so they need back-transforming before being read as correlations (a z of 0.816 for organisational trust, for example, corresponds to an r of about .67). Background on the construct is on our distributed leadership page.
Key takeaways
- A meta-analysis is a statistical synthesis inside a systematic review; PRISMA 2020 governs how both are reported, not how well they were done.
- In leadership research, use random effects, pool correlations via Fisher’s z, and always back-transform.
- Report a prediction interval alongside I², and treat funnel-plot methods as sensitivity checks rather than proof.
- Search Turkish theses and journals systematically, and guard against duplicate samples and dependent effect sizes.
References
- Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to meta-analysis (2nd ed.). Wiley.
- Borenstein, M., Higgins, J. P. T., Hedges, L. V., & Rothstein, H. R. (2017). Basics of meta-analysis: I² is not an absolute measure of heterogeneity. Research Synthesis Methods, 8(1), 5–18. https://doi.org/10.1002/jrsm.1230
- Çoğaltay, N., & Karadağ, E. (2016). The effect of educational leadership on organizational variables: A meta-analysis study in the sample of Turkey. Educational Sciences: Theory & Practice, 16(2), 603–646.
- Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. https://doi.org/10.1111/j.0006-341X.2000.00455.x
- Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
- Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. https://doi.org/10.1002/jrsm.5
- Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analyses. BMJ, 327(7414), 557–560. https://doi.org/10.1136/bmj.327.7414.557
- Karadağ, E., Bektaş, F., Çoğaltay, N., & Yalçın, M. (2015). The effect of educational leadership on students’ achievement: A meta-analysis study. Asia Pacific Education Review, 16(1), 79–93. https://doi.org/10.1007/s12564-015-9357-x
- Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
- Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007). Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Statistics in Medicine, 26(25), 4544–4562. https://doi.org/10.1002/sim.2889
- Pustejovsky, J. E., & Tipton, E. (2022). Meta-analysis with robust variance estimation: Expanding the range of working models. Prevention Science, 23(3), 425–438. https://doi.org/10.1007/s11121-021-01246-3
- Robinson, V. M. J., Lloyd, C. A., & Rowe, K. J. (2008). The impact of leadership on student outcomes: An analysis of the differential effects of leadership types. Educational Administration Quarterly, 44(5), 635–674. https://doi.org/10.1177/0013161X08321509
- Sterne, J. A. C., Sutton, A. J., Ioannidis, J. P. A., Terrin, N., Jones, D. R., Lau, J., Carpenter, J., Rücker, G., Harbord, R. M., Schmid, C. H., Tetzlaff, J., Deeks, J. J., Peters, J., Macaskill, P., Schwarzer, G., Duval, S., Altman, D. G., Moher, D., & Higgins, J. P. T. (2011). Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ, 343, d4002. https://doi.org/10.1136/bmj.d4002
- Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
Last reviewed: October 2026