The Number That Started the Reckoning
In 2015, a consortium of more than 270 researchers, organized through the Center for Open Science, published the results of the Reproducibility Project: an attempt to directly replicate 100 studies published in three major psychology journals. The result: only about 36 to 39 percent of the replications produced statistically significant findings matching the original studies, depending on how the result was measured — and even among those that did replicate, the size of the effect was, on average, about half as large as originally reported.
This wasn’t an isolated embarrassment. It confirmed a pattern that had already been building for several years, beginning with a series of individually damaging episodes — the 2011 publication of Daryl Bem’s studies claiming evidence for precognition (which passed the field’s standard statistical and peer-review bar at the time, a fact that itself became a major point of methodological reckoning), the exposure of outright fraud by social psychologist Diederik Stapel (who fabricated data across dozens of published papers), and a growing body of work by statisticians and methodologists identifying structural reasons why published psychology findings might be less reliable than their p-values suggested. The 2015 reproducibility project gave this growing concern a number, and the number was bad enough that “replication crisis” became the field’s own name for what had happened to it.
Why So Much of It Failed
The explanations that emerged from this period, and the corrective methodological practices that followed, are directly relevant to evaluating any personality research — including, but not limited to, astrology research, which sits within the same broader methodological ecosystem and is subject to the same structural vulnerabilities, often more acutely.
Underpowered studies were endemic. Much of psychology’s published research, especially in earlier decades, used sample sizes too small to reliably detect the effect sizes researchers were actually claiming — a problem that doesn’t just produce more “noise” results, but specifically inflates the apparent size of whatever effects do reach statistical significance, because in an underpowered study, only unusually large (and therefore likely to be partly or wholly due to chance) estimates clear the significance bar at all. This is sometimes called the “winner’s curse”: published effect sizes from small studies are systematically inflated relative to the true effect, simply because of which results survived the filter of statistical significance.
Researcher degrees of freedom, formalized in an influential 2011 paper by Joseph Simmons, Leif Nelson, and Uri Simonsohn, described how the many small, seemingly reasonable choices a researcher makes while analyzing data — which outliers to exclude, which covariates to control for, when to stop collecting data, which of several similar measures to report — could be combined, even without any deliberate intent to cheat, to dramatically inflate the rate at which a dataset with no real effect produces a “significant” result. Their paper demonstrated this could push the true false-positive rate from the nominal 5 percent up past 60 percent, using only analytical choices that, individually, looked like ordinary, defensible research practice.
Publication bias and the file drawer problem, discussed in more detail in a companion piece, meant that the studies which did get published and widely cited were systematically unrepresentative of the full population of studies actually conducted — skewed toward positive, novel, surprising findings, with null results and failed replications much less likely to ever see print.
Personality Psychology’s Specific Exposure
Personality research, as a subfield, has its own complicating factors that made it particularly exposed to these problems, beyond the general issues affecting all of social psychology.
Personality constructs are often measured through self-report questionnaires, which introduce their own well-documented distortions — social desirability bias (answering in ways that present oneself favorably), reference-group effects (rating yourself relative to whoever you implicitly compare yourself to, which varies across cultures and contexts), and straightforward inconsistency in how people interpret rating scales. These measurement issues compound with the broader replication-crisis problems: a personality finding built on noisy self-report data, run through small samples with substantial researcher degrees of freedom, faces multiple independent reasons to be unreliable, stacked on top of each other.
The field’s response, over the decade following 2011, has been substantial: large-scale adoption of pre-registration (publicly committing to a study’s hypotheses and analysis plan before collecting data, closing off the researcher-degrees-of-freedom problem), a strong push toward much larger sample sizes (directly addressing the underpowered-study and winner’s-curse problems), and growing institutional support for publishing and valuing replication studies and null results, which were previously treated as unpublishable. The Big Five framework discussed in the companion piece on astrology and personality research has, by most assessments, held up comparatively well through this reckoning — its core factor structure has been replicated across many independent samples, cultures, and languages, which is part of why it remains the standard framework for personality measurement even after a decade of more skeptical scrutiny across the field generally.
What This Means for Astrology Research Specifically
The replication crisis cuts in two directions for evaluating astrology research, and both directions are worth taking seriously rather than picking whichever supports a preferred conclusion.
In one direction: research finding no relationship between astrological factors and personality — the dominant finding throughout this series — is, if anything, more credible in a post-replication-crisis context than it would have been before, because null results have historically been undervalued and underpublished relative to positive findings. A null result that has nonetheless been published, replicated across multiple independent research groups, and held up using large samples (as the sun-sign-personality null result has) is exactly the kind of finding the field’s methodological reforms were designed to surface and protect, rather than the kind of finding that was historically vulnerable to file-drawer suppression. The consistency of the null finding across decades and multiple research groups is a point in its favor as a real, robust finding — not (as a casual reading of “everything in psychology is unreliable now” might suggest) a reason to doubt it.
In the other direction: any positive astrology finding — including the contested Gauquelin results discussed in the companion piece on meta-analysis — needs to be evaluated with exactly the same heightened scrutiny the replication crisis demands of any psychology finding generally. A significant result from an underpowered sample, analyzed with undisclosed researcher degrees of freedom, without pre-registration, carries much less evidential weight post-2011 than it would have carried in, say, 1995, when the field’s standards for what counted as a credible finding were considerably looser. This is true whether the positive finding favors or opposes astrology’s claims — the same skepticism that should be applied to a small study claiming evidence for telepathy or precognition applies equally to a small study claiming definitive proof that astrology doesn’t work, if that study has the same structural vulnerabilities.
The General Lesson, Applied Specifically
The replication crisis didn’t just lower confidence in some specific findings. It changed what counts as a credible methodology across the whole field, and that change applies retroactively to research conducted before the standards existed, not just prospectively to research conducted after. A study from 1985, however influential, should be evaluated by current standards regarding sample size, pre-registration, and researcher degrees of freedom — not given a pass because it was considered rigorous by the standards of its own time.
This is precisely why the discussion elsewhere in this series of Carlson’s 1985 study, and its 2023 reanalysis, matters as much as it does: the reanalysis isn’t simply relitigating an old dispute, it’s applying post-replication-crisis standards of statistical scrutiny to a study that predates those standards by decades, and finding that the original framing oversold the certainty of its conclusion in exactly the way the replication crisis has shown was endemic to psychology research from that era.
None of this resolves whether astrology, in any of its many forms, has genuine predictive validity. What it does is recalibrate how much weight any single study — for or against — deserves, given what’s now understood about how unreliable psychology’s pre-2011 research base turned out to be, and how much more rigorous the discipline’s standards have had to become since.