Why Your Existing Impression Isn’t Evidence
If you’ve used a divination system for any length of time — daily horoscopes, tarot, a multi-system synthesis — you almost certainly already have an impression of whether it “works” for you. The trouble, as the rest of this series has covered in detail, is that this impression was formed through a process saturated with mechanisms that produce a feeling of accuracy regardless of whether genuine signal is present: hindsight bias reshaping memory of what a reading actually said, base rate neglect making broadly-true statements feel specifically true, the clustering illusion making random streaks of apparent hits feel meaningful, and confirmation bias steering attention toward confirming instances while disconfirming ones pass unremarked.
This means your accumulated impression isn’t really evidence about the system. It’s evidence about how these cognitive mechanisms have operated on your particular history of exposure to the system. A genuine self-test requires deliberately structuring your engagement with the system to neutralize these mechanisms, at least partially — which is more effort than simply continuing to notice whether readings feel accurate, but is the only way to get information that actually bears on the question.
The Core Design Principle: Blind Yourself to the Outcome You’re Predicting
The single most important methodological principle, borrowed directly from the structure of rigorous studies like Carlson’s 1985 test (discussed elsewhere in this series), is this: the prediction has to be recorded, in some retrievable and unambiguous form, before you know the outcome it’s supposed to predict — and ideally, evaluated by a process that doesn’t allow your memory of the prediction to drift toward matching whatever actually happened.
This sounds simple but is harder to implement than it sounds, because the most natural way of “checking if a reading was right” — recalling what it said and comparing that recollection to what happened — is exactly the process hindsight bias corrupts. The fix is mechanical rather than psychological: write the prediction down, in your own words or via a screenshot, somewhere you genuinely won’t revisit until a predetermined evaluation date, and don’t rely on memory for either the prediction’s content or your initial confidence in it.
A Workable Protocol
Here’s a structure that incorporates the main lessons from the research covered throughout this series, designed to be practical for an individual rather than requiring institutional resources.
Step one: pre-register your own predictions. Before reading a forecast or reading, write down — in a notebook, a private document, anywhere with a timestamp you trust — what you expect the reading to be useful for predicting, and what would count as a hit versus a miss, before you read it. This sounds backward (shouldn’t the reading generate the prediction, not you?) but it’s necessary because without a pre-specified definition of “hit,” the base-rate-neglect and hindsight-bias mechanisms will retroactively redefine what counts as a match once you know the outcome.
Step two: record the reading’s actual content verbatim, or close to it. Don’t rely on your summary or your memory of “the gist.” Screenshot it, copy the text, or write it down immediately. This directly counters the memory-distortion component of hindsight bias, which has been shown experimentally to shift recalled predictions toward whatever actually happened.
Step three: separately, before evaluating, write down the base rate. Ask explicitly: if this exact statement were shown to a random sample of people, what fraction would say it applied to them? This is the single step most self-testing skips, and it’s the direct application of the base-rate-neglect research discussed elsewhere in this series. A reading that “comes true” but had an 85 percent base rate to begin with has told you almost nothing, however striking the match feels.
Step four: set a fixed evaluation date and a fixed evaluation method in advance. Decide, before you know the outcome, exactly when and how you’ll check whether the prediction was a hit — and stick to that method rather than letting the evaluation criteria drift based on what feels like a reasonable interpretation once you know what happened. This is the self-administered version of pre-registration, the methodology developed in response to the replication crisis discussed elsewhere in this series specifically to prevent researchers (and, by extension, individuals doing informal self-research) from unconsciously adjusting their analysis after seeing the data.
Step five: track misses with the same diligence as hits. This is the step most informal self-testing fails at completely, because misses don’t get remembered, discussed, or written about with anything like the attention hits receive. Without a comparably complete record of misses, you cannot calculate an actual hit rate — you can only generate a list of memorable successes, which tells you nothing about the denominator they’d need to be divided by.
What a Real Result Would Look Like
After enough trials — and “enough” matters here; a handful of predictions isn’t sufficient to distinguish a real effect from chance, for the same statistical-power reasons that small studies are unreliable throughout the research literature covered in this series — you’d have an actual hit rate: number of pre-specified hits divided by total pre-specified predictions, with the base rate of each prediction recorded alongside it.
The genuinely informative comparison isn’t “did this feel accurate” but “did the hit rate exceed what the base rates of the individual predictions would predict by chance.” If your predictions average an 80 percent base rate (broadly applicable statements that most people would endorse) and your hit rate comes out around 80 percent, you’ve learned that the system performs exactly as well as a system with no signal at all would, given how broadly applicable its statements are. If your predictions average a 20 percent base rate (specific, less commonly true statements) and your hit rate comes out around 60 percent, that’s a result worth taking seriously — though even then, a single individual’s self-test, run over weeks or months, doesn’t have the statistical power of a study with hundreds of participants, and shouldn’t be treated as more conclusive than it is.
The Honest Limits of This Method
It’s worth being clear-eyed about what even a carefully run personal A/B test can and can’t establish, because overclaiming in either direction undermines the exercise.
A single person’s self-test, however carefully designed, can’t control for everything a proper research study controls for. You’re both the experimenter and the subject, which introduces motivated-reasoning risks that a blinded, independent evaluator wouldn’t have — even with pre-registration, there’s room for unconscious flexibility in how you categorize a borderline result as hit or miss, particularly because you have an interest (in either direction) in the outcome. Sample size, for any individual running this over a reasonable period, will be small relative to what a proper study would use, meaning the result — whatever it is — carries real uncertainty that a “studies of one person” framing should make explicit rather than glossing over.
What this kind of self-testing can do, even with these limits, is shift your relationship to the system from passive impression-accumulation (which the cognitive biases throughout this series are specifically good at exploiting) toward something with at least partial methodological defenses against those biases. It won’t produce publication-grade evidence. It will very likely produce a more honest, more calibrated sense of what’s actually happening when you find a reading striking — closer to the ground truth than the unstructured impression you’d otherwise be relying on, even if it falls well short of a real controlled study.
Why Almost No One Actually Does This
It’s worth ending on the obvious question: if this method is available and methodologically sound, why does almost no one — including people who are skeptical of divination and curious about testing it — actually do it?
Part of the answer is simply effort: the protocol above requires sustained discipline over weeks or months, for a question most people aren’t sufficiently motivated to resolve with this much rigor. But part of the answer is more interesting, and connects to themes throughout this series: the unstructured, biased version of engaging with a reading — letting hindsight bias, base rate neglect, and the clustering illusion do their normal work — produces a more immediately pleasant experience than the rigorous version. The rigorous version risks generating a clear, disappointing answer. The unstructured version reliably generates the feeling of insight, regardless of what’s actually there.
This isn’t necessarily an argument against the unstructured version — the discussion elsewhere in this series about self-fulfilling prophecy and narrative psychology suggests that the unstructured experience may have genuine value, independent of predictive accuracy. But it does mean that the gap between “almost nobody rigorously tests this” and “this could be rigorously tested” isn’t really about feasibility. It’s about what most people are actually looking for when they consult an oracle — and a hit rate, calculated honestly against real base rates, usually isn’t it.