Survivorship bias in marketing, and how to spot it

A case study is a sample of one, already filtered by its outcome

Marketing runs on case studies, and the case study is the least reliable evidence format the industry uses. That is my position and it makes most of the reading in this trade close to worthless for deciding what to do.

The problem is not that the winners lie. It is that you never see the businesses that did the same thing and got nothing, because nobody writes those up. Whatever the winner did, you are looking at a sample that was selected on the outcome you are trying to explain.

You can still learn from a success story. What you can learn is a mechanism, and only if the story explains one. What you cannot learn is whether the tactic works.

What survivorship bias is

Survivorship bias is a selection effect. It happens when the sample you can see has already been filtered by the thing you want to understand, so you draw a conclusion from the survivors that the full set would not support.

It is a form of sampling bias, and you will see it called selection bias or survival bias. The mechanism is the same in each case: the failures are not merely absent from your data, they are absent from the world you can observe. There is no list of them anywhere.

That last point is what makes it different from most reasoning errors. You can correct for a bias you can see. This one removes the evidence that would tell you the bias is there.

The bomber story, and how it gets told

Every article on this subject uses the same example. Abraham Wald, working with the Statistical Research Group at Columbia during the second world war, was asked where to add armour to bombers. The engineers had mapped the damage on returning aircraft. Wald pointed out that the returning aircraft were the survivors, so the areas without damage were the areas where a hit stopped the plane coming back.

The story is sound and the underlying work is real. Wald's 1943 memoranda, 'A Method of Estimating Plane Vulnerability Based on Damage of Survivors', were technical, classified, and largely unknown until they were reprinted in 1980.

The way the story travels is worth a moment, because it demonstrates the point better than the story does.

The plane covered in red dots, the image attached to almost every article about survivorship bias, is not from Wald's work. It is a diagram added to Wikipedia in 2016 and described there as loosely based on data from an unillustrated report. Wald's papers contain no such picture. The line usually quoted, about putting the armour where the holes are not, is a modern paraphrase rather than something he wrote.

So the canonical example of learning only from survivors is itself circulated in a version that survived because it was memorable, with an artefact invented later and attributed backwards. I find that more instructive than the original anecdote, and it is the reason I now check the provenance of any story that arrives already polished.

The business books that got this wrong in public

Two books show the failure at full scale, and both were enormously influential on how marketing thinks.

'In Search of Excellence', Peters and Waterman, 1982, identified the shared attributes of America's best-run companies. Business Week ran a cover in 1984, 2 years later, finding that a substantial number of those companies were in trouble. The attributes had been derived from the winners with no comparison group of similar companies that did the same things and did not win.

'Good to Great', Jim Collins, 2001, selected 11 companies from a starting set of 1,435 on the basis of their share price performance, then looked for what they had in common. Circuit City, one of the 11, went bankrupt in 2009. Fannie Mae, another, collapsed in 2008 and went into government conservatorship.

Phil Rosenzweig set out the underlying methodological problem in 'The Halo Effect' in 2007. When a company is performing well, observers rate its strategy, culture and leadership as strong, and rate the same characteristics as weak when performance falls. Studies that gather those ratings after the fact are measuring the outcome twice and calling one of them a cause.

I am not saying either book is worthless. I am saying that a method which starts by selecting winners cannot tell you what caused the winning, and that this is the method behind most of what marketers read.

Find out who is missing from your evidence

This is the practical part and it is 4 columns. Take whatever you are about to base a decision on and put it through the second column, which is the one that does the work.

Evidence you are using	Who was filtered out before you saw it	What it pushes you towards	The question that recovers the missing half
An agency case study	Every client of that agency whose results were not worth publishing, and every client who left	Believing the tactic caused the result	How many clients did you run this for, and what happened to the rest
A competitor teardown	The competitors who did the same things and closed	Copying the visible choices of a business whose starting position you cannot see	What did they have at the start that I do not have
Your own past campaigns	The ideas that were killed before launch, usually by whoever was most senior in the room	Repeating a narrow set of formats because those are the ones that got measured	What did we decide not to run, and was that decision ever tested
Customer research	Everyone who left, never bought, or bounced without contact	Optimising for the people you already please	Where can I hear from the people who chose somebody else
Published test results	Every test that showed no effect, which almost nobody writes up	Overestimating how large a typical effect is	What was the result when this was tried and did nothing
Founder and expert advice	Everyone who followed the same advice and is not on stage	Treating a personal history as a method	What was true of their situation that is not true of mine

The pattern in the second column is that the missing group is defined by the same thing you are studying. That is the tell. When the filter and the question are the same, the evidence cannot answer it.

My own portfolio is a survivor sample

I should apply this to myself, because it would be inconsistent not to.

Everything I have ever put in a portfolio, a proposal or an article worked well enough to be worth describing. Over more than 20 years in communications and marketing I have also had projects that went nowhere: content that was carefully researched and never used, a rebuild that improved nothing I could detect, work that was overtaken by an internal reorganisation before it launched. None of that is written up anywhere, including on this site.

I am not offering that as a confession. I am pointing out that a reader assessing me has the same problem this article describes, and that I am one of the sources they should be asking the questions in that table about. The honest fix is small and I have started doing it: when I describe a piece of work, I say what I could not measure and what changed at the same time. The recruitment site example in what working in government taught me and the screening example in how content design solves user needs both carry that treatment now.

Why this is hard to notice

The survivors are more available. A business that closed is not blogging about why, and a campaign that produced nothing does not get a conference talk. The winners are also better storytellers about themselves, because success invites explanation and the explanation gets tidier each time it is told.

One number is worth keeping in view. The ONS business demography statistics put the 5-year survival rate for UK businesses born in 2019 at 38.4%. Most of the businesses that started alongside the ones giving you advice are gone. Any account of what makes a business work is drawn from the minority that remained, which is exactly the sample survivorship bias warns about.

That figure does not mean the advice is wrong. It means the advice has never been tested against the group it would need to be tested against.

What to do about it

You cannot get the missing data. You can change what you accept as a reason to act.

Ask for the denominator. When someone describes a result, ask how many times they did the same thing and what happened on the other occasions. A refusal to answer is an answer.

Prefer a mechanism to a tactic. A tactic is a thing that worked once somewhere. A mechanism is a reason it worked, stated clearly enough that you can check whether the reason applies to you. If a case study cannot supply the reason, it has told you nothing transferable.

Work out whether the test was ever possible. Most published results come from sites with traffic that a small business does not have, and I set out the sample sizes involved in what data cannot decide.

Keep your own record of what did not work. This is the only failure data you will ever have access to, and it disappears if you do not write it down. A single file with the decision, the reasoning at the time and what happened is enough. I treat it as part of the same discipline as the content lifecycle, which is about writing down the decisions you would otherwise have to remake from scratch.

Treat a confident claim about buyer behaviour as a claim to check rather than a fact to repeat. I go through the specific ones in the science of consumer behaviour and, for the research that failed to replicate, in System 1 decision-making.

Re-examine your last borrowed decision

Take the last marketing decision you made because of something you read, and ask the second column question about it. In my experience most decisions of that kind cannot survive being asked how many people tried the same thing.

Then start the file of things that did not work. It will be the most useful marketing document you own within a year, and you are the only person who can write it.