Data-driven content strategy, and what data cannot decide

What a data-driven content strategy is, and the thing it cannot do

A data-driven content strategy uses evidence about what people searched for, read, ignored and did next to decide what to publish, where to put it and what to stop making. That definition is uncontroversial and most articles on the subject stop there.

My position is that most small businesses calling themselves data-driven do not have enough traffic for the data to settle the questions they are asking it. They then make editorial decisions anyway, using judgement, and describe them afterwards as data-led. That is worse than admitting the judgement, because it hides where the decision was made and stops anyone arguing with it.

The useful version of this discipline is knowing which questions your evidence can close and which it cannot. Those are 2 different lists and almost nobody writes down either.

Measurement changes what gets made

The economist Charles Goodhart observed in 1975 that a statistical regularity tends to collapse once you use it for control. Marilyn Strathern gave it the phrasing everyone now uses, in a 1997 paper on audit in British universities: when a measure becomes a target, it ceases to be a good measure.

Content work runs into this immediately. Pick time on page as your measure and you will get longer articles, whether or not the reader needed one. Pick published volume and you will get volume. Pick click-through rate on titles and you will get titles that overpromise, which raises the measure while lowering the thing it was standing in for.

Eric Ries popularised the term vanity metric in 'The Lean Startup' in 2011: a number that reliably goes up and never changes a decision. Page views usually qualify. The test is whether you can name the action you would take if the number halved. If you cannot, you are reporting, not measuring.

The bias has a direction, and it is towards the short term

This is not merely that measurement is imperfect. It is that the parts of marketing which are easiest to measure are the parts that pay back fastest, so a measurement-led process reallocates money towards them by default.

Les Binet and Peter Field set this out in 'The Long and the Short of It' for the IPA in 2013, drawn from the same IPA Effectiveness Awards Databank I use in the role of emotions in purchase decisions. Their headline conclusion was the 60:40 split: roughly 60% of budget on brand building and 40% on sales activation produced the best long-run results. Activation gives a sharp, short-lived lift that shows up clearly in your analytics. Brand building works slowly and shows up late, in figures that are hard to attribute to anything specific.

There are 2 limits worth holding with that. Binet and Field have since described 60:40 as a crude average across many categories and brand situations, later broken into more granular advice. And it is advertising data from brands with media budgets, so the ratio is not a rule for a business publishing articles.

The mechanism transfers even though the number does not. The work with the fastest feedback loop wins every budget argument, because it is the only work with evidence attached by the time the argument happens.

Your site probably cannot detect the improvement you are testing for

Here is the part I would put in front of anyone about to build a testing programme. These are standard sample sizes for comparing 2 conversion rates, at 80% power and 5% significance, starting from a 2% conversion rate. Traffic is split evenly between 2 versions.

Improvement you want to detect	Visitors needed per version	Time at 500 visits a month	Time at 5,000 visits a month
2% to 2.2%, a 10% lift	About 81,000	About 27 years	About 32 months
2% to 2.4%, a 20% lift	About 21,000	About 7 years	About 8 months
2% to 3%, a 50% lift	About 3,800	About 15 months	About 6 weeks
2% to 4%, doubling	About 1,100	About 5 months	Under 3 weeks

The numbers are not a judgement about your business. They fall out of the arithmetic of detecting a small difference in a rare event, and they are the reason most published A/B testing advice is written for sites you do not have. The tests that showed nothing are also the ones nobody publishes, which is survivorship bias operating on your reading list.

There are 3 consequences. A test you stop early because it looks positive has told you nothing, and the smaller your traffic the more likely an early result is noise. If you cannot run the test, say so and decide editorially rather than running it anyway and treating the outcome as evidence. And the only changes worth testing at low traffic are the large ones, which usually means the offer, the price or the page's whole purpose, rather than a button colour.

What to measure when you cannot run tests

Small numbers still carry information if you stop asking them to prove causation.

Read the enquiries. A set of 20 enquiries tells you more about which page is doing the work than 20,000 sessions, because each one carries the reason attached. Note which article gets mentioned unprompted.

Track a small number of things that would change a decision. What people searched for before arriving. Which pages appear in the path to an enquiry. Which pages have not been visited at all in 6 months, which is a retirement decision rather than an optimisation one and belongs in the content lifecycle.

Watch for the questions your content is not answering, which show up in what people ask you after reading it. That is the highest-value content data a small business has and it does not appear in any analytics tool. I go through how to separate the request from the underlying need in user needs versus what users ask for.

Where creativity earns its place

Creativity is not the counterweight to data in this argument. It is what you use for the decisions data was never going to settle: what the thing is about, what it is for, what tone it takes, whether it is worth making at all. No amount of analytics generates an idea, and evidence about what performed cannot tell you what has not been tried.

An earlier version of this article said content strategists take the audience somewhere they do not realise they have been taken until it is too late. I wrote that and it was wrong on the substance as well as the tone. It describes manipulation and it contradicts the position I hold across the rest of this site, which is that the buyer's ability to decide for themselves is the thing content should be protecting. I have removed it rather than softening it, because softening it would have left the idea in place.

The honest version is duller. Good content strategy makes it easier for someone to work out whether you are any use to them, including when the answer is no.

What I do when the data is not there

Most of my public sector work involved decisions that could not be measured in the way this article's competitors assume. There was no meaningful test traffic on an individual page, no conversion value to attribute, and often no analytics access at all on the timescale the decision had to be made.

The practice that came out of that is the one I still use. Write down the decision, the reasoning, and what you expected to happen. Then when something does happen, you have a record of what you believed beforehand, which is the only thing that stops you reinterpreting the outcome to fit.

I want to be careful about what that is worth. It is a habit, not a method with evidence behind it, and I have never compared it against teams that did not do it. Its value is that it makes your reasoning checkable later, including by you.

What to do this week

Take your content reporting and go through it line by line, asking what you would do differently if each number halved. Delete the ones with no answer. Most reports lose more than half their contents to that question.

Then pick the decision you have been putting off because you were waiting for data. Work out from the table above whether the data was ever going to arrive. If it was not, make the decision on judgement, write down why, and say plainly in the meeting that this is what you have done.