We need better public policy science, and the sugar tax proves it.

How do you know if a government policy has worked? Last week we saw an attempt to answer this question, focusing on the 2018 soft drink tax – the “sugar tax” – which pushed beverage manufacturers to use less sugar in their products in an attempt to stop rising rates of obesity in the UK. .

Researchers at the University of Cambridge created a statistical model for the development of childhood obesity that included the period before and after the tax was announced in 2016 (after which beverage companies began to reformulate their products) and after its official entry into force.

Childhood obesity rates have increased over this period, so the argument is not that the tax helped reduce childhood obesity. But the researchers’ model suggested that there would be childhood obesity in a “counterfactual” world without the tax. even higher. In other words, the tax worked.

But the published results are messy. In children aged 4 to 5 years, no overall effects of exposure were found, and in some socioeconomic groups of this age, exposure appeared to be associated elevated Obesity. The effect of reducing obesity was observed only in girls aged 10-11 years, but not in boys of the same age.

The authors conducted various statistical tests – separately in boys and girls, in younger and older children, and in different socioeconomic groups. We expect some of these to be randomly statistically significant and appear to show that exposure reduces (or increases) obesity when it actually had no effect. The simplest explanation for this inconsistent and inconsistent data using Occam’s razor is that it is mainly due to statistical noise.

Even taking the results at face value, there are contradictions. Boys tend to drink more sugary drinks. a priori We expected the tax to have the greatest impact on them, but this was the opposite of what was found. The researchers try to explain the unexpected gender difference in two ways: by saying that 10-year-old boys are more receptive to soft drink ads, and 10-year-old girls are more receptive to public discussions about taxes. The plausibility of these assumptions can be disputed, but what is certain is that the study did not collect any data to support these assumptions.

There are other concerns as well. In their original 2017 study protocol, the researchers planned to analyze children’s weight data in three different ways: continuous measurement of BMI and categorical measurements of overweight and obesity. However, in the new study, instead of looking at BMI, they look at obesity and a new category called “overweight.”

Such inexplicable and unjustified changes in research cause sharp criticism in science. Without explanation, readers of the study have no way of knowing if the analysis was skipped for completely innocuous reasons — perhaps the data was no longer available — or if the analysis was performed and the results dropped because they fell short of expectations.

There is a more important point here. The idea behind public policy evaluation research is that if it is rigorous enough, it will convince skeptics that it actually works (any discussion moral or political viability managed separately from politics).

Instead, we have scientists studying trends in the data who can see something, perhaps in some subgroups. Those who were already for the sugar tax remain; those opposed remain unconvinced by the meager evidence presented in the study.

This needs to be changed. Instead of looking at high-level aggregated data and assuming that the changes are due to politics, scientists need to bring it all together: they need to show that specific people who gained less weight were also those who drank alcohol with less sugar. and that they drank them because of a change in policy. It’s not easy, but creating compelling research on complex topics is never easy.

Researchers also need to be prepared for the idea that policies — even ones they support themselves — might not work, and that understanding the impact might be gold in their data.

In fact, policies can be configured with built-in scoring. It is not always possible to test recommendations with randomized controlled trials, which would be ideal (although it is possible more often than many people think). But with a little creativity, elements of experimentation can be added—for example, “tiered” policies change gradually over time, allowing scientists to see if policies and health outcomes change together. Policies can also be implemented differently in different parts of the country, and results can be compared.

So how do you know if a government policy has worked? There are many good ideas for policy testing. A Cambridge study last week gives us a practical lesson on how Not to do this.