Making sense of the conflicting results arising from the analysis of road safety data
by
Tony Lewis
Massey University

The New Zealand drink-drive television advertising campaign, which began in earnest in October 1995, and continues with a changed emphasis as we speak, has been analysed for effectiveness at least seven times. Five times the campaign has been found to be effective, twice it has been not been found to be effective. As the protagonists criticised one another's work, they dwelt on data sources, and on the methods of analysis. The criticism was usually about technical aspects of the analysis. The other authors did something wrong, they didn't take account of this or that possible effect, or their conclusions were not supported by their results. Some authors questioned the independence of other authors. Some authors thought direct measurements of driving behaviour, such as drink-driving convictions, should be used as the dependent variable. Others thought the results of drinking and driving, road deaths or serious accidents, should be the dependent variable. There was also much discussion of how the campaign itself should be measured, what proxy measures for difficult-to-measure explanatory variables should be used; for instance some authors used unemployment as a proxy for economic conditions, others used new car registrations. The discussion has not proved helpful for policy makers. The arcane language of professional analysts means that the results are not accessible to the lay public. The policy makers have to rely on not-very-helpful translations; translations of the reasons put forward for the conflicting results are often even less helpful. This paper shows that using one data source and one set of variables, one author can obtain the results of the seven different analysts. The analytical methods used are rather simple, and easy to comprehend for policy makers. It turns out that the culprit is not the data source or proxy variable choice, but is that different behavioural models underlie the statistical models used by the analysts. Moreover there is no way of verifying which model is correct. The solution is to adopt an experimental approach, and such an approach is suggested in this paper. So far, suggestions for an experimental approach have not found favour with the authorities who determine how road safety programmes are conducted. Approval from my colleagues and a little publicity might change this.

Date received: September 29, 2001