Estimating interviewer and observer effects for binary responses
by
Alastair Scott
University of Auckland
Coauthors: Peter Davis (Christchurch Scool of Medicine)

Many health surveys require interviewers or observers with a high degree of specialist knowledge and training. For example, we are currently involved in a study of adverse events in hospitals in New Zealand which involves the careful examination of a sample of patient records. This requires medically-trained observers with considerable experience and judgement. Since people with the required skills are hard to find, we are often forced to make do with a small number of highly trained people, each having a high case load as a consequence. It is well-known that this can lead to interviewer variability having a relatively large impact on the total error of estimates of simple quantities such as means and proportions but there is a widespread belief that the impact is much smaller when we look at more complex statistics such as regression coefficients and odds ratios. In a previous paper (Davis and Scott (1995)) we looked at the impact for continuous responses using a correlated components of variance model. However, most responses in health questionnaires are binary and it is known that this approach results in underestimating the effects for binary responses. In this talk we use a multi-level model to look at interviewer effects for binary data and illustrate with results from the adverse events study mentioned above.

Reference. Davis, P.D. and Scott, A.J. (1995). The effect of interviewer variance on domain comparisons. Survey Methodology, 21, 99-106.

Date received: August 13, 2001