Measuring the impact of poor questionnaire design

C.C. Gilbert (1),  E.M. Allchurch (1), T. Kuti (2)

1Campden & Chorleywood Food Research Association (UK)
2Campden & Chorleywood Food Industry Development Institute (Hungary)

Questionnaires are a powerful tool for exploring and quantifying attitudes, behaviours and preferences amongst a selected group of respondents. However, the validity and reliability of the data are dependent on the quality of the questionnaire used. This project aimed to investigate and directly measure the effect of altering various questionnaire parameters, including question construction and phrasing, order and type of questions, and selection of scales.

The research was conducted within the context of a panel motivation survey, administered to descriptive profile panellists. Two questionnaires were designed, both with the same intention of gaining insight into panellists' views on panel motivation and feedback. The 'real' questionnaire was finalised first, addressing all of the aims of the study. The ‘alternative’ questionnaire was then developed to achieve the same aims but using questions that were phrased or formatted in a slightly different way. For validation purposes, two questions remained identical on both questionnaires, enabling us to check that there was no effect of sampling. Within each panel, panellists were randomly assigned to either the real or alternative questionnaire. In total, 23 sensory panels took part, resulting in 135 and 133 panellists for the real and alternative questionnaires respectively.

Comparison of responses from the two questionnaires highlighted many interesting findings. For example, several significant differences were found when comparing agreement levels to a series of statements, particularly when questions were longer or poorly phrased. On the other hand, no significant differences were found between responses to the same question, located at either the beginning or end of the questionnaire. Results also revealed several examples of how the choice of question type and scale could hinder the analysis and interpretation of the data. Although respondents for this study were sensory panellists, the results have wider implications for the design of questionnaires for consumer research.