How many items are too many? An analysis of respondent disengagement when completing questionnaires

Abstract

Researchers are often confronted with the dilemma of how many items to include in a questionnaire. On the one hand the researcher requires accurate and comprehensive information but, on the other, textbooks alert us to the danger of respondent fatigue and respondents disengaging as a result of completing lengthy questionnaires. This paper aims to empirically address the question of how many items a questionnaire can contain before additional items can be considered to be superfluous, as well as the question of at what stage fatigue and disengagement set in. The responses of 3 180 individuals to a questionnaire of 153 items were analysed, focusing on the number of missing cases as well as on zero-variance and flat response patterns. The results reveal that neither the number of missing values, nor zero-variance or flat response patterns, are a function of the number of items (already) completed. The results further reveal that the response pattern of respondents, on a battery of 153 items, can be attributed to the types of items included in the questionnaire rather than to the number of items completed. Recommendations are presented on strategies to design questionnaires that include a low number of items, but still fulfil the requirements of reliability and validity.