Questioning the questionnaire: Expediency of reviewing and publication versus adequate description and methodological justification

Abstract

The questionnaire is one of the fundamental research instruments in the field of Information Technology and Information Systems research. The survey as data capturing method with the questionnaire as data capturing tool is used in various research designs. The wide‐ranging application field of the questionnaire as research instrument has led to varied practices in conducting and reporting questionnaire driven research. Standards of quality vary along with the assumptions underlying different philosophical traditions. Considering the ontological and methodological differences between paradigms the transfer of methods and tools between paradigms can introduce deviations in accepted practices. This raises questions about the critical constraints on using the questionnaire as research tool. The present study investigates issues related to the rigor of survey based research reporting. Precisely, it evaluates questionnaire‐driven research reporting and then seeks to determine the reasons for deviating from accepted reporting practices such as providing access to the questionnaire. To fulfil this objective, the research design entails document analysis of papers presented at a leading South African Computer Science and Information Systems conference over a six‐year period (2008 to 2013). The analyses sought to identify trends in the reporting of the questionnaire design. The survey‐driven studies were identified from the proceedings and the reporting practice was analyzed. The specific conference was selected since it is a reputable annual conference in the South African IT/IS field. The analysis revealed that the questionnaire was made available in less than one third of the survey‐based articles on average. Having found a definite trend towards omitting the questionnaire, an short survey was conducted with 12 well‐published researchers to get their opinion on including the questionnaire and also to uncover the underlying reasons for the omission practice. Reporting practices impact the rigor of any research since rigorous research needs to be done and rigorous research needs to be seen to be done. The nonreporting practice affects the quality of the findings measured in terms of reliability and validity for quantitative research and in terms of trustworthiness, confirmability, and consistency for qualitative research; therefore it is important to question why this practice prevails. The contribution of the study is to highlight a practice of not providing the questionnaire as research instrument and provides some reasons why the practice prevails. This investigation is meant to open a larger debate on the governance of reporting practices. The paper should be of interest to researchers that use surveys and consume survey based findings as well as reviewers, editors and academic conference chairpersons.