The effect of answering in a preferred versus a non-preferred survey mode on measurement

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Previous research has shown that offering respondents their preferred mode can increase response rates, but the effect of doing so on how respondents process and answer survey questions (i.e., measurement) is unclear. In this paper, we evaluate whether changes in question format have different effects on data quality for those responding in their preferred mode than for those responding in a non-preferred mode for three question types (multiple answer, openended, and grid). Respondents were asked about their preferred mode in a 2008 survey and were recontacted in 2009. In the recontact survey, respondents were randomly assigned to one of two modes such that some responded in their preferred mode and others did not. They were also randomly assigned to one of two questionnaire forms in which the format of individual questions was varied. On the multiple answer and open-ended items, those who answered in a non-preferred mode seemed to take advantage of opportunities to satisfice when the question format allowed or encouraged it (e.g., selecting fewer items in the check-all than the forcedchoice format and being more likely to skip the open-ended item when it had a larger answer box), while those who answered in a preferred mode did not. There was no difference on a grid formatted item across those who did and did not respond by their preferred mode, but results indicate that a fully labeled grid reduced item missing rates vis-á-vis a grid with only column heading labels. Results provide insight into the effect of tailoring to mode preference on commonly used questionnaire design features.

Original languageEnglish (US)
Pages (from-to)137-152
Number of pages16
JournalSurvey Research Methods
Volume8
Issue number3
StatePublished - Jan 1 2014

Fingerprint

questionnaire
data quality

Keywords

  • Check-all
  • Data quality
  • Forced-choice
  • Grid questions
  • Mail survey
  • Mode preference
  • Open-ends
  • Web survey

ASJC Scopus subject areas

  • Education

Cite this

The effect of answering in a preferred versus a non-preferred survey mode on measurement. / Smyth, Jolene D.; Olson, Kristen; Kasabian, Alian.

In: Survey Research Methods, Vol. 8, No. 3, 01.01.2014, p. 137-152.

Research output: Contribution to journalArticle

@article{07edaaf3d51e4bc9aed9ec2b4e1d2b62,
title = "The effect of answering in a preferred versus a non-preferred survey mode on measurement",
abstract = "Previous research has shown that offering respondents their preferred mode can increase response rates, but the effect of doing so on how respondents process and answer survey questions (i.e., measurement) is unclear. In this paper, we evaluate whether changes in question format have different effects on data quality for those responding in their preferred mode than for those responding in a non-preferred mode for three question types (multiple answer, openended, and grid). Respondents were asked about their preferred mode in a 2008 survey and were recontacted in 2009. In the recontact survey, respondents were randomly assigned to one of two modes such that some responded in their preferred mode and others did not. They were also randomly assigned to one of two questionnaire forms in which the format of individual questions was varied. On the multiple answer and open-ended items, those who answered in a non-preferred mode seemed to take advantage of opportunities to satisfice when the question format allowed or encouraged it (e.g., selecting fewer items in the check-all than the forcedchoice format and being more likely to skip the open-ended item when it had a larger answer box), while those who answered in a preferred mode did not. There was no difference on a grid formatted item across those who did and did not respond by their preferred mode, but results indicate that a fully labeled grid reduced item missing rates vis-{\'a}-vis a grid with only column heading labels. Results provide insight into the effect of tailoring to mode preference on commonly used questionnaire design features.",
keywords = "Check-all, Data quality, Forced-choice, Grid questions, Mail survey, Mode preference, Open-ends, Web survey",
author = "Smyth, {Jolene D.} and Kristen Olson and Alian Kasabian",
year = "2014",
month = "1",
day = "1",
language = "English (US)",
volume = "8",
pages = "137--152",
journal = "Survey Research Methods",
issn = "1864-3361",
publisher = "European Survey Research Association",
number = "3",

}

TY - JOUR

T1 - The effect of answering in a preferred versus a non-preferred survey mode on measurement

AU - Smyth, Jolene D.

AU - Olson, Kristen

AU - Kasabian, Alian

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Previous research has shown that offering respondents their preferred mode can increase response rates, but the effect of doing so on how respondents process and answer survey questions (i.e., measurement) is unclear. In this paper, we evaluate whether changes in question format have different effects on data quality for those responding in their preferred mode than for those responding in a non-preferred mode for three question types (multiple answer, openended, and grid). Respondents were asked about their preferred mode in a 2008 survey and were recontacted in 2009. In the recontact survey, respondents were randomly assigned to one of two modes such that some responded in their preferred mode and others did not. They were also randomly assigned to one of two questionnaire forms in which the format of individual questions was varied. On the multiple answer and open-ended items, those who answered in a non-preferred mode seemed to take advantage of opportunities to satisfice when the question format allowed or encouraged it (e.g., selecting fewer items in the check-all than the forcedchoice format and being more likely to skip the open-ended item when it had a larger answer box), while those who answered in a preferred mode did not. There was no difference on a grid formatted item across those who did and did not respond by their preferred mode, but results indicate that a fully labeled grid reduced item missing rates vis-á-vis a grid with only column heading labels. Results provide insight into the effect of tailoring to mode preference on commonly used questionnaire design features.

AB - Previous research has shown that offering respondents their preferred mode can increase response rates, but the effect of doing so on how respondents process and answer survey questions (i.e., measurement) is unclear. In this paper, we evaluate whether changes in question format have different effects on data quality for those responding in their preferred mode than for those responding in a non-preferred mode for three question types (multiple answer, openended, and grid). Respondents were asked about their preferred mode in a 2008 survey and were recontacted in 2009. In the recontact survey, respondents were randomly assigned to one of two modes such that some responded in their preferred mode and others did not. They were also randomly assigned to one of two questionnaire forms in which the format of individual questions was varied. On the multiple answer and open-ended items, those who answered in a non-preferred mode seemed to take advantage of opportunities to satisfice when the question format allowed or encouraged it (e.g., selecting fewer items in the check-all than the forcedchoice format and being more likely to skip the open-ended item when it had a larger answer box), while those who answered in a preferred mode did not. There was no difference on a grid formatted item across those who did and did not respond by their preferred mode, but results indicate that a fully labeled grid reduced item missing rates vis-á-vis a grid with only column heading labels. Results provide insight into the effect of tailoring to mode preference on commonly used questionnaire design features.

KW - Check-all

KW - Data quality

KW - Forced-choice

KW - Grid questions

KW - Mail survey

KW - Mode preference

KW - Open-ends

KW - Web survey

UR - http://www.scopus.com/inward/record.url?scp=84916640731&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84916640731&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:84916640731

VL - 8

SP - 137

EP - 152

JO - Survey Research Methods

JF - Survey Research Methods

SN - 1864-3361

IS - 3

ER -