Do interviewer postsurvey evaluations of respondents' engagement measure who respondents are or what they do? A behavior coding study

Research output: Contribution to journalReview article

3 Citations (Scopus)

Abstract

Survey interviewers are often tasked with assessing the quality of respondents' answers after completing a survey interview. These interviewer observations have been used to proxy for measurement error in interviewer-administered surveys. How interviewers formulate these evaluations and how well they proxy for measurement error has received little empirical attention. According to dual-process theories of impression formation, individuals form impressions about others based on the social categories of the observed person (e.g., sex, race) and individual behaviors observed during an interaction. Although initial impressions start with heuristic, rule-of-thumb evaluations, systematic processing is characterized by extensive incorporation of available evidence. In a survey context, if interviewers default to heuristic information processing when evaluating respondent engagement, then we expect their evaluations to be primarily based on respondent characteristics and stereotypes associated with those characteristics. Under systematic processing, on the other hand, interviewers process and evaluate respondents based on observable respondent behaviors occurring during the question-answering process. We use the Work and Leisure Today Survey, including survey data and behavior codes, to examine proxy measures of heuristic and systematic processing by interviewers as predictors of interviewer postsurvey evaluations of respondents' cooperativeness, interest, friendliness, and talkativeness. Our results indicate that CATI interviewers base their evaluations on actual behaviors during an interview (i.e., systematic processing) rather than perceived characteristics of the respondent or the interviewer (i.e., heuristic processing). These results are reassuring for the many surveys that collect interviewer observations as proxies for data quality.

Original languageEnglish (US)
Pages (from-to)817-846
Number of pages30
JournalPublic Opinion Quarterly
Volume81
Issue number4
DOIs
StatePublished - Jan 1 2017

Fingerprint

coding
interview
evaluation
Processing
Measurement errors
heuristics
Evaluation
attitude formation
Heuristics
data quality
information processing
stereotype
Measurement Error
human being

ASJC Scopus subject areas

  • Communication
  • History
  • Sociology and Political Science
  • Social Sciences(all)
  • History and Philosophy of Science

Cite this

@article{ab3fc1fadc61454a836cc16112591289,
title = "Do interviewer postsurvey evaluations of respondents' engagement measure who respondents are or what they do? A behavior coding study",
abstract = "Survey interviewers are often tasked with assessing the quality of respondents' answers after completing a survey interview. These interviewer observations have been used to proxy for measurement error in interviewer-administered surveys. How interviewers formulate these evaluations and how well they proxy for measurement error has received little empirical attention. According to dual-process theories of impression formation, individuals form impressions about others based on the social categories of the observed person (e.g., sex, race) and individual behaviors observed during an interaction. Although initial impressions start with heuristic, rule-of-thumb evaluations, systematic processing is characterized by extensive incorporation of available evidence. In a survey context, if interviewers default to heuristic information processing when evaluating respondent engagement, then we expect their evaluations to be primarily based on respondent characteristics and stereotypes associated with those characteristics. Under systematic processing, on the other hand, interviewers process and evaluate respondents based on observable respondent behaviors occurring during the question-answering process. We use the Work and Leisure Today Survey, including survey data and behavior codes, to examine proxy measures of heuristic and systematic processing by interviewers as predictors of interviewer postsurvey evaluations of respondents' cooperativeness, interest, friendliness, and talkativeness. Our results indicate that CATI interviewers base their evaluations on actual behaviors during an interview (i.e., systematic processing) rather than perceived characteristics of the respondent or the interviewer (i.e., heuristic processing). These results are reassuring for the many surveys that collect interviewer observations as proxies for data quality.",
author = "Antje Kirchner and Kristen Olson and Smyth, {Jolene D.}",
year = "2017",
month = "1",
day = "1",
doi = "10.1093/poq/nfx026",
language = "English (US)",
volume = "81",
pages = "817--846",
journal = "Public Opinion Quarterly",
issn = "0033-362X",
publisher = "Oxford University Press",
number = "4",

}

TY - JOUR

T1 - Do interviewer postsurvey evaluations of respondents' engagement measure who respondents are or what they do? A behavior coding study

AU - Kirchner, Antje

AU - Olson, Kristen

AU - Smyth, Jolene D.

PY - 2017/1/1

Y1 - 2017/1/1

N2 - Survey interviewers are often tasked with assessing the quality of respondents' answers after completing a survey interview. These interviewer observations have been used to proxy for measurement error in interviewer-administered surveys. How interviewers formulate these evaluations and how well they proxy for measurement error has received little empirical attention. According to dual-process theories of impression formation, individuals form impressions about others based on the social categories of the observed person (e.g., sex, race) and individual behaviors observed during an interaction. Although initial impressions start with heuristic, rule-of-thumb evaluations, systematic processing is characterized by extensive incorporation of available evidence. In a survey context, if interviewers default to heuristic information processing when evaluating respondent engagement, then we expect their evaluations to be primarily based on respondent characteristics and stereotypes associated with those characteristics. Under systematic processing, on the other hand, interviewers process and evaluate respondents based on observable respondent behaviors occurring during the question-answering process. We use the Work and Leisure Today Survey, including survey data and behavior codes, to examine proxy measures of heuristic and systematic processing by interviewers as predictors of interviewer postsurvey evaluations of respondents' cooperativeness, interest, friendliness, and talkativeness. Our results indicate that CATI interviewers base their evaluations on actual behaviors during an interview (i.e., systematic processing) rather than perceived characteristics of the respondent or the interviewer (i.e., heuristic processing). These results are reassuring for the many surveys that collect interviewer observations as proxies for data quality.

AB - Survey interviewers are often tasked with assessing the quality of respondents' answers after completing a survey interview. These interviewer observations have been used to proxy for measurement error in interviewer-administered surveys. How interviewers formulate these evaluations and how well they proxy for measurement error has received little empirical attention. According to dual-process theories of impression formation, individuals form impressions about others based on the social categories of the observed person (e.g., sex, race) and individual behaviors observed during an interaction. Although initial impressions start with heuristic, rule-of-thumb evaluations, systematic processing is characterized by extensive incorporation of available evidence. In a survey context, if interviewers default to heuristic information processing when evaluating respondent engagement, then we expect their evaluations to be primarily based on respondent characteristics and stereotypes associated with those characteristics. Under systematic processing, on the other hand, interviewers process and evaluate respondents based on observable respondent behaviors occurring during the question-answering process. We use the Work and Leisure Today Survey, including survey data and behavior codes, to examine proxy measures of heuristic and systematic processing by interviewers as predictors of interviewer postsurvey evaluations of respondents' cooperativeness, interest, friendliness, and talkativeness. Our results indicate that CATI interviewers base their evaluations on actual behaviors during an interview (i.e., systematic processing) rather than perceived characteristics of the respondent or the interviewer (i.e., heuristic processing). These results are reassuring for the many surveys that collect interviewer observations as proxies for data quality.

UR - http://www.scopus.com/inward/record.url?scp=85044274846&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85044274846&partnerID=8YFLogxK

U2 - 10.1093/poq/nfx026

DO - 10.1093/poq/nfx026

M3 - Review article

AN - SCOPUS:85044274846

VL - 81

SP - 817

EP - 846

JO - Public Opinion Quarterly

JF - Public Opinion Quarterly

SN - 0033-362X

IS - 4

ER -