Embodied conversational agent-based kiosk for automated interviewing

Jay Nunamaker, Douglas C Derrick, Aaron Elkins, Judee Burgoon, Mark Patton

Research output: Contribution to journalArticle

89 Citations (Scopus)

Abstract

We have created an automated kiosk that uses embodied intelligent agents to interview individuals and detect changes in arousal, behavior, and cognitive effort by using psychophysiological information systems. In this paper, we describe the system and propose a unique class of intelligent agents, which are described as Special Purpose Embodied Conversational Intelligence with Environmental Sensors (SPECIES). SPECIES agents use heterogeneous sensors to detect human physiology and behavior during interactions, and they affect their environment by influencing human behavior using various embodied states (i.e., gender and demeanor), messages, and recommendations. Based on the SPECIES paradigm, we present three studies that evaluate different portions of the model, and these studies are used as foundational research for the development of the automated kiosk. The first study evaluates human-computer interaction and how SPECIES agents can change perceptions of information systems by varying appearance and demeanor. Instantiations that had the agents embodied as males were perceived as more powerful, while female embodied agents were perceived as more likable. Similarly, smiling agents were perceived as more likable than neutral demeanor agents. The second study demonstrated that a single sensor measuring vocal pitch provides SPECIES with environmental awareness of human stress and deception. The final study ties the first two studies together and demonstrates an avatar-based kiosk that asks questions and measures the responses using vocalic measurements.

Original languageEnglish (US)
Pages (from-to)17-48
Number of pages32
JournalJournal of Management Information Systems
Volume28
Issue number1
DOIs
StatePublished - Jul 1 2011

Fingerprint

Sensors
Intelligent agents
Information systems
Physiology
Human computer interaction
Interviewing
Sensor
Agent-based

Keywords

  • Avatars
  • NeuroIS
  • deception detection
  • embodied conversational agents

ASJC Scopus subject areas

  • Management Information Systems
  • Computer Science Applications
  • Management Science and Operations Research
  • Information Systems and Management

Cite this

Embodied conversational agent-based kiosk for automated interviewing. / Nunamaker, Jay; Derrick, Douglas C; Elkins, Aaron; Burgoon, Judee; Patton, Mark.

In: Journal of Management Information Systems, Vol. 28, No. 1, 01.07.2011, p. 17-48.

Research output: Contribution to journalArticle

Nunamaker, Jay ; Derrick, Douglas C ; Elkins, Aaron ; Burgoon, Judee ; Patton, Mark. / Embodied conversational agent-based kiosk for automated interviewing. In: Journal of Management Information Systems. 2011 ; Vol. 28, No. 1. pp. 17-48.
@article{6438a2363aa14e87816a6f866e5b4226,
title = "Embodied conversational agent-based kiosk for automated interviewing",
abstract = "We have created an automated kiosk that uses embodied intelligent agents to interview individuals and detect changes in arousal, behavior, and cognitive effort by using psychophysiological information systems. In this paper, we describe the system and propose a unique class of intelligent agents, which are described as Special Purpose Embodied Conversational Intelligence with Environmental Sensors (SPECIES). SPECIES agents use heterogeneous sensors to detect human physiology and behavior during interactions, and they affect their environment by influencing human behavior using various embodied states (i.e., gender and demeanor), messages, and recommendations. Based on the SPECIES paradigm, we present three studies that evaluate different portions of the model, and these studies are used as foundational research for the development of the automated kiosk. The first study evaluates human-computer interaction and how SPECIES agents can change perceptions of information systems by varying appearance and demeanor. Instantiations that had the agents embodied as males were perceived as more powerful, while female embodied agents were perceived as more likable. Similarly, smiling agents were perceived as more likable than neutral demeanor agents. The second study demonstrated that a single sensor measuring vocal pitch provides SPECIES with environmental awareness of human stress and deception. The final study ties the first two studies together and demonstrates an avatar-based kiosk that asks questions and measures the responses using vocalic measurements.",
keywords = "Avatars, NeuroIS, deception detection, embodied conversational agents",
author = "Jay Nunamaker and Derrick, {Douglas C} and Aaron Elkins and Judee Burgoon and Mark Patton",
year = "2011",
month = "7",
day = "1",
doi = "10.2753/MIS0742-1222280102",
language = "English (US)",
volume = "28",
pages = "17--48",
journal = "Journal of Management Information Systems",
issn = "0742-1222",
publisher = "M.E. Sharpe Inc.",
number = "1",

}

TY - JOUR

T1 - Embodied conversational agent-based kiosk for automated interviewing

AU - Nunamaker, Jay

AU - Derrick, Douglas C

AU - Elkins, Aaron

AU - Burgoon, Judee

AU - Patton, Mark

PY - 2011/7/1

Y1 - 2011/7/1

N2 - We have created an automated kiosk that uses embodied intelligent agents to interview individuals and detect changes in arousal, behavior, and cognitive effort by using psychophysiological information systems. In this paper, we describe the system and propose a unique class of intelligent agents, which are described as Special Purpose Embodied Conversational Intelligence with Environmental Sensors (SPECIES). SPECIES agents use heterogeneous sensors to detect human physiology and behavior during interactions, and they affect their environment by influencing human behavior using various embodied states (i.e., gender and demeanor), messages, and recommendations. Based on the SPECIES paradigm, we present three studies that evaluate different portions of the model, and these studies are used as foundational research for the development of the automated kiosk. The first study evaluates human-computer interaction and how SPECIES agents can change perceptions of information systems by varying appearance and demeanor. Instantiations that had the agents embodied as males were perceived as more powerful, while female embodied agents were perceived as more likable. Similarly, smiling agents were perceived as more likable than neutral demeanor agents. The second study demonstrated that a single sensor measuring vocal pitch provides SPECIES with environmental awareness of human stress and deception. The final study ties the first two studies together and demonstrates an avatar-based kiosk that asks questions and measures the responses using vocalic measurements.

AB - We have created an automated kiosk that uses embodied intelligent agents to interview individuals and detect changes in arousal, behavior, and cognitive effort by using psychophysiological information systems. In this paper, we describe the system and propose a unique class of intelligent agents, which are described as Special Purpose Embodied Conversational Intelligence with Environmental Sensors (SPECIES). SPECIES agents use heterogeneous sensors to detect human physiology and behavior during interactions, and they affect their environment by influencing human behavior using various embodied states (i.e., gender and demeanor), messages, and recommendations. Based on the SPECIES paradigm, we present three studies that evaluate different portions of the model, and these studies are used as foundational research for the development of the automated kiosk. The first study evaluates human-computer interaction and how SPECIES agents can change perceptions of information systems by varying appearance and demeanor. Instantiations that had the agents embodied as males were perceived as more powerful, while female embodied agents were perceived as more likable. Similarly, smiling agents were perceived as more likable than neutral demeanor agents. The second study demonstrated that a single sensor measuring vocal pitch provides SPECIES with environmental awareness of human stress and deception. The final study ties the first two studies together and demonstrates an avatar-based kiosk that asks questions and measures the responses using vocalic measurements.

KW - Avatars

KW - NeuroIS

KW - deception detection

KW - embodied conversational agents

UR - http://www.scopus.com/inward/record.url?scp=80051709642&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=80051709642&partnerID=8YFLogxK

U2 - 10.2753/MIS0742-1222280102

DO - 10.2753/MIS0742-1222280102

M3 - Article

VL - 28

SP - 17

EP - 48

JO - Journal of Management Information Systems

JF - Journal of Management Information Systems

SN - 0742-1222

IS - 1

ER -