Measuring Lineup Difficulty By Matching Distance Metrics With Subject Choices in Crowd-Sourced Data

Niladri Roy Chowdhury, Dianne Cook, Heike Hofmann, Mahbubul Majumder

Research output: Contribution to journalArticle

Abstract

Graphics play a crucial role in statistical analysis and data mining. Being able to quantify structure in data that is visible in plots, and how people read the structure from plots is an ongoing challenge. The lineup protocol provides a formal framework for data plots, making inference possible. The data plot is treated like a test statistic, and lineup protocol acts like a comparison with the sampling distribution of the nulls. This article describes metrics for describing structure in data plots and evaluates them in relation to the choices that human readers made during several large Amazon Turk studies using lineups. The metrics that were more specific to the plot types tended to better match subject choices, than generic metrics. The process that we followed to evaluate metrics will be useful for general development of numerically measuring structure in plots, and also in future experiments on lineups for choosing blocks of pictures. Supplementary materials for this article are available online.

Original languageEnglish (US)
Pages (from-to)132-145
Number of pages14
JournalJournal of Computational and Graphical Statistics
Volume27
Issue number1
DOIs
StatePublished - Jan 2 2018

Fingerprint

Distance Metric
Metric
Sampling Distribution
Evaluate
Test Statistic
Statistical Analysis
Data Mining
Quantify
Experiment

Keywords

  • Cognitive perception
  • Data mining
  • Data science
  • Data visualization
  • Distance metrics
  • Exploratory data analysis
  • Information visualization
  • Statistical graphics
  • Visual inference

ASJC Scopus subject areas

  • Statistics and Probability
  • Discrete Mathematics and Combinatorics
  • Statistics, Probability and Uncertainty

Cite this

Measuring Lineup Difficulty By Matching Distance Metrics With Subject Choices in Crowd-Sourced Data. / Chowdhury, Niladri Roy; Cook, Dianne; Hofmann, Heike; Majumder, Mahbubul.

In: Journal of Computational and Graphical Statistics, Vol. 27, No. 1, 02.01.2018, p. 132-145.

Research output: Contribution to journalArticle

@article{8b512bdac26e498b9da5e1c562c779aa,
title = "Measuring Lineup Difficulty By Matching Distance Metrics With Subject Choices in Crowd-Sourced Data",
abstract = "Graphics play a crucial role in statistical analysis and data mining. Being able to quantify structure in data that is visible in plots, and how people read the structure from plots is an ongoing challenge. The lineup protocol provides a formal framework for data plots, making inference possible. The data plot is treated like a test statistic, and lineup protocol acts like a comparison with the sampling distribution of the nulls. This article describes metrics for describing structure in data plots and evaluates them in relation to the choices that human readers made during several large Amazon Turk studies using lineups. The metrics that were more specific to the plot types tended to better match subject choices, than generic metrics. The process that we followed to evaluate metrics will be useful for general development of numerically measuring structure in plots, and also in future experiments on lineups for choosing blocks of pictures. Supplementary materials for this article are available online.",
keywords = "Cognitive perception, Data mining, Data science, Data visualization, Distance metrics, Exploratory data analysis, Information visualization, Statistical graphics, Visual inference",
author = "Chowdhury, {Niladri Roy} and Dianne Cook and Heike Hofmann and Mahbubul Majumder",
year = "2018",
month = "1",
day = "2",
doi = "10.1080/10618600.2017.1356323",
language = "English (US)",
volume = "27",
pages = "132--145",
journal = "Journal of Computational and Graphical Statistics",
issn = "1061-8600",
publisher = "American Statistical Association",
number = "1",

}

TY - JOUR

T1 - Measuring Lineup Difficulty By Matching Distance Metrics With Subject Choices in Crowd-Sourced Data

AU - Chowdhury, Niladri Roy

AU - Cook, Dianne

AU - Hofmann, Heike

AU - Majumder, Mahbubul

PY - 2018/1/2

Y1 - 2018/1/2

N2 - Graphics play a crucial role in statistical analysis and data mining. Being able to quantify structure in data that is visible in plots, and how people read the structure from plots is an ongoing challenge. The lineup protocol provides a formal framework for data plots, making inference possible. The data plot is treated like a test statistic, and lineup protocol acts like a comparison with the sampling distribution of the nulls. This article describes metrics for describing structure in data plots and evaluates them in relation to the choices that human readers made during several large Amazon Turk studies using lineups. The metrics that were more specific to the plot types tended to better match subject choices, than generic metrics. The process that we followed to evaluate metrics will be useful for general development of numerically measuring structure in plots, and also in future experiments on lineups for choosing blocks of pictures. Supplementary materials for this article are available online.

AB - Graphics play a crucial role in statistical analysis and data mining. Being able to quantify structure in data that is visible in plots, and how people read the structure from plots is an ongoing challenge. The lineup protocol provides a formal framework for data plots, making inference possible. The data plot is treated like a test statistic, and lineup protocol acts like a comparison with the sampling distribution of the nulls. This article describes metrics for describing structure in data plots and evaluates them in relation to the choices that human readers made during several large Amazon Turk studies using lineups. The metrics that were more specific to the plot types tended to better match subject choices, than generic metrics. The process that we followed to evaluate metrics will be useful for general development of numerically measuring structure in plots, and also in future experiments on lineups for choosing blocks of pictures. Supplementary materials for this article are available online.

KW - Cognitive perception

KW - Data mining

KW - Data science

KW - Data visualization

KW - Distance metrics

KW - Exploratory data analysis

KW - Information visualization

KW - Statistical graphics

KW - Visual inference

UR - http://www.scopus.com/inward/record.url?scp=85041561007&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041561007&partnerID=8YFLogxK

U2 - 10.1080/10618600.2017.1356323

DO - 10.1080/10618600.2017.1356323

M3 - Article

AN - SCOPUS:85041561007

VL - 27

SP - 132

EP - 145

JO - Journal of Computational and Graphical Statistics

JF - Journal of Computational and Graphical Statistics

SN - 1061-8600

IS - 1

ER -