### Abstract

This paper establishes a negative result for clustering: above a certain ratio of random noise to nonrandom information, it is impossible for a large class of cost functions to distinguish between two partitions of a data set. In particular, it is shown that as the dimension increases, the ability to distinguish an accurate partitioning from an inaccurate one is lost unless the informative components are both sufficiently numerous and sufficiently informative. We examine squared error cost functions in detail. More generally, it is seen that the VC-dimension is an essential hypothesis for the class of cost functions to satisfy for an impossibility proof to be feasible. Separately, we provide bounds on the probabilistic behavior of cost functions that show how rapidly the ability to distinguish two clusterings decays. In two examples, one simulated and one with genomic data, bounds on the ability of squared-error and other cost functions to distinguish between two partitions are computed. Thus, one should not rely on clustering results alone for high dimensional low sample size data and one should do feature selection.

Original language | English (US) |
---|---|

Pages (from-to) | 30-53 |

Number of pages | 24 |

Journal | Statistical Analysis and Data Mining |

Volume | 4 |

Issue number | 1 |

DOIs | |

State | Published - Feb 1 2011 |

### Fingerprint

### Keywords

- Clustering impossibility
- Cost function
- High dimensions
- VC-dimension

### ASJC Scopus subject areas

- Analysis
- Information Systems
- Computer Science Applications

### Cite this

*Statistical Analysis and Data Mining*,

*4*(1), 30-53. https://doi.org/10.1002/sam.10095

**On the limits of clustering in high dimensions via cost functions.** / Koepke, Hoyt A.; Clarke, Bertrand S.

Research output: Contribution to journal › Article

*Statistical Analysis and Data Mining*, vol. 4, no. 1, pp. 30-53. https://doi.org/10.1002/sam.10095

}

TY - JOUR

T1 - On the limits of clustering in high dimensions via cost functions

AU - Koepke, Hoyt A.

AU - Clarke, Bertrand S.

PY - 2011/2/1

Y1 - 2011/2/1

N2 - This paper establishes a negative result for clustering: above a certain ratio of random noise to nonrandom information, it is impossible for a large class of cost functions to distinguish between two partitions of a data set. In particular, it is shown that as the dimension increases, the ability to distinguish an accurate partitioning from an inaccurate one is lost unless the informative components are both sufficiently numerous and sufficiently informative. We examine squared error cost functions in detail. More generally, it is seen that the VC-dimension is an essential hypothesis for the class of cost functions to satisfy for an impossibility proof to be feasible. Separately, we provide bounds on the probabilistic behavior of cost functions that show how rapidly the ability to distinguish two clusterings decays. In two examples, one simulated and one with genomic data, bounds on the ability of squared-error and other cost functions to distinguish between two partitions are computed. Thus, one should not rely on clustering results alone for high dimensional low sample size data and one should do feature selection.

AB - This paper establishes a negative result for clustering: above a certain ratio of random noise to nonrandom information, it is impossible for a large class of cost functions to distinguish between two partitions of a data set. In particular, it is shown that as the dimension increases, the ability to distinguish an accurate partitioning from an inaccurate one is lost unless the informative components are both sufficiently numerous and sufficiently informative. We examine squared error cost functions in detail. More generally, it is seen that the VC-dimension is an essential hypothesis for the class of cost functions to satisfy for an impossibility proof to be feasible. Separately, we provide bounds on the probabilistic behavior of cost functions that show how rapidly the ability to distinguish two clusterings decays. In two examples, one simulated and one with genomic data, bounds on the ability of squared-error and other cost functions to distinguish between two partitions are computed. Thus, one should not rely on clustering results alone for high dimensional low sample size data and one should do feature selection.

KW - Clustering impossibility

KW - Cost function

KW - High dimensions

KW - VC-dimension

UR - http://www.scopus.com/inward/record.url?scp=79551701149&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79551701149&partnerID=8YFLogxK

U2 - 10.1002/sam.10095

DO - 10.1002/sam.10095

M3 - Article

AN - SCOPUS:79551701149

VL - 4

SP - 30

EP - 53

JO - Statistical Analysis and Data Mining

JF - Statistical Analysis and Data Mining

SN - 1932-1872

IS - 1

ER -