Reproducibility of Predictor Variables from a Validated Clinical Rule

P. S. Heckerling, R. C. Conant, Thomas Gerald Tape, Robert Swift Wigton

Research output: Contribution to journalArticle

16 Citations (Scopus)

Abstract

It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6% prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regres sion. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5%, 13.5%, 48.5%, 85.5%, 98.0%, and 100% of replicate models of sample size 500, respectively, met the criteria, whereas 0%, 0%, 16.5%, 49.0%, 71.5%, and 97.5% of models of sample size 250 met the criteria (all comparisons by sample size p <.0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5% of models of sample size 500, and for 75.3% of models of sample size 250. It is concluded that the most important predictor variables from a validated rule were included in the majority of replicate models, although the rule itself was reproduced in only a small minority of cases. In addition, the replicate models demonstrated good discriminatory power, and met statistical criteria for validation in an independent testing sample in a high percentage of cases. Key words: clinical prediction rules; replicate models; validation. (Med Decis Making 1992;12:280-285)

Original languageEnglish (US)
Pages (from-to)280-285
Number of pages6
JournalMedical Decision Making
Volume12
Issue number4
DOIs
StatePublished - Jan 1 1992

Fingerprint

Sample Size
Decision Support Techniques
Pneumonia

ASJC Scopus subject areas

  • Health Policy

Cite this

Reproducibility of Predictor Variables from a Validated Clinical Rule. / Heckerling, P. S.; Conant, R. C.; Tape, Thomas Gerald; Wigton, Robert Swift.

In: Medical Decision Making, Vol. 12, No. 4, 01.01.1992, p. 280-285.

Research output: Contribution to journalArticle

@article{f157112cfbe14d02aec55a7f0b3f175d,
title = "Reproducibility of Predictor Variables from a Validated Clinical Rule",
abstract = "It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6{\%} prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regres sion. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5{\%}, 13.5{\%}, 48.5{\%}, 85.5{\%}, 98.0{\%}, and 100{\%} of replicate models of sample size 500, respectively, met the criteria, whereas 0{\%}, 0{\%}, 16.5{\%}, 49.0{\%}, 71.5{\%}, and 97.5{\%} of models of sample size 250 met the criteria (all comparisons by sample size p <.0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5{\%} of models of sample size 500, and for 75.3{\%} of models of sample size 250. It is concluded that the most important predictor variables from a validated rule were included in the majority of replicate models, although the rule itself was reproduced in only a small minority of cases. In addition, the replicate models demonstrated good discriminatory power, and met statistical criteria for validation in an independent testing sample in a high percentage of cases. Key words: clinical prediction rules; replicate models; validation. (Med Decis Making 1992;12:280-285)",
author = "Heckerling, {P. S.} and Conant, {R. C.} and Tape, {Thomas Gerald} and Wigton, {Robert Swift}",
year = "1992",
month = "1",
day = "1",
doi = "10.1177/0272989X9201200406",
language = "English (US)",
volume = "12",
pages = "280--285",
journal = "Medical Decision Making",
issn = "0272-989X",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - Reproducibility of Predictor Variables from a Validated Clinical Rule

AU - Heckerling, P. S.

AU - Conant, R. C.

AU - Tape, Thomas Gerald

AU - Wigton, Robert Swift

PY - 1992/1/1

Y1 - 1992/1/1

N2 - It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6% prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regres sion. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5%, 13.5%, 48.5%, 85.5%, 98.0%, and 100% of replicate models of sample size 500, respectively, met the criteria, whereas 0%, 0%, 16.5%, 49.0%, 71.5%, and 97.5% of models of sample size 250 met the criteria (all comparisons by sample size p <.0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5% of models of sample size 500, and for 75.3% of models of sample size 250. It is concluded that the most important predictor variables from a validated rule were included in the majority of replicate models, although the rule itself was reproduced in only a small minority of cases. In addition, the replicate models demonstrated good discriminatory power, and met statistical criteria for validation in an independent testing sample in a high percentage of cases. Key words: clinical prediction rules; replicate models; validation. (Med Decis Making 1992;12:280-285)

AB - It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6% prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regres sion. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5%, 13.5%, 48.5%, 85.5%, 98.0%, and 100% of replicate models of sample size 500, respectively, met the criteria, whereas 0%, 0%, 16.5%, 49.0%, 71.5%, and 97.5% of models of sample size 250 met the criteria (all comparisons by sample size p <.0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5% of models of sample size 500, and for 75.3% of models of sample size 250. It is concluded that the most important predictor variables from a validated rule were included in the majority of replicate models, although the rule itself was reproduced in only a small minority of cases. In addition, the replicate models demonstrated good discriminatory power, and met statistical criteria for validation in an independent testing sample in a high percentage of cases. Key words: clinical prediction rules; replicate models; validation. (Med Decis Making 1992;12:280-285)

UR - http://www.scopus.com/inward/record.url?scp=0026669499&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0026669499&partnerID=8YFLogxK

U2 - 10.1177/0272989X9201200406

DO - 10.1177/0272989X9201200406

M3 - Article

VL - 12

SP - 280

EP - 285

JO - Medical Decision Making

JF - Medical Decision Making

SN - 0272-989X

IS - 4

ER -