### Abstract

It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6% prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regres sion. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5%, 13.5%, 48.5%, 85.5%, 98.0%, and 100% of replicate models of sample size 500, respectively, met the criteria, whereas 0%, 0%, 16.5%, 49.0%, 71.5%, and 97.5% of models of sample size 250 met the criteria (all comparisons by sample size p <.0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5% of models of sample size 500, and for 75.3% of models of sample size 250. It is concluded that the most important predictor variables from a validated rule were included in the majority of replicate models, although the rule itself was reproduced in only a small minority of cases. In addition, the replicate models demonstrated good discriminatory power, and met statistical criteria for validation in an independent testing sample in a high percentage of cases. Key words: clinical prediction rules; replicate models; validation. (Med Decis Making 1992;12:280-285)

Original language | English (US) |
---|---|

Pages (from-to) | 280-285 |

Number of pages | 6 |

Journal | Medical Decision Making |

Volume | 12 |

Issue number | 4 |

DOIs | |

State | Published - Dec 1992 |

### Fingerprint

### ASJC Scopus subject areas

- Health Policy

### Cite this

*Medical Decision Making*,

*12*(4), 280-285. https://doi.org/10.1177/0272989X9201200406

**Reproducibility of Predictor Variables from a Validated Clinical Rule.** / Heckerling, P. S.; Conant, R. C.; Tape, Thomas Gerald; Wigton, Robert Swift.

Research output: Contribution to journal › Article

*Medical Decision Making*, vol. 12, no. 4, pp. 280-285. https://doi.org/10.1177/0272989X9201200406

}

TY - JOUR

T1 - Reproducibility of Predictor Variables from a Validated Clinical Rule

AU - Heckerling, P. S.

AU - Conant, R. C.

AU - Tape, Thomas Gerald

AU - Wigton, Robert Swift

PY - 1992/12

Y1 - 1992/12

N2 - It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6% prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regres sion. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5%, 13.5%, 48.5%, 85.5%, 98.0%, and 100% of replicate models of sample size 500, respectively, met the criteria, whereas 0%, 0%, 16.5%, 49.0%, 71.5%, and 97.5% of models of sample size 250 met the criteria (all comparisons by sample size p <.0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5% of models of sample size 500, and for 75.3% of models of sample size 250. It is concluded that the most important predictor variables from a validated rule were included in the majority of replicate models, although the rule itself was reproduced in only a small minority of cases. In addition, the replicate models demonstrated good discriminatory power, and met statistical criteria for validation in an independent testing sample in a high percentage of cases. Key words: clinical prediction rules; replicate models; validation. (Med Decis Making 1992;12:280-285)

AB - It has been suggested that clinical prediction rules are not reproducible, and that the most important variables frequently do not appear in replicate models. The authors studied the reproducibility of a validated rule for predicting radiographic evidence of pneumonia (ROC areas for the training and validation cohorts, 0.816 and 0.821, respectively). Two hundred replicate samples of size 250 and size 500 were generated by sampling without replacement from the original training cohort of 905 patients with a 14.6% prevalence of pneumonia. Forward selection was performed among 31 candidate variables by stepwise logistic regres sion. Using as reproducibility criteria: 1) inclusion of all five variables from the original model in the original order; 2) inclusion of all five variables in any order; 3) inclusion of the first three variables; 4) inclusion of the first two variables; 5) inclusion of the first variable; and 6) inclusion of any of the five variables: 2.5%, 13.5%, 48.5%, 85.5%, 98.0%, and 100% of replicate models of sample size 500, respectively, met the criteria, whereas 0%, 0%, 16.5%, 49.0%, 71.5%, and 97.5% of models of sample size 250 met the criteria (all comparisons by sample size p <.0001 except for criteria 1 and 6, p = 0.07). Mean ROC areas in the training and validation samples were 0.829 and 0.791 for replicate models of sample size 500, and 0.831 and 0.779 for models of sample size 250. There was no significant difference in ROC areas between training and validation cohorts for 80.5% of models of sample size 500, and for 75.3% of models of sample size 250. It is concluded that the most important predictor variables from a validated rule were included in the majority of replicate models, although the rule itself was reproduced in only a small minority of cases. In addition, the replicate models demonstrated good discriminatory power, and met statistical criteria for validation in an independent testing sample in a high percentage of cases. Key words: clinical prediction rules; replicate models; validation. (Med Decis Making 1992;12:280-285)

UR - http://www.scopus.com/inward/record.url?scp=0026669499&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0026669499&partnerID=8YFLogxK

U2 - 10.1177/0272989X9201200406

DO - 10.1177/0272989X9201200406

M3 - Article

C2 - 1484477

AN - SCOPUS:0026669499

VL - 12

SP - 280

EP - 285

JO - Medical Decision Making

JF - Medical Decision Making

SN - 0272-989X

IS - 4

ER -