In recent years transportation planners and engineers have begun to utilize traffic simulation models to estimate and forecast new transportation operations and reliability metrics. For example, the Highway Capacity Manual, Sixth Edition: A Guide for Multimodal Mobility Analysis (HCM-6) has recently adopted 1) passenger car estimation methods that are based on the microsimulation model VISSIM, and 2) urban arterial reliability estimation methods that are based on a Monte Carlos simulation technique. The advantage to simulation methods is that the metrics, which may be based on central tendency (e.g. mean, median), dispersion (variance, percentile), or even a combination of other metrics (e.g. reliability index), may be easily calculated and/or estimated. For this reason, the number of metrics developed and used has continued to increase. As one example, many researchers over the past decade have focused on developing and estimating metrics related to network reliability and resilience. However, it is an open research question on when and where these simulation approaches are appropriate to use. This paper will discuss a number of issues related to using simulation for estimating transportation metrics with a focus on model assumptions and model calibration. Specific examples from realworld test beds will be provided. Lastly, the paper will provide an overview of lessons learned and areas of future research.