Testing and Evaluation

Testing and evaluation undertaken by the developers of new NWP techniques from the research community is generally focused on case studies.  Extensive testing and evaluation must be performed to ensure that these new techniques are indeed ready for operational consideration.  Testing and evaluation by the DTC generally focuses on extended retrospective time periods.  The cases selected for these retrospective tests encapsulate a broad range of weather regimes ranging from null, to weak and strong events.  The exact periods chosen vary based on the type of phenomenon that is the focus of the test.  For some test activities, these cases will be chosen from all four seasons (e.g., extra-tropical for general predictions), whereas for others the cases will come from a particular season (e.g., hurricane season, convective season).  The design for each test is created in consultation with the developers, relevant area experts, and verification experts.  The DTC's evaluation of these retrospective forecasts includes standard verification techniques, as well as new verification techniques when appropriate. All verification statistics also undergo a statistical significance assessment.  The DTC currently conducts extensive testing and evaluation that focus on limited area mesoscale modeling, hurricane forecast models, data assimilation and ensemble systems.

By conducting carefully controlled, rigorous testing, including the generation of objective verification statistics, the DTC is able to provide the operational community with guidance for selecting new NWP technologies with potential value for operational implementation.  DTC testing also provides the research community with baselines against which the impacts of new techniques can be evaluated.  The statistical results may also aid researchers in selecting model configurations to use for their projects.