 |
GMTB Workflow Documentation
|
|
|
GMTB Workflow Documentation
|
|
The Model Evaluation Tools (MET; DTC 2016) software package is used by the GMTB team to perform objective verification. MET is a state-of-the-art verification package that is freely available and supported by the DTC to the community. MET has tools to perform standard verification (e.g., RMSE and a variety of skill scores) as well as more advanced methods (e.g., object-based and neighborhood methods). The use of MET is relevant because it has been selected by NGGPS as the framework under which the current EMC, DTC, and NOAA/ESRL/GSD verification tools will be unified. A number of tools available in MET are employed to conduct grid-to-grid (Grid-stat) and grid-to-point (Point-stat) comparisons; the below sections provide an overview of the steps taken to perfom verification within the GMTB workflow. While a subset of MET configuration file settings used specifically for GMTB purposes will be described below, please note that all MET configuration files have a variety of user-defined settings and a full description regarding each option can be found in the MET Users’ Guide (DTC 2016).
The MET point-stat tool matches gridded forecast output to point observations. For GMTB, MET point-stat is used verify a variety of surface and upper-air fields against either the NAM Data Assimilation System (NDAS) or Global Data Assimilation System (GDAS) prepbufr point observations. There are two specific tasks that handle point-based verification within the GFS_VX.xml:
The two point verification tasks will be detailed in the following sub-sections. In addition, all verification tasks require a set of common environment variables to be set in the master source file, vx_env.ent, including:
The task to conduct point verification for surface fields in GFS_VX.xml is called met_point_sfc, which is under the verification_06hr metatask. When met_point_sfc is submitted it calls met_point_sfc.ent, which, in addition to sourcing the vx_env.ent environment variables mentioned above, it sets:
In order for the surface point verification task to be run by the workflow, the following dependencies are set in met_point_sfc.ent:
When all dependencies above are satisfied, the met_point_sfc.ent task calls the met_point_verf_sfc.ksh script.
The met_point_verf_sfc.ksh script first processes the prepbufr observation files using the pb2nc tool within MET. The pb2nc configuration file (PB2NCConfig_GMTB.vx) describes the directives on how to process the observation files. A subset defined for GMTB include:
The script then moves on to run point-stat for each verification domain using the point-stat configuration files PointStatConfig_ADPSFC_NDAS_REGRID and PointStatConfig_ADPSFC_MPR_NDAS_REGRID. Again, the configuration files have a number of user-defined settings; for GMTB, the following fields are set:
The MET point-stat task to conduct point verification for upper-air fields in GFS_VX.xml is called met_point_ua, which is under the verification_12hr metatask. When met_point_ua is submitted it calls met_point_ua.ent, which sources the vx_env.ent environment variables mentioned above and sets similar environment variables as the surface point task, including:
In order for the upper-air point verification task to be run by the workflow, the following dependencies are set in met_point_ua.ent:
When all dependencies above are satisfied, the met_point_ua.ent task calls the met_point_verf_ua.ksh script.
The met_point_verf_ua.ksh script first processes the prepbufr observation files using the pb2nc tool within MET, which may or may not differ from the observation files used for the surface point verification. In the case presented here, the GMTB uses the same pb2nc configuration file (PB2NCConfig_GMTB) for both surface and upper-air observation processing and the only difference is that the surface task processes NDAS prepbufr files, while the upper-air task processes GDAS prepbufr files.
The script then moves on to run point-stat for each verification domain using the point-stat configuration files PointStatConfig_ADPUPA_REGRID_G218 and PointStatConfig_ADPUPA_REGRID_G3. A subset of user-defined settings used within GMTB include:
The MET grid-stat tool matches gridded forecast output to gridded observations or analyses. For GMTB, MET grid-stat is used to verify a variety of gridded fields and the comparison against GFS analyses to calculate the 500 hPa anomaly correlation will be described in this section. The specific task in GFS_VX.xml is:
As a reminder, all verification tasks require a set of common environment variables to be set in the master source file, vx_env.ent, including:
The task to compute the anomaly correlation for 500 hPa heights in GFS_VX.xml is called met_grid_anl, which is under the verification_06hr metatask. When met_grid_anl is submitted it calls met_grid_anl.ent, where the common environment variables are sourced though the vx_env.ent and a few additional environment variables are set, including:
In order for the anomaly correlation verification task to be run by the workflow, the following dependencies are set in met_grid_anl.ent:
met_grid_anl.ent task should be executedWhen all dependencies above are satisfied, the met_grid_anl.ent task calls the met_grid_anl_verf.ksh script.
The met_grid_anl_verf.ksh script runs grid-stat for each verification domain using the grid-stat configuration file GridStatConfig_AC_REGRID. A subset of the user-defined settings for GMTB include:
The MET grid-stat tool matches a gridded forecast with a gridded forecast or analyses. For GMTB, MET grid-stat is used to verify quantitative precipitation forecasts (QPF). Built-in functionality in MET allows for the forecast and/or observations to be "regrid-on-the-fly" in order to get the forecast and observations on the same grid fore verification. There are three specific tasks that handle precipitation verification within the GFS_VX.xml:
All three tasks all call the same korn shell script that executes MET grid-stat; it is the environment variables and configuration files that allow for met_qpf_verf_all.ksh to run to specification for each individual task. All three QPF verification tasks require the following environment variables to be set to the vx_env.ent file, which is the file all verification-related ent files source because it contains several common environment variables:
For 6-hr precipitation verification over the CONUS, the Climatology-Calibrated Precipitation Analyses (CCPA) are used as “truth” and have been pulled from the NOAA HPSS. The CCPA files used for GMTB verification are files available every six hours and are in GRIB2 format. The native resolution of CCPA data is ~4.8 km.
The task for 6-hr precipitation verification in the GFS_VX.xml is called met_qpf_6hr, which is under the 6_acc metatask (i.e., metatask for 6-hr QPF). When met_qpf_6hr is submitted, it calls met_qpf_6hr.ent, which set the following environment variables:
In order for the 6-hr QPF task to be run by the workflow, the following dependencies are set in met_qpf_6hr.ent:
When all dependencies above are satisfied, the met_qpf_6hr.ent task calls the met_qpf_verf_all.ksh script, which uses the configuration file GridStatConfig_APCP06_REGRID_ccpa (need to get actual name; then link to sample config file). The configuration file has user-defined settings; for GMTB, the following fields are set:
The scripts first processes the observations and model output using the MET tool, pcp_combine which has capabilities to add and subtract fields to get necessary accumulations for verification
For 24-hr precipitation verification over the CONUS, the Climatology-Calibrated Precipitation Analyses (CCPA) are used as “truth” had have been pulled from the NOAA HPSS (add location). The CCPA files used for GMTB verification are files available every six hours and are in GRIB2 format. The native resolution of CCPA data is ??. For daily QPF verification, both over the CONUS and globe, the daily accumulations are set to be from 12 UTC – 12 UTC, which is consistent with how NCEP performs daily QPF verification. Given the files are in 6-hr intervals, a tool in MET, pcp-combine, is used to sum the individual 6-hr CCPA files to create a daily file. This step is accounted for in the met_qpf_verf_all.ksh, given all environment variables are properly set.
The met_qpf_24hr.ent task calls the met_qpf_verf_all.ksh script, which uses the configuration file GridStatConfig_APCP24_REGRID_ccpa, which set the following environment variables:
In order for the 24-hr CONUS QPF task to be run by the workflow, the following dependencies are set in met_qpf_24hr.ent:
When all dependencies above are satisfied, the met_qpf_24hr.ent task calls the met_qpf_verf_all.ksh script, which uses the configuration file GridStatConfig_APCP24_REGRID_cmorph (need to get actual name). The configuration file has user-defined settings; for GMTB, the following fields are set:
The scripts first processes the observations and model output using the MET tool, pcp_combine which has capabilities to add and subtract fields to get necessary accumulations for verification
For 24-hr precipitation verification over the globe, the CPC MORPHing technique (CMORPH; Joyce et al. 2004) data are used as “truth” had have been pulled from a NOAA ftp site. A simple pre-processing script was used convert the hourly data from binary to NetCDF (should we provide this? – need to get from dakota). The CMORPH files used in GMTB verification are files available both hourly and daily and are in NetCDF format. The native resolution of CMORPH data is 8-km, and it covers the globe from ~60°S - ~60°N. Also, CMORPH has known biases over land; therefore, verification was done over land+sea and sea only.
The met_qpf_global_24hr.ent task calls the met_qpf_verf_all.ksh script, which uses the configuration file GridStatConfig_APCP24_REGRID_cmorph, which set the following environment variables:
In order for the met_qpf_global_24hr.ent task to be run by the workflow, the following dependencies are set in met_qpf_global_24hr.ent:
When all dependencies above are satisfied, the met_qpf_global_24hr.ent task calls the met_qpf_verf_all.ksh script, which uses the configuration file GridStatConfig_APCP24_REGRID_cmorph. The configuration file has user-defined settings; for GMTB, the following fields are set:
Developmental Testbed Center, 2016: MET: Version 5.2 Model Evaluation Tools Users' Guide. Available at: http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v5.2.pdf 328 pp.
Joyce, R. J., J. E. Janowiak, P. A. Arkin, and P. Xie, 2004: CMORPH: A method that produces global precipitation estimates from passive microwave and infrared data at high spatial and temporal resolution.. J. Hydromet., 5, 487-503.
1.8.10