GMTB Workflow Documentation
Data Ingest Component

We are using HWRF's produtil Python package to ingest various datasets needed for our experiment.

Scripts and Configuration Files

Files are located in the util/gmtb_utils directory. The main script used to pull data from disk or the mass store is hwrf_pull_inputs.py. This script as well as the following configuration files can be found in the ush directory.

hwrf_input.conf

This configuration file is used to specify the disk location and filename conventions of your datasets. It is found in the parm directory. The [hwrf_data] section specifies the default naming conventions of the files you pull. We have customized the configuration files with our locations and naming conventions.

Input Tasks

The tables below show the three tasks we run in the workflow to ingest data and the corresponding configuration files, naming conventions, and disk locations.

Task Name Task File Purpose Configuration File Dataset
fcst_in_task input_fcst.ent Pull in SIGIO Initial Condition Files gfs_fcst_input.pull gfs_sanl, gfs_sfcanl
gfs_fcst_in_task Input_gfsfcst.ent Pull in GFS forecasts used for baseline gfs_refcst_input.pull gfs_gribA, gfs_gribB
vx_in_task input_vx.ent Pull in GDAS forecasts used for vx gfs_vx_input.pull gfs_pgrb_anl


Task Name Production Archive Tar File Production Filename Parallel Filename Output Directory
fcst_in_task com_gfs_prod_gfs.yyyymmddhh.anl.tar gfs.t00z.sanl, gfs.t00z.sfcanl gfnanl.gfs.yyyymmddhh, sfnanl.gfs.yyyymmddhh GMTB_HOME/chgres/input
gfs_fcst_in_task com2_gfs_prod_gfs.yyyymmddhh.pgrb2_0p25.tar gfs.t00z.pgrb2.0p25.fhhh pgrbqfhhh.gfs.yyyymmddhh.grib2 ROTDIR/yyyymmddhh
vx_in_task com2_gfs_prod_gdas.yyyymmddhh.tar gdas1.t00z.pgrbanl pgrbqanl.gdas.yyyymmddhh.grib2 GFS_ANL_OBS_DIR

Command to pull data:

 GMTB_UTIL/ush/hwrf_pull_inputs.py -v -w OUTPUT_DIR -o OUTPUT_DIR GMTB_UTIL/ush/PULL_CONF_FILE yyyymmddhh

      where GMTB_HOME        = /scratch4/BMC/gmtb/gmtb-tierIII
            ROTDIR           = GMTB_HOME/<model>
            GMTB_UTIL        = GMTB_HOME/util/gmtb_utils
            GFS_ANL_OBS_DIR  = /scratch3/BMC/dtc-hwrf/GMTB/vx_data/analyses/gfs_0p25/yyyymmddhh


             model = refcst, sasctrl, gftest