We are using HWRF's produtil Python package to ingest various datasets needed for our experiment.

Scripts and Configuration Files

Files are located in the util/gmtb_utils directory. The main script used to pull data from disk or the mass store is hwrf_pull_inputs.py. This script as well as the following configuration files can be found in the ush directory.

hwrf_input.conf

This configuration file is used to specify the disk location and filename conventions of your datasets. It is found in the parm directory. The [hwrf_data] section specifies the default naming conventions of the files you pull. We have customized the configuration files with our locations and naming conventions.

Input Tasks

The tables below show the three tasks we run in the workflow to ingest data and the corresponding configuration files, naming conventions, and disk locations.

Task Name	Task File	Purpose	Configuration File	Dataset
fcst_in_task	input_fcst.ent	Pull in SIGIO Initial Condition Files	gfs_fcst_input.pull	gfs_sanl, gfs_sfcanl
gfs_fcst_in_task	Input_gfsfcst.ent	Pull in GFS forecasts used for baseline	gfs_refcst_input.pull	gfs_gribA, gfs_gribB
vx_in_task	input_vx.ent	Pull in GDAS forecasts used for vx	gfs_vx_input.pull	gfs_pgrb_anl

Task Name	Production Archive Tar File	Production Filename	Parallel Filename	Output Directory
fcst_in_task	com_gfs_prod_gfs.yyyymmddhh.anl.tar	gfs.t00z.sanl, gfs.t00z.sfcanl	gfnanl.gfs.yyyymmddhh, sfnanl.gfs.yyyymmddhh	GMTB_HOME/chgres/input
gfs_fcst_in_task	com2_gfs_prod_gfs.yyyymmddhh.pgrb2_0p25.tar	gfs.t00z.pgrb2.0p25.fhhh	pgrbqfhhh.gfs.yyyymmddhh.grib2	ROTDIR/yyyymmddhh
vx_in_task	com2_gfs_prod_gdas.yyyymmddhh.tar	gdas1.t00z.pgrbanl	pgrbqanl.gdas.yyyymmddhh.grib2	GFS_ANL_OBS_DIR

Command to pull data:

 GMTB_UTIL/ush/hwrf_pull_inputs.py -v -w OUTPUT_DIR -o OUTPUT_DIR GMTB_UTIL/ush/PULL_CONF_FILE yyyymmddhh

      where GMTB_HOME        = /scratch4/BMC/gmtb/gmtb-tierIII
            ROTDIR           = GMTB_HOME/<model>
            GMTB_UTIL        = GMTB_HOME/util/gmtb_utils
            GFS_ANL_OBS_DIR  = /scratch3/BMC/dtc-hwrf/GMTB/vx_data/analyses/gfs_0p25/yyyymmddhh


             model = refcst, sasctrl, gftest