MD files can have a variety of shapes. A schematic depiction of a general MD file is shown in:
The exact shape of an MD file is determined by its schema. In order to specify which schema to use when creating an MD file, on must first:
To illustrate the process of importing textual observation data into McIDAS, we detail the process for a dataset we grabbed from the CODIAC data management system that is maintained by the Joint Office for Field Support (JOSS) program of UCAR.
First, we need to grab the dataset. We do this from the Unix shell prompt:
mkdir /data/workshop/comet cd /data/workshop/comet ftp ftp.unidata.ucar.edu <user> umcidas <pass> uni_mci! cd data/textdata binary get STORM.TXT quit
After grabbing the dataset, we need to make sure that it is accessible from our McIDAS session. We do this by adding a file REDIRECTion to the file, STORM.TXT, which is located in the /var/data/mcidas directory:
SEE STORM.TXT NLINES=10 TYPE MOD DAY TIME ID LAT LON ZS PSL PRE T TD SPD DIR PCP UNIT=X X X X X X X X X X C C X X X 0 0 95080 00 NEW 30.03 90.03 3. 1015.90 1016.00 21.75 17.25 3.60 15 0.00 -9999 0 0 95080 00 LCH 30.12 93.22 5. 1010.90 1011.00 22.25 19.45 8.20 15 0.00 -9999 0 0 95080 00 LFT 30.20 91.98 11. 1013.20 1014.20 22.85 18.35 5.10 15 0.00 -9999 0 0 95080 00 AUS 30.28 97.70 179. 987.60 1007.70 22.85 18.35 6.70 13 SEE: DoneThe correspondence in names and column entries is such that the first column refers to a Key named TYPE, the second MOD, the third DAY, etc. The ordering of the parameters is not important as long as each line in the file has the same correspondence with the Key names.
The second line, the line that begins UNIT=, is optional. If included it defines the units of the parameters. This line is needed when the units of the input data is different than the units that are to be stored in the MD file. One word of warning is in order: McIDAS implicitly expects data to be stored in American meteorological convention units.
From the listing, you can see that the parameters identified as T and TD (temperature and dew point, respectively), are given in degrees Centigrade. Since temperatures will be stored in the output MD file in K, we need to alert TXT2MD to do the necessary conversion.
The naming of the keys is not random and is not free-form. The Key names listed must match those that will be stored in the MD file. Furthermore, MD file Key names must be no more than four characters in lenght.
To understand how we chose the key names listed, we must look at the file schema that will be used to create the file. In McIDAS file schemas are stored in so-called LW files. LW files are files with fixed, 80-character records that are not separated with line feeds. The naming convention for McIDAS schema files is DCxxxx. Here, xxxx refers to a name that is four or less characters in length.
Since the data set contained in STORM.TXT is surface data taken from a mesonet, we will use the McIDAS surface MD file form to store our data. The current schema for McIDAS surface data (such as that created by XCD decoders is named ISFC. Its template is contained in the file DCISFC which is distributed in the /home/mcidas/data directory.
Let's take a look at the ISFC template file:
" ISFC (JSP) 0289 SCHEMA -- GENERAL SURFACE SCHEMA " NAME VSN DATE ID "TEXTID " ---- --- ----- -- ------- SCHEMA ISFC 7 99085 0 "SURFACE HOURLY OBSERVATIONS ROWS 72 "24 HOURS, 3 REPORTS/HOUR TYPE "0=HOURLY,1=SPECIAL 1,2=SPECIAL 2 DAY CYD "YEAR AND JULIAN DAY (CCYYDDD) TIME HMS "NOMINAL TIME (HH0000) NREC "APPROX. # OF RECORDS WRITTEN IN THIS ROW COLUMNS 3500 "MAXIMUM OF 3500 REPORTING STATIONS ID CHAR "STATION CALL LETTERS LAT 4 DEG "RANGE (-90 => +90) POSITIVE NORTH LON 4 DEG "RANGE (-180 => +180) POSITIVE WEST ZS M "ELEVATION ST CHAR "STATE ID CO CHAR "COUNTRY ID DATA MOD "USER MODIFICATION FLAG (1=MODIFIED) HMS HMS "ACTUAL TIME OF OBSERVATION CIGC "CEILING CLOUD COVER (0=CLEAR,1=SCATTERED, CC1 "FIRST NON-CEILING 2=BROKEN,3=OVERCAST) CC2 "SECOND NON-CEILING CLOUD COVER CIGH -2 FT "HEIGHT OF CLOUD CEILING ZCL1 -2 FT "HEIGHT OF 1ST (LOWEST) NON-CEILING CLOUD LAYER ZCL2 -2 FT "HEIGHT OF 2ND NON-CEILING CLOUD LAYER VIS 1 MI "VISIBILITY WX1 CHAR "WEATHER TYPE (FIRST FOUR CHARACTERS) WX2 CHAR " (NEXT FOUR CHARACTERS) T 2 K "TEMPERATURE TD 2 K "DEW POINT TEMPERATURE DIR DEG "WIND DIRECTION SPD 1 MPS " SPEED GUS 1 MPS " GUSTS PSL 2 MB "ALTIMETER SETTING REDUCED TO SEA LEVEL BY STD ATM PCP 2 IN "3 HOURLY PRECIPITATION TOTAL SNO IN "CUMULATIVE SNOW DEPTH PRE 2 MB "SEA LEVEL PRESSURE P24 2 IN "24 HOUR PRECIPITATION TOTAL WXC1 CODE "WMO WEATHER CODE 1 WXC2 CODE "WMO WEATHER CODE 2 WXC3 CODE "WMO WEATHER CODE 3 WXC4 CODE "WMO WEATHER CODE 4 ENDSCHEMA SEE: Done...EOF Encountered.The listing illustrates the set of Key names include in the file. You should notice that the number of Keys in the schema template is greater than those specified in the data we are going to import. The parameters that are missing from the data set will be filed in the MD as MISSing values.
The second column in the schema tells us the power of 10 that the data value will be multiplied by before it is written into the MD file. All numeric data are stored as integers in McIDAS data files.
The third column specifies the unit of the data values as they will be stored in the MD file. You will see that temperatures T and TD will be stored/are expected to be stored in K.
Any information including and following the " is treated as a comment.
As was mentioned earlier, the format for the schema template file is 80-character, records that are not separated by carriage returns or line feeds. This means that this file type is not editable by a regular ASCII text editor (such as vi or emacs). McIDAS provides a pair of conversion utilities that are used to convert from LW files to ASCII text and back again: LWODOS and DOSTOLW. We talk more about this later.
In order to use this or any other schema, TXT2MD needs it stored in the global schema repository file, SCHEMA. SCHEMA must be locatable in your session, how would you verify that it is?
To see if the schema that you wish to use with TXT2MD is available for use, you need to interrogate SCHEMA:
NAME: ISFC VERSION: 7 DATE: 1999085 TEXTID: "SURFACE HOURLY OBSERVATIONS ---- ------- DEFAULT NUMBER OF ROWS: 72 INTEGER ID: 0 COLS: 3500 MISSING DATA VALUE: -2139062144 REPEAT GROUP: NUMBER OF REPETITIONS: 1 STARTING POSITION: 11 SIZE: 25 NUMBER OF KEYS IN ROW HEADER: 4 COL HEADER: 6 STARTING AT POSITION 5 DATA RECORD: 25 STARTING AT POSITION 11 ---- 35 TOTAL KEY SCALE UNIT KEY SCALE UNIT KEY SCALE UNIT ---- ----- ---- ---- ----- ---- ---- ----- ---- TYPE 0 DAY 0 CYD TIME 0 HMS NREC 0 ID 0 CHAR LAT 4 DEG LON 4 DEG ZS 0 M ST 0 CHAR CO 0 CHAR MOD 0 HMS 0 HMS CIGC 0 CC1 0 CC2 0 CIGH -2 FT ZCL1 -2 FT ZCL2 -2 FT VIS 1 MI WX1 0 CHAR WX2 0 CHAR T 2 K TD 2 K DIR 0 DEG SPD 1 MPS GUS 1 MPS PSL 2 MB PCP 2 IN SNO 0 IN PRE 2 MB P24 2 IN WXC1 0 CODE WXC2 0 CODE WXC3 0 CODE WXC4 0 CODE LSCHE: DONEyou are ready to proceed. If not, then you need to register the ISFC schema in your SCHEMA file:
Group Name Server IP Address -------------------- ---------------------------------------- BLIZZARD <LOCAL-DATA> MYDATA <LOCAL-DATA> RTGRIDS ADDE.UCAR.EDU RTIMAGES ADDE.UCAR.EDU RTNEXRAD ADDE.UCAR.EDU NEXRCOMP ADDE.UCAR.EDU RTPTSRC ADDE.UCAR.EDU RTWXTEXT ADDE.UCAR.EDU TOPO CTTCSUN.UNIDATA.UCAR.EDU <LOCAL-DATA> indicates that data will be accessed from the local data directory. DATALOC -- doneThe dataset of interest for this exercise is MYDATA. Let's take a closer look:
Dataset Names of Type: POINT in Group: MYDATA Name NumPos Content ------------ ------ -------------------------------------- PTSRCS 9999 All MDXX files DSINFO -- doneThe file we created, MDXX1000, will occupy the 1000th position in this dataset. We can, therefore, refer to it directly as MYDATA/PTSRCS.1000. We use this identifier in a number of commands to list out information from/about the data:
Plot the same data in Farenheit:
Since the mesonet was located in the central US, lets focus our attention there, plot the temperature for 3 UTC, and reduce the size of the plotted values:
Finally, plot the temperature data as integers over the same region:
You will notice that the MD file created by the above process was rather large give the amount of data that was imported. The reason for this is the schema that was used in its creation. In McIDAS you do not have to choose an existing schema when importing data. Instead, you can create your own, customized schema that is tailored to the data that you want to work with. This is especially important if the data that you want to import doesn't match any existing schema. A good example of this was the NLDN lightning data that Unidata sites can get free-of-charge from the State University of New York at Albany.
Let's repeat the exercise of importing the data in STORM.TXT, but this time, let's create a custom schema to use. Since the ISFC schema listed previously was a pretty good match for our data, let's use it as a starting point.
First, make an ASCII text copy of the schema repository file DCISFC From the McIDAS command mode:
" JSFC (TCY) 0289 SCHEMA -- GENERAL SURFACE SCHEMA " NAME VSN DATE ID "TEXTID " ---- --- ----- -- ------- SCHEMA JSFC 1 98264 0 "CODIAC MESONET OBSERVATIONS ROWS 24 "24 HOURS, 3 REPORTS/HOUR TYPE "0=HOURLY,1=SPECIAL 1,2=SPECIAL 2 DAY SYD "YEAR AND JULIAN DAY (YYDDD) TIME HMS "NOMINAL TIME (HH0000) NREC "APPROX. # OF RECORDS WRITTEN IN THIS ROW COLUMNS 500 "MAXIMUM OF 3500 REPORTING STATIONS ID CHAR "STATION CALL LETTERS LAT 4 DEG "RANGE (-90 => +90) POSITIVE NORTH LON 4 DEG "RANGE (-180 => +180) POSITIVE WEST ZS M "ELEVATION DATA MOD "USER MODIFICATION FLAG (1=MODIFIED) T 2 K "TEMPERATURE TD 2 K "DEW POINT TEMPERATURE DIR DEG "WIND DIRECTION SPD 1 MPS " SPEED PSL 2 MB "ALTIMETER SETTING PCP 2 IN "3 HOURLY PRECIPITATION TOTAL PRE 2 MB "SEA LEVEL PRESSURE ENDSCHEMANotice that we have modified a few entries in the file and deleted a number of items that are not needed.
The next job is to turn the ASCII file back into a McIDAS LW file:
-rw-rw-r-- 1 mcidas ustaff 5734256 Sep 21 09:17 MDXX1000 -rw-rw-r-- 1 mcidas ustaff 357856 Sep 21 09:25 MDXX1001See how much smaller the new file is? Rerun the list and display commands presented earlier to verify that the information in the second, smaller MD file is the same as in the first.