Previous: Importing Foreign POINT Datasets Into McIDAS Next: McIDAS-X Useful Information Table of contents Index Glossary Images Frames ITC-NGIC McIDAS-X Training Workshop > Applying Lessons Learned > Importing Foreign Datasets Into McIDAS

15.7.1 Importing Foreign IMAGE Datasets Into McIDAS

Binary Image Source Data

Image source data in binary format may be imported into a McIDAS-compatible data file (AREAnnnn) using the routine MAKEAREA. Detailed information on AREA files may be found in AREA Files - Basic Concepts.

To illustrate the process of importing foreign image product data into McIDAS, we use a dataset from the EARS website as produced by their Energy & Water Balance Monitoring System (EWBMS). For this exercise we may use any of the following EWBMS products:

To save time, this exercise focuses on only one of the EARS's CMI products (year 1999). However, the procedure detailed in this exercise remains more or less the same for the other EWBMS products and years. In short, we first have to make the foreign dataset accessable from our mcidas-sessions (by copying, downloading, and/or uncompressing it), make a link to it so it can be read by MAKEAREA, import it into an AREA file, and then add a navigation (projection).

Here we go:

First, we need to grab the dataset from the ftp-site of EARS. We do this from the Unix shell prompt aka terminal (click Applications>Accessories>Terminal).

Type:

     mkdir ~/workdata/EWBMS/
     cd ~/workdata/EWBMS/

     ftp home.ears.nl
       <user> ngic
       <pass> ub0708
       cd NGIC/ITC-EARS/example\ EWBMS\ products/ClimaticMoistureIndex
       binary
       get MG_1999.CMI
       quit

After the dataset is downloaded, we need to make sure that it is accessible from our McIDAS session. We do this by making sure the filename is not more than 8 characters long and by adding a REDIRECTion to the file(type), *.CMI, which is now located in the ~/workdata/EWBMS directory. In mcidas:

Next, we make sure that we entered the REDIRECT command correctly: If McIDAS can find the file using DMAP, then we should see the following print:

PERM      SIZE LAST CHANGED FILENAME    DIRECTORY                                                          
---- --------- ------------ ----------- ---------                                                          
-rw-    576000 Jan 20 13:00 MG_1999.CMI /home/mcidas/workdata/EWBMS
Next, you must decide the name of the AREA file that you want to write the data from EWBMS into. McIDAS uses AREA file numbers in the range from 9000 to 9999 for TOPOGRAPHY datasets. The Learning Guide data set uses AREA file numbers in the 8000s. So let's create the output data file in the file number range 4000s for this exercise, starting with AREA4000. But first, we have to make sure that it does not already exist: If it does, delete it before proceeding: Next, you must decide whether to publish this dataset using the routine DSSERVE. This will open up public access to the dataset for remote clients (McIDAS, McIDAS-Lite, IDL/ENVI, Matlab, VisAS, IDV, etc.) provided they support the ADDE protocol, possibly through an optional add-on. Detailed information on ADDE may be found in Abstract Data Distribution Environment - ADDE.

Here we choose a name used by ADDE to reference a dataset; the dataset name consists of a group name and a descriptor name separated by a slash ( / ); for example, MODIS/MOD02.

This entry assigns the dataset name EWBMS/CMI to areas 4000 through 4999 on the local server (1000 positions). Before adding AREA files to this new ADDE data group, you must decide where to store them. By default, McIDAS stores AREAs to the /home/mcidas/data directory, unless we specify a REDIRECTion to a different location for just the files in the range from 4000 to 4999: Note you can specify any directory here. A unique directory for every AREA file number range would, however, easen your file management and backup procedures. But for this exercise a common location like the one use in our previous REDIRECT instance (/home/mcidas/workdata) suffices.

Next, we will convert the CMI product using the routine MAKEAREA to a McIDAS compatible dataset of type IMAGE. As you can deduct from the syntax of MAKAREA, we need to know more about the formating of our CMI binary data object first. In the table below we list some relevant metadata supplied to us by EARS:

Term Value
Data type Binary data (16 bit or 2-byte)
Stratification levels 255
Image width 800 pixels (columns)
Image height 360 lines (rows)
Spatial Resolution 0.05-by-0.05 degree latitude / longitude
Upper left latitude 54.44 (decimal degrees)
Upper left longitude -82.92 (decimal degrees)

Next, we will create a navigation entry for our AREA file using the routine MAKNAV. As you can see, the metadata also has the information necessary to add navigation to the image, so let's geolocate it with MAKNAV:

This entry generates a geographic (i.e., rectilinear, lat/lon) projection codicil entry for local area 4000. After successful creation, you can use McIDAS display, list, and analysis tools to manipulate your data. First, let's check to see that we have defined a dataset that includes the file AREA4000: As per our previous steps, the dataset of interest for this exercise is EWBMS. Let's take a closer look:

        Dataset Names of Type: IMAGE in Group: EWBMS

Name         NumPos   Content
------------ ------   --------------------------------------
CMI           1000    Climatic Moisture Index (105E)

DSINFO -- done
The file we created, AREA4000, will occupy the 1st position in the EWBMS/CMI dataset. We can, therefore, refer to it directly as EWBMS/CMI.1. We use this identifier in a number of commands to list out information from/about the data.

Let's plot the data:

This plotted the CMI for 1999 over continental Asia overlayed by a MAP (vector file) with the alias name "SAT". Note that the image was plotted in dimeless units, as CMI is a relative parameter. We also observe that the image was correctly navigated from the brightness values of the waterbodies which nicely coincide with our vector overlay.

Now let's plot some image characteristics. Use command IMGLIST with FORM=ALL to list the image directory. An example is shown below.:

       Image file directory listing for:EWBMS/CMI                                           
 Pos Satellite/         Date       Time      Center      Res (km)   Image_Size       
     sensor                                 Lat  Lon    Lat   Lon                    
 --- -------------  ------------  --------  ---- ----  ----- ----- ------------      
  1                  6 FEB 08037  20:36:06    45 -103                                
   Band: 1    No Information Available                  5.56  3.90   360 x  800      
     proj:    0 created: 2008037 203606  memo: MG_1999.CMI                          
     type:VISR     cal type:BRIT                                                     
     offsets:  data=    3328 navigation=  256 calibration= 2816 auxiliary=    0      
     doc length:   0   cal length:   0   lev length:   0 PREFIX=   0                 
     valcod:          0 zcor:  0 avg-smp: N                                          
     lcor:    1  ecor:     1  bytes per pixel: 1  ss: 99                             
     Resolution Factors (base=1):   Line=    5.0   Element=    5.0                   
IMGLIST: Description for SS number 99 was not found in the file SATANNOT             
IMGLIST: done                                                                                         
Notice that from the above listing we can see that only the file name (memo: MG_1999.CMI) is retained in the image characteristics when we produced a McIDAS-compatible data file (MAKAREA). By default, McIDAS has no descriptive information, i.e. type of band (Band: 1 No Information Available), available to describe our image data properly, so we have to supply this information using the routine IMGCHA. The Unidata documentation shows which values can be changed with the IMGCHA command. This reveals that the sensor source (SS) number is one of them, which is a good thing because the default SS number assigned to our McIDAS-compatible data is non-existing (Description for SS number 99 was not found in the file SATANNOT). Before we start changing our image metadata, let's print the contents of the SATANNOT file to beter understand it's architecture. In the linux terminal:
DERIVED DATA                  0                                                 
TEST IMAGE                    1                                                 
GRAPHICS                      2                                                 
--More--
FY-1B              X         95                                                 
FY-1C              X         96                                                 
FY-1D              X         97                                                 
TERRA-L1B          X         101                                                
--More--
As you can see from the above listing, there is no SS number "99", but there is one for derived data with SS number "0". Apparently AREA files can have a variety of contents, either observed or derived data aka products. As we are dealing with a product here, let's look more closely at how McIDAS handles metadata for products.

The exact contents of an AREA file is described by the satellite annotation information aka metadata. This consists of two components:

The last provides information on the bands of satellites. So let's examine this file in more detail. In the linux terminal:
Sat 0
Cal EAS
BRes 1.0 1.0
 1 DESC='Data Points Per Pixel'
 2 DESC='11.17 um Average' WL=11.17 um WN=895.26 cm-1
 3 DESC='11.17 um Stand. Dev.' WL=11.17 um WN=895.26 cm-1
--More--
32 DESC='3.94 um Stand. Dev.' WL=3.94 um WN=2538.07 cm-1
EndSat
As you can see from the above, for each SS there is a opening-tag Sat [SS] and a closing-tag EndSat. In-between these parental tags you see a listing of band descriptions consisting of a band sequence number BAND followed by a string field DESC which we may use to our descretion to describe the variable and it's units. Now that we better understand the formatting of the SATBAND file, let's edit it to reflect our EWBMS product data. In the linux terminal: and add the following lines:
Cal VISR
BRes 1.0 1.0
33 DESC='National-level CMI'
in between:
32 DESC='3.94 um Stand. Dev.' WL=3.94 um WN=2538.07 cm-1
EndSat
so that this results:
Sat 0
Cal EAS
BRes 1.0 1.0
 1 DESC='Data Points Per Pixel'
 2 DESC='11.17 um Average' WL=11.17 um WN=895.26 cm-1
 3 DESC='11.17 um Stand. Dev.' WL=11.17 um WN=895.26 cm-1
--More--
32 DESC='3.94 um Stand. Dev.' WL=3.94 um WN=2538.07 cm-1
Cal VISR
BRes 1.0 1.0
33 DESC='National-level CMI'
EndSat
Notice we specified the calibration type (CType) as VISR. Now close your text-editor gedit and restart your McIDAS-session so the modified SATBAND file is re-read from disk. Next, we have to tell McIDAS to use this new information using the IMGCHA command. In McIDAS: Notice we changed: Remember, because a workstation can store and display data, it can act as a server and a client. As the last two fields are used to specify the Observation (or Production) Date/Time of our image, remote clients (McIDAS, McIDAS-Lite, IDL/ENVI, Matlab, VisAS, IDV, etc.) accessing this dataset can also read-out and make use of this metadata in order to represent it the way it was intended.

To exercise your growing skills with McIDAS, try repeating the same procedure of other image (product) data in combination with the IDV as a remote client to see the changes you make to your McIDAS ADDE server in real-time.

 


Previous: Importing Foreign POINT Datasets Into McIDAS Next: McIDAS-X Useful Information Table of contents Index Glossary Images Frames ITC-NGIC McIDAS-X Training Workshop > Applying Lessons Learned > Importing Foreign Datasets Into McIDAS