/home/jim/STAGING/src/pipe/doc/lev1.8.txt


			Lev1.8 DISCUSSION
			-----------------

1.) Production will not routinely archive lev1.8 datasets.
Therefore all the lev1.8 will be made as temporary ds with
a=t1 to allow them to survive through at least 1 midnight.
They are made as temporary ds so that the /PDS storage is 
conveniently used and automatically cleaned up. Otherwise 
some outside storage would have to be managed as an additional
piece of administration. They are NOT made as temporary ds so 
that they can be later queried.
In general, you would not expect to find any map file with a 
d=1 (query from DB) for any input of a lev1.8 ds. If the lev1.8
is needed as input, then it is made in the same map file.

2.) A lev1.8 ds will normally be created on demand. For example,
when a user wants lev1.8 for export they would select the ds from
the current lev1.5 menu and then select a lev1.8 button at the 
end. (Of course, the front end could just display pages w/ds marked 
1.8 to make it easier on the user.) When the back end scripts get the
export request they must treat lev1.8 differently and not just
query the DB to locate the ds, but make map files and run pe to
create the requested ds which then can be copied to the user's target.

3.) One of the requirements is that given any lev1.8 ds, it can 
be reproduced with the exact same input (even the executable?).
The lev1.8 module (call it exe18 for now) will take an ARG_DATA_IN "in"
and a number of ARG_DATA_IN "table" arguments. Whenever a new table is
available it will be ingested into the DB as a permanent ds. The default
table used for exe18 will therefore be the latest one, but any former
table can be used by knowing its level number. The exe18 must save the
details of its input in keywords in its fits files. This is all normally
handled by the ds.rdb file made with a ds, but the lev1.8 is not archived
and the connection with the ds.rdb can get lost.

4.) Here is a typical pe map file:

!setenv mdi wd:{dbase}/{level}/{series}.{#%05d#series},bn:{series}.{#%05d#series}
DSDSOUT=100
p=exe18 d=1 a=t1 \
in=prog:mdi,level:lev1.5,series:fd_V_01h[60000] \
out=prog:mdi,level:lev1.8,series:fd_V_01h[60000] \
tableA=prog:mdi,level:lev1.8,series:tableA \
tableB=prog:mdi,level:lev1.8,series:tableB

Where an output file looks like:
/PDS15/D3479680/lev1.8/fd_V_01h.60000/fd_V_01h.60000.0001.fits

The exe18 module must save the particulars of its run as keywords in its 
fits files, in order that the output can be reproduced.
Among the keyword exe18 receives that it must save are:

in_0_data:          KEYTYP_STRING   prog:mdi,level:lev1.5,series:fd_V_01h[60000]
in_0_level_sn:      KEYTYP_INT      6
tableA_0_data:      KEYTYP_STRING   prog:mdi,level:lev1.8,series:tableA
tableA_0_level_sn:  KEYTYP_INT      3
tableB_0_data:      KEYTYP_STRING   prog:mdi,level:lev1.8,series:tableB
tableB_0_level_sn:  KEYTYP_INT      2

Please note that the "out" level number will not have any meaning,
as it can go up and down as these are temporary ds.

5.) Concerning reproduction of the executable. If we want this, it is a
new beast. We have always used the latest executable produced by the
last CM build. We have always had the capability of rebuilding any old
executable given the CM build number (e.g. V4R5B0) but have not used it
and in any event, two build versions of the same executable don't 
coexist. We could modify the CM build scripts to rename exe18 when it
is made to reflect the current build number, e.g.

exe18.V4R5B0
exe18 -> exe18.V4R5B0

All the build versions would remain availabe in the $STAGING/bin dirs.
If pe is also modified to pass a new keyword to a module then exe18
could save this also as a fits keyword and the same executable could
be reused to regenerate output:

cm_version:    KEYTYP_STRING   V4R5B0