(from "JSOC tape archiving" mail 03/14/2011 16:08)


Draft plan, written in psuedo code:

while ('keep_running')                 // keep_running is some external flag, e.g. empty file existence.
    foreach tapegroup
        lookup date of most recent archive for this tapegroup
        lookup MaxWaitArchive for this tapegroup
        lookup MinWriteSize for this tapegroup
        lookup supergroup for this tapegroup
        Build list of ALL archive-pending for this tapegroup: sunum, totalsize.
        If (totalsize >= MinWriteSize ||           // MinWriteSize is e.g. 400GB
            totalsize >= FullTapeSize ||           // FullTapeSize =~ 800GB*0.95 ?
            time since archive > MaxWaitArchive)   // MaxWaitArchive is e.g. 7 days
        then
            WriteList(list, supergroup *writtenlist)
            if (successful)
                update SUMS main table with info in writtenlist
            else
                complain to log, do not mark SUs as archived.
        endif
    endfor
    sleep for some hours.
endwhile

function WriteList(list, supergroup, *writtenlist)
   Get current tapeID for this supergroup, if none, assign one.
   Request a drive from pool, lock.
   Sort SUs by sunum
   Tar in min 500MB (or 1GB?)
   *writtenlist = list of sunum, filenum.
   continue in list until tape is "full" or list is consumed.
   unlock drive assignment.
end


For large quickly accumulating series, set MinWriteSize == FullTapeSize
for tlm, set MaxWaitArchive == 1 day

for small slowly growing series set MinWriteSize ~= 200GB and MaxWaitArchive = 7d

tune for each tapegroup with experience.