01/31/08 Summary - SUMS garbage collection - DRMS garbage collection - Remote SUMS: an alternative implementation - ^C induced segfault in direct connect module SUMS garbage collection ----------------------- These garbage SUs are generated by delete_series To cleanup, SUMS needs to mark them in sum_partn_alloc table. The main problem to update sum_partn_alloc table from DRMS is that of permission. Currently sum_partn_alloc is owned by jim and write accessible only by jim and production. My first try was to create a public writable table to store sunum information to be 'deleted' by SUMS. E.g., table sum_del will have the following columns: sunum, seriesname(fully qualified), timestamp where 'timestamp' denotes the date/time that delete_series runs Since this table is writable by everybody, the main issue is malicious or accidental entries. To prevent such entries, when it's time for SUMS to cleanup, it needs to make sure such seriesname does not exists in drms_series() that's created before the given timestamp. The following query must return no rows for SUMS to be certain. select * from drms_series() where seriesname = 'seriesname' and created < 'timestamp'; Unfortunately, this solution still can't completely prevent malicious entries in sum_del since the timestamp can be fabricated as well. The first try failed. My second try was to call SUMS synchronously to mark the dead SU's within delete_series. The order of "drop table" statement and SUMS call matters. Here is why. Without a call to SUMS, "drop table" will hang until all modules that use seriesname exit. With a call to SUMS before "drop table", because SUMS access is not protected by transactions, marking a SU in sum_partn_alloc table may result in the SU being scrubbed while other modules are still in the middle of using it (race condition). Must call SUMS after "drop table", which will block until other modules using that series exit. Need to copy the sunums into a temp table and commit it so that SUMS can see it. then drop table... Implemented in drms_series.c and drms_server.c pending SUMS new function. command code DROPSERIES now has additional SUMS calls. Used #ifndef DRMS_CLIENT in drms_series.c moved it out of COMMON_OBJ_$(d) in client/Rules.mk. ---------------------------------------------------------------------- DRMS garbage collection ----------------------- Add one step to cleanup temporary storage (SU: no archive, retention time out) given sunum, online_loc open Records.txt, read seriesname, recnum (n) permission problem came up again. this row/information has to be deleted later when user decides to remove a record whose temporary SU has expired. Let create_series also create a table called seriesname_nosu that has the same owner as seriesname, but also appendable by SUMS (user production) this table contains the following columns: recnum sunum timestamp (The alternative of writing all into one table makes it hard to define delete permission.) ---------------------------------------------------------------------- Remote SUMS ----------- Discussion with Jim. Alternative to sunum translation: remote SUMS ingest copy of data segment using its original sunum. Requires a new SUMS call SUM_alloc2(sunum): allocate SU for a given sunum. impact: - must check to make sure sunum is outside local range to ensure uniqueness. - perhaps affect sum_main table performance a little, since sunum (ds_index) is no longer sorted (monotonically increasing). Plan standalone ingest program for remote SUMS. no DRMS API required input: sunum and dir Remote_SUMS_ingest { SUM_open() /* Important: check site-id bits to make sure it's not local (this should be absorbed into SUM_alloc2() function call)*/ SUM_alloc2(sunum) returns wd (and adds sunum into local SUMS table) copy contents into wd SUM_put() SUM_close() } It looks like it's better to keep the SUMS site id in SUMS, since DRMS does not need it at all (no sunum alias translation business any more). ---------------------------------------------------------------------- Segfault, Segfault, Segfault ---------------------------- Jim reported some ^C induced segfaults in a direct connect module. I was able to produce many segfaults using ingest_lev0 and ^C. In a direct connect module, there are three threads: - the main/server thread: it runs DoIt as well as connecting to the database. - the signal thread: it decides when/how to quit - the SUMS thread if module access SUs Signals are blocked out for all threads except for the signal thread. ^C puts the signal thread to execute drms_server_abort() which does two things (among other): A1. close db connection A2. call drms_free_env() For A1, I made sure the db abort_now flag is only checked after db handle is locked. otherwise such checking has no use, i.e., anything can happen in between. For A2, depending on the timing between the execution of the signal and main thread, We may see a variety of errors: E1. SQL statement failure. They are caused by the close DB connection E2. Segment fault due to freed pointers. Segfault any access to data structure that drms_free_env() has cleaned up, e.g., record cache and numerous others. E.g., records are freed by drms_free_env(). Recall by default, all records are in the record cache, which is freed by drms_free_env) For E2, I put in a flag for drms_server_abort() that conditionally calls drms_server_abort(). A third category of error I saw was a result of E1. E.g., depending on a timing between the signal thread and the main thread, create_records() may fail in the middle, even before a record enters the record cache. If a record is not in the record cache, drms_free_record() segfaults. I fixed this particular problem. Overall, this group of problems basically exercise some of the early exit routes our code. Keep sending me these segfault cases. Or better yet, show me how to reproduce the problem.