³ò »\ # INPUTS: # If a remote server is specified an attempt at downloading by (s)ftp is made: # # remote server # directory on remote site with L1A files # remote user # password # OPTIONAL INPUT PARAMETERS: # -source=source default: /mnt/cdrom # source directory to look for *.buf or *.buf.gz files. # For ftp this is also the directory where downloaded # *.buf.gz are put (write access to the source directory # is needed) # If L1A files are retrieved from a local L1A mirror # (-mirror is set) then this is the directory where the # L1A files taken from the mirror are copied to. # -destination=destination # destination directory for CCD frames: # default: $SMEIDB # # -gunzip look for gzipped *.buf.gz files and unzip them, # instead of looking for *.buf files # -maxftp=maxftp maximum number of l1a*.buf.gz files downloaded by ftp, # or copied from local L1A mirror (if -mirror is set) # # -nounpack only download and unzip L1A files. Don't unpack them. # The names of successfully downloaded and unzipped files are # WILL BE ADDED to the list of locally processed L1A files. # -detach only used for ftp downloads. # If set then an L1A file is unpacked immediately after # download is complete, in parallel with the download of the # next L1A file. If not set, all L1A files are downloaded # first, and are then unpacked in a single operation. # # -l1a=l1a name of specific L1A file to be processed # -all_l1a by default only L1A files are processed that are not yet # listed in file 'file_list' (under the assumption that # these have already been dealt with). Set this keyword to # unconditionally process L1A files. # -file_list=file_list # fully-qualified name of file with names of already processed # L1A files. The file must exist already, so if necessary # create an empty one before running this script. # -nowrite suppress writing of CCD frames (useful for testing only) # -overwrite force ovewriting of existing CCD frames. # By default frames are not written to drive if a file for the # frame already exists. This is currently the primary control # for picking up either the first or second telemetry dump # (setting -overwrite speeds up processing by a factor 2). # -nodups if set then an attempt is made to remove duplicate frames # before starting to write individual frames. # -zipdir zip CCD frames when leaving subdirectory # -remove delete *.buf files after processing # -move=move move *.buf files to directory 'move' after processing # -noclones skip clone frames (see href=smei_buf_prep=) # -unblock skips test for running instances of l1a.py in ftp mode. # -nic write .nic files instead of .fts files (OBSOLETE) # -kblob buffer size for reading remote L1A files for sftp. # default is 64 kB. If set to zero the file is downloaded # with a single read (this is very slow for some reason). # -sftp download using sftp instead of ftp. # -mirror looks for L1A files in locally maintained mirror of # remote L1A files (instead of downloading by ftp or sftp) # OUTPUTS: # Updated data in # Exit code is number of L1A files processed # CALLS: # l1a_near, l1a_ftp, l1a_idl, l1a_redo, l1a_gunzip # SEE ALSO: # smei_buf # PROCEDURE: # > Only one instance of the script is allowed to run. # > The list of L1A files found locally or (when using ftp) on the remote site # are compared against the list of l1a file names in file 'file_list' # Only files not on this local list will be downloaded. Override this # behaviour using keyword all_l1a. # MODIFICATION HISTORY: # MAR-2003, Paul Hick (UCSD/CASS) # APR-2003, Paul Hick (UCSD/CASS) # Added -overwrite, -nodups, -zipdir, -remove keywords. # OCT-2003, Paul Hick (UCSD/CASS) # Added -alive keyword. # FEB-2004, Paul Hick (UCSD/CASS) # Added -nounpack keyword. # Added -noclones keyword to l1a_idl # Added -move keyword # MAR-2004, Paul Hick (UCSD/CASS) # Added keyword 'file_list' instead of hardcoding file name $HOME/l1a_files.txt. # FEB-2005, Paul Hick (UCSD/CASS) # Removed -sparseftp option (never used it). # Added -sftp switch to use sftp instead of plain ftp (requires # packages paramiko and pycrypto). # Removed all references to the module string. # SEP-2005, Paul Hick (UCSD/CASS) # After installing Paramiko 1.4 on 64-bit AMD (running Fedora Core) # download would fail for every other file (first downloaded OK, # second failed, third OK, etc.). Modified code to open a new # ssh session for each download. Also introduced kblob keyword. # DEC-2005, Paul Hick (UCSD/CASS) # Added keyword -mirror to be able to process L1A files from a # locally-maintained mirror of the L1A data base. # FEB-2008, Paul Hick (UCSD/CASS) # Added check for existence of env var $L1A. # Added check to prevent two copies of l1a.py to run at the # same time. # DEC-2009, Paul Hick (UCSD/CASS; pphick@ucsd.edu) # Exit code now is number of L1A files processed (was always zero) #- iÿÿÿÿNtTUB(thide_envtis_theretstarttrun_cmdtargstcount_instancescCs[d}tiitƒo%ttdƒiƒ}|iƒ}ng}t|ƒdjo|dt|ƒGHnêxh|D]`}tii|ƒ\}}|djo |}ntii|ƒ\}}|i |i ƒƒquW|i ƒg}xB|D]:}|i |ƒdjo|i |ƒqð|d|GHqðW|}ttdƒi di|ƒdƒ|S( sC #+ # NAME: # l1a_near # PURPOSE: # Function to update / retrieve list of already processed L1A files # CALLING SEQUENCE: # near_list = l1a_near( buf_files ) # INPUTS: # buf_files list of file names to be added to the list of already # processed L1A files stored in file 'file_list' # (this may be an empty list) # OUTPUTS: # near_list list of processed L1A files currently # PROCEDURE: # The file names (name and type only) of all processed files are stored # in the ascii file 'file_list # MODIFICATION HISTORY: # APR-2003, Paul Hick (UCSD/CASS; pphick@ucsd.edu) #- s l1a_near, tris%d local L1A filess.gzs duplicate tws (tostpathtexistst file_listtopentreadtsplittlentsplitexttappendtlowertsorttcounttwritetjoin(t buf_filestsayt near_listtbuf_filettmp_filetdropt new_files((s./l1a.pytl1a_near~s. #c Csd}g}xÏ|D]Ç}|o3tii|ƒ\}}|djo |}qkntii|ƒ\}}|iƒ}|i|ƒ} |o"| djo|i|ƒqÚq| djo||dGHq|i|ƒqW|o|dt|ƒGHn|dt|ƒGH|S(s¯ #+ # NAME: # l1a_redo # PURPOSE: # Check list of available files against list of files to be reprocessed. # CALLING SEQUENCE: # list = l1a_redo( byftp, buf_files, redo_list, negate ) # INPUTS: # byftp if set, then buf_files is a list of *.buf.gz files (obtained from ftp nlist cmd) # if not set, then buf_files is a list of *.buf files available locally. # # buf_files # byftp set: list of *.buf.gz files (file names, plus extension only) # byftp not set: list of *.buf files (fully-qualified file names) # # redo_list # list of *.buf files (only file name, plus extension) # # negate negate set: return subset of buf_files NOT present in redo_list # negate NOT set: return subset of buf_files also present in redo_list # OUTPUTS: # list list of files will be a subset of buf_files. # PROCEDURE: # Compares entries in buf_files against entries in redo_list. # MODIFICATION HISTORY: # APR-2003, Paul Hick (UCSD/CASS; pphick@ucsd.edu) #- s l1a_redo, s.gzis skippeds%d remote L1A files to processs%d local L1A files to process(R R RRRRRR( tbyftpRt redo_listtnegateRt redo_filesRRRR((s./l1a.pytl1a_redoÂs(   c Cs/d}g}x|D]}tii|ƒ\}}|dj}|otii|ƒ\}}n|djoºt|ƒdjp|i|ƒdjotii||ƒ}|t|ƒGH|o:titiddd|ƒ} tii|ƒ\}}ntii |ƒo|i |ƒq#q'qqW|S(sÔ #+ # NAME: # l1a_gunzip # PURPOSE: # Unzip list of *.buf.gz files (usually files downloaded from Hanscom) # CALLING SEQUENCE: # buf_files = l1a_gunzip(gz_list, ldir, redo_list) # INPUTS: # gz_list list of *.buf.gz files (only file name, plus extension) # ldir directory where *.buf.gz files are located # redo_list only entries in gz_list also on redo_list are processed # (if redo_list is empty all files in gz_list are processed) # OUTPUTS: # buf_files list of successfully unzipped *.buf files # (fully-qualified file names, located in ldir) # PROCEDURE: # Loops over all files in far_list, spawning gunzip for each file individually. # MODIFICATION HISTORY: # APR-2003, Paul Hick (UCSD/CASS; pphick@ucsd.edu) #- s l1a_gunzip, s.gzs.bufitgunzips-f( R R RRRRRtspawnlptP_WAITtisfileR( tgz_listtldirR!RRtgz_fileRtextR%tsts((s./l1a.pyt l1a_gunzip s" # )cCsòtiitiddƒ}tii|ƒ}|o¹t|dƒiƒ}|idƒ}t|dƒd}dt ƒ}t i dƒ}t|d ƒi d |d d |d dƒti |ddd|ƒt|d ƒi |dƒn|S(sÒ #+ # NAME: # l1a_summary # PURPOSE: # CALLING SEQUENCE: # status = l1a_summary( wait ) # INPUTS: # OUTPUTS: # (none) # PROCEDURE: # MODIFICATION HISTORY: # JUL-2003, Paul Hick (UCSD/CASS; pphick@ucsd.edu) #- tHOMEsl1a_doy_summary.txtRs iis%(doy)ds.proRssmei_frm_summary, timeset(doy=s),hr=[0,24]), dest=getenv("TUB"), /silent stmp = do_file(/delete, "s ", /silent) sexit tidls-quiet(R R RtenvironR R RRtinttvarsttempfiletmktempRR&(twaitt doy_summarytstatustdoytmain_pro((s./l1a.pyt l1a_summary9s cCs^ti|ƒd}ti|ƒd}||jo d}n||jo d}nd}|S(Niiiiÿÿÿÿ(R tstat(tf1tf2tp1tp2R-((s./l1a.pyt l1a_mtimehs    cCsd}td|ƒ}td|ƒo d}ntd|ƒ}td|ƒ}td|ƒ}td|ƒ} td |ƒ} td |ƒ} td tiƒ} td tiƒ} td tiƒ}|djo*tii|ƒpdG|GHd}qöntd|ƒ}|djo d}ntidƒ}tidƒ}t |dƒi d|dd|dd|ddƒt |dƒ}|i |dƒ|i |dƒ|i ddg|dƒ|i dddg|dƒ|i d d!g|dƒ|djo|i ddg|dƒn|i |dƒ|i ddg| dƒ|i d"d#g| dƒ|i d$d%g| dƒ|i d&d'g| dƒ| o|i t ƒn |i ƒ|i di |ƒdƒ|iƒti|d(d(d)|ƒd*S(+s. #+ # NAME: # l1a_idl # PURPOSE: # Unpack *.buf files writing separate files for each frame # CALLING SEQUENCE: # l1a_idl( sys_argv, wait, buf_files, ddir) # INPUTS: # buf_files list of *.buf files or *.buf.gz (fully-qualified names) # Must have at least one element. # ddir destination directory # OUTPUTS: # (none) # PROCEDURE: # > Creates main program for starting up the IDL procedure smei_buf # > Writes ascii file with input information for smei_buf, # > Then spawns the main IDL program # > Both the IDL main program and the ascii file will be deleted by smei_buf # MODIFICATION HISTORY: # APR-2003, Paul Hick (UCSD/CASS) # FEB-2004, Paul Hick (UCSD/CASS) # Added -noclones keyword # MAR-2005, Paul Hick (UCSD/CASS; pphick@ucsd.edu) # Added -nic keyword. By default Fits files are written to disk. #- s l1a_idl, s-nodupss-nowriteis -overwrites-zipdirs-removes -nocloness-silents-mtimes-nics-statuss-move=tsnon-existent directory:s-email=tno_emails.pros.txtRs smei_buf, "s" stmp = do_file(/delete, "s ", /silent) sexit s tkeeptremoves skip existingsoverwrite existings do not writetrawtgziptloudtsilents.ftss.nictblindR8R0s-quietN(RtsystargvRR R tisdirR4R5R RRRARtcloseR&(tsys_argvR6RtddirRtnodupst overwritetzipdirREtnoclonesRItmtimetnicR8tmovetotemailR:ttmp_listtiu((s./l1a.pytl1a_idlvsZ     #   c Cs~tidƒ}t|dƒ}|id|dƒx#|D]}|id|dƒq:W|idƒ|iƒtd|d|d ƒ}ti|ƒ|idƒ}g}x˜|D]}|i d ƒd jot|id ƒ}|i ƒ|d d d!d|d dd!d|d d d!|d dd!|d <|i |ƒq´q´W|i ƒg}x|D]}|i |d ƒq_W|S(Ns.ftpRscd s smodtime squit scat s | ftp itL1A_is ii t/iiii( R4R5R RRNRR RERtfindtreverseRR( tsitetfoldertfar_listt ftp_scriptRZtxtlsttgmtty((s./l1a.pytl1a_ftp_modtimeís0     J c-Css d}tdtiƒ}|o6td|ƒ}|djo d}qUt|ƒ}nt|ƒ} g} tiiti ddƒ} d} tii | ƒowt | dƒi ƒ} | i d ƒ} xP| D]D}|i ƒ}|i| d ƒd jo|d } |d }qÇqÇWn| djo<t| ƒd jo|dGH| Sn| d } | d}n|d| d| d d| dGH|pHttii | d ƒd d| d d ƒdjotiƒqÐn|oœddk}ddk}tiiti dddƒ}t |dƒ}xŠ|D]]}|i dƒ}t|ƒd joq$n|\}}}|i| d ƒd joPq$q$W|d| d GH|iƒtiƒ|iƒ|idti|ƒƒ}|i| d ƒ}|id|d| d|ƒ|iƒ}|id| ddƒ|iƒi ƒ}|iƒ|iƒ|i d ƒ}|d t|ƒd !}nNdd kl}|| d | |ƒ}|i | dƒ|i!ƒ}|iƒg}xÄ|D]¼}tii"|ƒ\} }!|!d!jo&tii"| ƒ\} }!|!d!}!n|!d"jo$t|ƒd#jo|i#|ƒqÍ|!d$jo$t|ƒd%jo|i#|ƒqÍd&|GHqÍW|}t|ƒd jot$d ||d ƒ}n|p%t%gƒ}t$d ||d ƒ}nt|ƒd jo |d'GHnetd(|ƒ}"td)|ƒ}#|#djo d*}#nt&|#ƒ}#|i'ƒt|ƒ|#jo|d |#!}ntd+|ƒ}$|$ot(| d | d|ƒ}n|p+|| d | |ƒ}|i | dƒnxc|D][}|iƒ}%tii||i)ƒƒ}&y|oÏ|i| d ƒ}|id|d| d|ƒ|i*ƒ}|i tii| d|ƒddƒ}|i+ƒi,}'||d,t-|ƒd-|'GH|d joF|i ƒ}(|iƒ|iƒt|(ƒ})t |&d.ƒi.|(ƒn”|d/}*d })t |&d.ƒ}+|i |*ƒ}(xDt|(ƒd jo0|+i.|(ƒ|)t|(ƒ})|i |*ƒ}(q3W|+iƒ|iƒ|iƒ|)|'jo=|iƒ|%}%|)d0})d1|)d0|%d2|)|%f},q&d3|},n:||d,t-|ƒGH|i/d4|t |&d.ƒi.ƒ},Wnf|o|iƒnd5|GHti0|&ƒt%|&gƒ}d6d7|&d d8d9t1d d:GHqçX||,GH|o|,id;ƒd jp"| ow|,d<jp |,d=jo]|"o/|&g}&t%|&ƒ}t2|ti3|&|ƒqB tii |&ƒd }&| i#|&ƒqçti0|&ƒqçW|p|iƒn|"o |d>GHn| S(?sÚ #+ # NAME: # l1a_ftp # PURPOSE: # Download .buf.gz files # CALLING SEQUENCE: # buf_files = l1a_ftp( sys_argv, ldir, redo_list, all_l1a, ddir, unblock ) # INPUTS: # sys_argv command line list sys.argv # contains the info to log into the ftp site # ldir local directory where to put the downloaded files # redo_list list of l1a files to process # Only used if len(redo_list) != 0 # all_l1a if set, process all *.buf files # if not set, only *.buf files not processed locally yet will be # downloaded. # OUTPUTS: # buf_files list *.buf.gz to be processed further # if detach is set then the return value buf_files is an empty list. # CALLS: # l1a_near, l1a_redo, l1a_idl # PROCEDURE: # > The presence of ftp-specific keywords -detach, -sparseft and -maxftp is tested # for inside this routine. # MODIFICATION HISTORY: # APR-2003, Paul Hick (UCSD/CASS) # FEB-2004, Paul Hick (UCSD/CASS; pphick@ucsd.edu) # Local L1A file names are now in lower case (l1a* instead of L1A*) #- s l1a_ftp, s-sftps-kblob=RBi@R/s.netrcRs iiiis5specify remote info (site, dir [, user and password])is remote site: s<*****>@t:it iÿÿÿÿNs.ssht known_hostss no key for tdatathostkeytusernametpasswordscd s ; ls *.buf(tFTPs.gzs.buf.gzi!s.bufisnot downloading s$no remote L1A files left to downloads-detachs-maxftp=t5s-mtimes -> s (%d bytes)twbig@s,sftp complete %.2f MB, %.2f hours, %.2f kB/sg ¬@s$local and remote size do not match, sRETR s exception on s s.=================== WARNING ================= s0Download interrupted. To force another download sremove entry from s0============================================= s sftp completes226 Transfer complete.s226 File send OK.sdetached unpack finished(4RRKRLRR2RR R RR1R R RRRRRtexittparamikottimeR^RNtRSAKeytbase64t decodestringt Transporttconnectt open_sessiont exec_commandtmakefiletftplibRptcwdtnlstRRR$RtevalRRhRtopen_sftp_clientR<tst_sizeRRt retrbinaryRER R[tP_NOWAIT(-ROR*R!tall_l1aRPtunblockRtuse_sftptkblobRLRtnetrcRntmachinestmtpartRoRtRutkeyfileRZtlinetkeylistthoststkeytypetkeytftptsftpRbRpRtfar_gziptbufR,tdetachtmaxftpRUtttt near_gziptsz_farRltsz_neartblobtjuR-((s./l1a.pytl1a_ftps:"        %6                     (             &        c CsAd}t|ƒ}g}ti|dƒ}g}xÄ|D]¼} tii| ƒ\} } | djo&tii| ƒ\} } | d} n| djo$t| ƒdjo|i| ƒq8| djo$t| ƒdjo|i| ƒq8d| GHq8W|}t|ƒd jotd ||d ƒ}n|p%tgƒ}td ||d ƒ}nt|ƒd jo |d GHnÈt d |ƒ} | d jo d} nt | ƒ} |i ƒt|ƒ| jo|d | !}nxd|D]\} | i ƒ} t dtii|d| ƒdtii|| ƒd ƒ}|i| ƒqÝW|S(Ns l1a_mirror, is.gzs.buf.gzi!s.bufisnot processing iisno L1A files left to processs-maxftp=RBRqscp -vp Rj(RR tlistdirR RRRR$RRRRRRR(ROR*R!R†RRLRRbRR–R—R,R™R›ttmp((s./l1a.pyt l1a_mirrorZsH          =t__main__iis, tL1Asenv var L1A not defineds-alivetSMEIDB1t2003_033tSMEIDB2t2003_180s"SMEI drives available; all is wells)SMEI drives not found; cold boot requireds-email=RBs#echo "Cold reboot required" | mail s -s"SMEI drives on tHOSTNAMEs apparently off-line"s -file_list=s0no list of already-processed l1a files availablesfile does not exist:s-source=s /mnt/cdroms-mirrors-unblocksl1a.py -source=tL1A_GETsl1a.py s -destination=tSMEIDBsSMEIDB?snon-existent directory, s-l1a=t,s-gunzips-all_l1as -nounpacks.bufsno *.buf files in s-move=snon-existent directory:s========== ALL DONE ==========(;t__doc__R RKR4RwR1ttempdirttinyRRRRRRRR$R.R;RAR[RhR R£t__name__R RRLtnameRthas_keyRsRMRRXtcmdR8R R R*tbymirrortFalseR R‡RRPR!R%R†tnounpackRR¡RRRRRRWtrenameR'(((s./l1a.pysxsÌ0. D G 0 /  w + ÿC A   ))       !!   )(   7