Accessing RHESSI Data
Overview of RHESSI Data Accessing RHESSI Data in the RHESSI Software Flare Catalog Copying Files to Your Computer Observing Summary Data Mounting or Sharing Local Archive Level-0 Telemetry Data Setting Environment Variables And More... Using Simulated Data Data Directory Structure
Overview of RHESSI Data
There are two primary archive locations for the RHESSI data: GSFC (Maryland) and FHNW (Switzerland). Users can access these locations through a browser or through anonymous ftp (GSFC only). In addition, a few of the smaller data files are distributed with the SSW software; every time you update SSW, you get the latest version of these files.
The RHESSI Flare Catalog is distributed as a text file as well as a monthly binary FITS and text files. Both are in the dbase directory of the RHESSI data archive, and are also distributed in the dbase directory of the SSW hessi tree. The flare catalog is automatically updated as soon as the Level-0 FITS files are created. A flare finding program searches new data for changes in the count rates consistent with flare signatures, and determines start, peak, and end times, peak rate, total counts, and a few other parameters. Currently there are no flare positions in the catalog, pending foolproof aspect solution software (expected early Spring, 2003). Click here for a discussion of the contents of the flare catalog, and here for information on how to read the flare catalog in IDL. Use the hsi_flarecat routine or the 'Browse Flare Catalog' button in the RHESSI GUI to use a widget interface to access the flare catalog.
The RHESSI Observing Summary Data (also called Quicklook Data) files contain various rates pre-binned to coarse energy and time resolution. These files are stored in daily FITS files in the metadata/catalog directory of the RHESSI data archive. They are named according to the date, e.g. hsi_obssumm_20030224_040.fits. The last three digits in the name are the version number. The file for the current day is constantly updated throughout the day (new version number with each update), and is usable at any time. The final file for each day is approximately 1.2 MB. These files can be read with any standard FITS reader, however we recommend using the RHESSI SSW software's specialized readers. You can use IDL routines at the command line (described here) to read and plot the observing summary data, or use the RHESSI GUI.
A companion file called hsi_obssumm_filedb.fits keeps a record of the current file name for each day. The filedb file is in the dbase directory of the RHESSI data archive, and is also distributed in the dbase directory of the SSW hessi tree. The RHESSI software determines the file name to read for your selected time interval by examining the hsi_obssumm_filedb.fits file (hence the importance of keeping the filedb and your local database of files in synch).
Note: Prior to ~November 2002, the observing summary data was appended to the Level-0 FITS files. Any reprocessing of the observing summary data required re-copying the large Level-0 files to the archives and user databases. That inconvenience prompted the switch to the separate daily observing summary files, which are much smaller and more manageable. The conversion to the daily files was completed in February, 2003.
The RHESSI Level-0 Data files contain the full raw telemetry data in packed format. These files are stored in the RHESSI data archive in directories organized by year. They are in the form of FITS files named according to the start date and time of the data in the file, e.g. hsi_20020704_011740_001.fits. The last three digits in the name are the version number. The files usually cover an orbit (~1.5 hours), but during periods of high activity they are limited to ~110 MB. Each successive file starts where the previous one ends, i.e. the files contain the data for all times, whether quiet or during a flare. Even though these files are in standard FITS format, they should be read only though the RHESSI SSW software due to the complicated unpacking algorithms. To create spectra, lightcurves, or images from the Level-0 data, you can use the IDL command line interface (described here) or the RHESSI GUI.
A companion file called hsi_filedb.fits keeps a record of the file names and the time intervals they cover. The filedb file is in the dbase directory of the RHESSI data archive, and is also distributed in the dbase directory of the SSW hessi tree. The RHESSI software determines the file name to read for your selected time interval by examining the hsi_filedb.fits file (hence the importance of keeping the filedb and your local database of files in synch).
The following types of plots are available for every orbit of
the mission in the metadata directory of the RHESSI data archive in directories
organized by year (2002,
Observing Summary Count Rates
Observing Summary Corrected Count Rates
Rates in the Front Segments of each Detector
Rates in the Rear Segments of each Detector
Daily and monthly summaries of data gaps are provided in text and plot form in the metadata/data_gap_files directory of the RHESSI data archive.
Daily averages of various state of health parameters are available in text and plot form for the entire mission in the metadata/hsi_1day_sohdata directory of the RHESSI data archive.
The directories at the top level of the RHESSI data archive are:
Simulated data files
Simulated data files
Level-0 data files for 2002
Level-0 data files for 2003
File database files. Includes filedb files (cross reference between time intervals and file names) for Level-0 and Observing Summary data, and flare list files. This directory is also distributed in the dbase directory of the SSW tree ($SSW/hessi/dbase).
metadata/2002 Orbit plots for 2002 metadata/2003 Orbit plots for 2003 metadata/catalog Daily Observing Summary Files metdata/data_gap_files Daily and Monthly data gap files metadata/hsi_1day_sohdata State of Health mission plots metadata/latest Latest orbit plots
Simulated data files
File database files for simulated data files. (Same as what's in $SSW/hessi/dbase_sim.)
The two primary types of data files the RHESSI Data Analysis Software needs access to are the Level-0 files and the Observing Summary files. Remote users must copy the files of interest to their own computer, either manually or through an automatic feature of the RHESSI software. Users at the primary archive sites can mount (or share) the data directories from a local file server.
Other key files the software needs are the filedb files (for both the Level-0 and Observing Summary files) and the flare catalog. These files are distributed with the SSW tree, so you should have the latest version after an SSW update. The filedb files contain a cross-reference between time intervals and file names. Occasionally there will be a mismatch between a data file you copied earlier, and an updated file pointed to by the filedb, requiring you to copy the new version of the data file.
The software uses two environment variables to find data files: it first looks in the HSI_DATA_ARCHIVE directory, then the HSI_DATA_USER (if defined) directory. Setting these environment variables is explained below.
Please see the
RHESSI FAQ for
more detailed information on RHESSI software and data setup.
The easiest way to copy the RHESSI data files to your computer is through the automatic search and copy feature of the RHESSI software (turned off by default). This is explained in detail in the RHESSI FAQ, but here is a short summary:
Another option is to copy all files for a specific time interval by using the RHESSI data search tool. This browser tool will find all the files you need for your time interval, create a zip file, and email you with the URL of that file. Or if you prefer to copy files individually, you can use a browser or an FTP program and the URLs and directories listed below. In either case, copy the files to the directory defined by HSI_DATA_ARCHIVE (or HSI_DATA_USER) or any subdirectory below HSI_DATA_ARCHIVE.
|Browser||Enter one of the following URLs depending on your
geographic location: |
Right-click on any file and select the save option to copy it to your computer. You can only copy one file at a time.
|FTP||Connect using anonymous ftp to the following site
and cd to the directory listed below. Then cd to the directory
containing the files you want to copy (listed
If your computer is on the same local network as one of the primary archives, you can mount or share the archive without copying files. See the examples below for the names of the archives at each archive site.
The environment variable HSI_DATA_ARCHIVE must point to the
location of the data files. If you use the automatic file copying
feature, you should also set HSI_DATA_USER, in a similar manner. There are
a number of ways to set environment variables in SSW. Three methods are shown
below. Use the hessi_var command in SSW IDL to
check the values of your environment variables.
|If you haven't done so already, copy the setup.hessi_env
file from the $SSW/hessi/setup directory to the $SSW/site/setup directory
(where $SSW means the root of the ssw tree on UNIX or Windows), and follow
the instructions in the file header. After editing the file, make
sure the line setting HSI_DATA_ARCHIVE which looks like:|
points to the location of your data files.
(On UNIX, you could copy the setup.hessi_env to your home
directory instead of the site setup directory. Also, on UNIX, to
make the change take effect, you must re-run setup.ssw, via
|Method 2||Add the following line to your idl_startup file:
|Method 3||After entering IDL (but before doing anything with RHESSI
type at the IDL prompt: |
Note: Method 1 should have no quotes, equal signs, or commas in the command, and Methods 2 and 3 require commas and quotes on both the second and third fields. 'full_data_path_name' should be replaced by the full path name of the data location. The syntax of the command is the same for UNIX and Windows, while the syntax of 'full_data_path_name' depends on the OS.
|Anywhere UNIX||If you have a copy of the data files on your UNIX machine
(e.g. in /hessi/data), then the required setup.hessi_env line is|
setenv HSI_DATA_ARCHIVE /hessi/data
|Anywhere Windows||If you have a copy of the data files on your PC (e.g. in
c:\hessi\data), then the required setup.hessi_env line is
|GSFC UNIX||The archive is on
hesperia.gsfc.nasa.gov/hessi/data1/hessidata which can be NFS-mounted on
your computer. The mount point name on your computer is arbitrary. If the
mount point is /hessi/data, then the required setup.hessi_env line is
|GSFC Windows||The archive is shared from hesperia as
\\hesperia.gsfc.nasa.gov\data1\hessidata. Sharing is done through
the SAMBA program. Your computer must be added to the SAMBA
configuration (ask Kim Tolbert or Terry Beck) to enable sharing to your
PC. The required setup.hessi_env line is |
|SSL UNIX||The archive is already NFS-mounted on
the Sun cluster, so the mount point is the same on all machines in the cluster. The required setup.hessi_env line is |
|SSL Windows||If you have NFS-client software on your PC (e.g.
Maestro), the archive can be NFS-mounted on your PC. The
network path to connect is \\soleil\mydisks\disk1\data. If you
connected soleil as drive h: then the required setup.hessi_env line is
The simulated data files that were created before the RHESSI launch are no longer valid. However, you can create your own simulations in SSW IDL.
Note that by default, the RHESSI software is set up to use flight data; attempts to use simulated data are aborted unless you explicitly switch to simulation mode.
To switch to simulated data and simulated aspect solution, type (in SSW IDL):
hsi_switch, /sim, /aspect_sim
To switch back to flight data, type:
hsi_switch, /flight, /aspect_flight
You can combine a flight aspect solution with simulated data by using the /sim, /aspect_flight keywords together.