Accessing RHESSI Data

Overview of RHESSI Data   Accessing RHESSI Data in the RHESSI Software
    Flare Catalog       Copying Files to Your Computer
    Observing Summary Data       Mounting or Sharing Local Archive
    Level-0 Telemetry Data       Setting Environment Variables
    And More...       Using Simulated Data
    Data Directory Structure    

 

Overview of RHESSI Data

There are two primary archive locations for the RHESSI data: GSFC (Maryland) and FHNW (Switzerland).  Users can access these locations through a browser or through anonymous ftp (GSFC only).  In addition, a few of the smaller data files are distributed with the SSW software; every time you update SSW, you get the latest version of these files.

Flare Catalog

The RHESSI Flare Catalog is distributed as a text file as well as a monthly binary FITS and text files.  Both are in the dbase directory of the RHESSI data archive, and are also distributed in the dbase directory of the SSW hessi tree.  The flare catalog is automatically updated as soon as the Level-0 FITS files are created.  A flare finding program searches new data for changes in the count rates consistent with flare signatures,  and determines start, peak, and end times, peak rate, total counts, and a few other parameters.  Currently there are no flare positions in the catalog, pending foolproof aspect solution software (expected early Spring, 2003).  Click here for a discussion of the contents of the flare catalog, and here for information on how to read the flare catalog in IDL.  Use the hsi_flarecat routine or the 'Browse Flare Catalog' button in the RHESSI GUI to use a widget interface to access the flare catalog.

Observing Summary Data

The RHESSI Observing Summary Data (also called Quicklook Data) files contain various rates pre-binned to coarse energy and time resolution. These files are stored in daily FITS files in the metadata/catalog directory of the RHESSI data archive.  They are named according to the date, e.g. hsi_obssumm_20030224_040.fits.  The last three digits in the name are the version number.  The file for the current day is constantly updated throughout the day (new version number with each update), and is usable at any time.   The final file for each day is approximately 1.2 MB.  These files can be read with any standard FITS reader, however we recommend using the RHESSI SSW software's specialized readers.  You can use IDL routines at the command line (described here) to read and plot the observing summary data, or use the RHESSI GUI

A companion file called hsi_obssumm_filedb.fits keeps a record of the current file name for each day.  The filedb file is in the dbase directory of the RHESSI data archive, and is also distributed in the dbase directory of the SSW hessi tree.  The RHESSI software determines the file name to read for your selected time interval by examining the hsi_obssumm_filedb.fits file (hence the importance of keeping the filedb and your local database of files in synch).

Note:  Prior to ~November 2002, the observing summary data was appended to the Level-0 FITS files.  Any reprocessing of the observing summary data required re-copying the large Level-0 files to the archives and user databases.  That inconvenience prompted the switch to the separate daily observing summary files, which are much smaller and more manageable.  The conversion to the daily files was completed in February, 2003.

Level-0 Telemetry Data

The RHESSI Level-0 Data files contain the full raw telemetry data in packed format.  These files are stored in the RHESSI data archive in directories organized by year.  They are in the form of FITS files named according to the start date and time of the data in the file, e.g. hsi_20020704_011740_001.fits.  The last three digits in the name are the version number.  The files usually cover an orbit (~1.5 hours), but during periods of high activity they are limited to ~110 MB.  Each successive file starts where the previous one ends, i.e. the files contain the data for all times, whether quiet or during a flare.  Even though these files are in standard FITS format, they should be read only though the RHESSI SSW software due to the complicated unpacking algorithms.  To create spectra, lightcurves, or images from the Level-0 data, you can use the IDL command line interface (described here) or the RHESSI GUI.

A companion file called hsi_filedb.fits keeps a record of the file names and the time intervals they cover.  The filedb file is in the dbase directory of the RHESSI data archive, and is also distributed in the dbase directory of the SSW hessi tree.  The RHESSI software determines the file name to read for your selected time interval by examining the hsi_filedb.fits file (hence the importance of keeping the filedb and your local database of files in synch).

And More...

The following types of plots are available for every orbit of the mission in the metadata directory of the RHESSI data archive in directories organized by year (2002, 2003, ...):
Observing Summary Count Rates
Observing Summary Corrected Count Rates
Rates in the Front Segments of each Detector
Rates in the Rear Segments of each Detector
Particle Rates

Daily and monthly summaries of data gaps are provided in text and plot form in the metadata/data_gap_files directory of the RHESSI data archive.

Daily averages of various state of health parameters are available in text and plot form for the entire mission  in the metadata/hsi_1day_sohdata directory of the RHESSI data archive.

Data Directory Structure

The directories at the top level of the RHESSI data archive are:

 

2000

Simulated data files

2001

Simulated data files

2002

Level-0 data files for 2002

2003

Level-0 data files for 2003

dbase

File database files.  Includes filedb files (cross reference between time intervals and file names) for Level-0 and Observing Summary data, and flare list files.  This directory is also distributed in the dbase directory of the SSW tree ($SSW/hessi/dbase).

metadata/2002 Orbit plots for 2002
metadata/2003 Orbit plots for 2003
metadata/catalog Daily Observing Summary Files
metdata/data_gap_files Daily and Monthly data gap files
metadata/hsi_1day_sohdata State of Health mission plots
metadata/latest Latest orbit plots

test

Simulated data files

test_dbase

File database files for simulated data files. (Same as what's in $SSW/hessi/dbase_sim.)

 

Accessing RHESSI Data in the RHESSI Software

The two primary types of data files the RHESSI Data Analysis Software needs access to are the Level-0 files and the Observing Summary files.   Remote users must copy the files of interest to their own computer, either manually or through an automatic feature of the RHESSI software. Users at the primary archive sites can mount (or share) the data directories from a local file server. 

Other key files the software needs are the filedb files (for both the Level-0 and Observing Summary files) and the flare catalog.    These files are distributed with the SSW tree, so you should have the latest version after an SSW update.  The filedb files contain a cross-reference between time intervals and file names.  Occasionally there will be a mismatch between a data file you copied earlier, and an updated file pointed to by the filedb, requiring you to copy the new version of the data file.

The software uses two environment variables to find data files:  it first looks in the HSI_DATA_ARCHIVE directory, then the HSI_DATA_USER (if defined) directory.  Setting these environment variables is explained below.

Please see the RHESSI FAQ for more detailed information on RHESSI software and data setup.
 

Copying Files to Your Computer

The easiest way to copy the RHESSI data files to your computer is through the automatic search and copy feature of the RHESSI software (turned off by default).  This is explained in detail in the RHESSI FAQ, but here is a short summary:

  1. Define HSI_DATA_USER to point to a directory to copy files to.
  2. In IDL, type search_network, /enable
  3. If your computer doesn't have a required file, it will copy the file from one of the archives (the closest geographically) to the HSI_DATA_USER directory.

Another option is to copy all files for a specific time interval by using the RHESSI data search tool.    This browser tool will find all the files you need for your time interval, create a zip file, and email you with the URL of that file.   Or if you prefer to copy files individually, you can use a browser or an FTP program and the URLs and directories listed below.  In either case, copy the files to the directory defined by HSI_DATA_ARCHIVE (or HSI_DATA_USER) or any subdirectory below HSI_DATA_ARCHIVE.

 
Browser Enter one of the following URLs depending on your geographic location:
http://hesperia.gsfc.nasa.gov/hessidata/
http://soleil.i4ds.ch/hessidata/

Right-click on any file and select the save option to copy it to your computer.  You can only copy one file at a time.

FTP Connect using anonymous ftp to the following site and cd to the directory listed below.  Then cd to the directory containing the files you want to copy (listed above).
 
FTP site Directory
hesperia.gsfc.nasa.gov hessidata
NOTE:

  1. Copy mode for ftp should be binary. 
  2. If an error is found in the files they may be regenerated.  In this case, you will need to copy a new set of files.

Mounting or Sharing Local Archive

If your computer is on the same local network as one of the primary archives, you can mount or share the archive without copying files. See the examples below for the names of the archives at each archive site.  

Setting Environment Variables

The environment variable HSI_DATA_ARCHIVE must point to the location of the data files. If you use the automatic file copying feature, you should also set HSI_DATA_USER, in a similar manner.  There are a number of ways to set environment variables in SSW. Three methods are shown below.  Use the hessi_var command in SSW IDL to check the values of your environment variables.
 
 
Method 1
(Preferred)
If you haven't done so already, copy the setup.hessi_env file from the $SSW/hessi/setup directory to the $SSW/site/setup directory (where $SSW means the root of the ssw tree on UNIX or Windows), and follow the instructions in the file header.  After editing the file, make sure the line setting HSI_DATA_ARCHIVE which looks like:
setenv HSI_DATA_ARCHIVE full_data_path_name
points to the location of your data files.

(On UNIX, you could copy the setup.hessi_env to your home directory instead of the site setup directory.  Also, on UNIX, to make the change take effect, you must re-run setup.ssw, via
source $SSW/gen/setup/setup.ssw )

Method 2 Add the following line to your idl_startup file:
set_logenv, 'HSI_DATA_ARCHIVE', 'full_data_path_name'
Method 3 After entering IDL (but before doing anything with RHESSI software), type at the IDL prompt:
set_logenv, 'HSI_DATA_ARCHIVE', 'full_data_path_name'

Note:  Method 1 should have no quotes, equal signs, or commas in the command, and Methods 2 and 3 require commas and quotes on both the second and third fields. 'full_data_path_name' should be replaced by the full path name of the data location.  The syntax of the command is the same for UNIX and Windows, while the syntax of 'full_data_path_name' depends on the OS.

Examples using Method 1 to set HSI_DATA_ARCHIVE
 
Anywhere UNIX If you have a copy of the data files on your UNIX machine (e.g. in /hessi/data), then the required setup.hessi_env line is
setenv HSI_DATA_ARCHIVE /hessi/data
Anywhere Windows If you have a copy of the data files on your PC (e.g. in c:\hessi\data), then the required setup.hessi_env line is
setenv HSI_DATA_ARCHIVE c:\hessi\data
GSFC UNIX The archive is on hesperia.gsfc.nasa.gov/hessi/data1/hessidata which can be NFS-mounted on your computer. The mount point name on your computer is arbitrary. If the mount point is /hessi/data, then the required setup.hessi_env line is
setenv HSI_DATA_ARCHIVE /hessi/data/
GSFC Windows The archive is shared from hesperia as \\hesperia.gsfc.nasa.gov\data1\hessidata. Sharing is done through the SAMBA program.  Your computer must be added to the SAMBA configuration (ask Kim Tolbert or Terry Beck) to enable sharing to your PC.  The required setup.hessi_env line is
setenv HSI_DATA_ARCHIVE \\hesperia.gsfc.nasa.gov\data1\hessidata
SSL UNIX The archive is already NFS-mounted on the Sun cluster, so the mount point is the same on all machines in the cluster. The required setup.hessi_env line is
setenv HSI_DATA_ARCHIVE /disks/hessidata
SSL Windows  If you have NFS-client software on your PC (e.g. Maestro), the archive can be NFS-mounted on your PC.  The network path to connect is  \\soleil\mydisks\disk1\data.  If you connected soleil as drive h: then the required setup.hessi_env line is
setenv HSI_DATA_ARCHIVE h:\hessi\test_data

 

Using Simulated Data

The simulated data files that were created before the RHESSI launch are no longer valid.  However, you can create your own simulations in SSW IDL. 

Note that by default, the RHESSI software is set up to use flight data; attempts to use simulated data are aborted unless you explicitly switch to simulation mode. 

To switch to simulated data and simulated aspect solution, type (in SSW IDL):

hsi_switch, /sim, /aspect_sim

To switch back to flight data, type:

hsi_switch, /flight, /aspect_flight

You can combine a flight aspect solution with simulated data by using the /sim, /aspect_flight keywords together.


Kim Tolbert
Last Modified: 12 January 2011