\documentclass[11pt]{article} \usepackage{epsf} \usepackage{natbib} \usepackage{aas_macros} \usepackage{hyperref} \newcommand{\lam}{$\lambda$} \newcommand{\kms}{km~s$^{-1}$} \newcommand{\cm}{cm$^{-3}$} \newcommand{\ion}[2]{#1\,{\sc #2}} \newcommand{\ecssa}{erg cm$^{-2}$ s$^{-1}$ sr$^{-1}$ \AA$^{-1}$} % erg/cm2/s/angstrom \newcommand{\ecss}{erg cm$^{-2}$ s$^{-1}$ sr$^{-1}$} % erg/cm2/s/sr \setlength{\oddsidemargin}{0cm} \setlength{\evensidemargin}{0cm} \setlength{\topmargin}{0cm} \setlength{\textwidth}{16cm} \setlength{\textheight}{23cm} \setlength{\parindent}{0cm} \setlength{\parskip}{0.1cm} \newcommand{\eisprep}{\textsc{EIS\_PREP}} \newcommand{\arcsec}{$^{\prime\prime}$} \newenvironment{centerpage}{\mbox{} \protect\vspace*{\fill}}{ \protect\vspace*{\fill} \mbox{} \protect\\ \mbox{}} \begin{document} \begin{titlepage} \hrule \noindent\textbf{EUV IMAGING SPECTROMETER}\\ \vspace{-0.7\baselineskip}\\ {\noindent\Huge\bf Hinode} \vspace{2mm} \hrule \vspace{3mm} \centerline{\bf EIS SOFTWARE NOTE No. 1} \vspace{3mm} \hrule \noindent Version 3.8 \hfill 26 Oct 2022 \vspace{2mm} \hrule \begin{centerpage} \begin{center} {\Large\bf Calibrating EIS data: the EIS\_PREP routine}\\ \mbox{}\\ \mbox{}\\ Peter Young\\ NASA Goddard Space Flight Center \\ Code 671 \\ Greenbelt, MD 20771, USA \end{center} \end{centerpage} \end{titlepage} \section{Calling \eisprep}\label{sect.call} \eisprep\ takes as input the name of a level-0 EIS FITS file and creates a level-1 data-set with the data arrays containing calibrated intensities. The standard calling sequence, and the one recommended for beginners, is: \begin{verbatim} IDL> eis_prep, l0name, /default, /quiet, /save, /retain \end{verbatim} The five keywords do the following: \begin{description} \item[/default] Tells \eisprep\ to use the standard, simple dark current subtraction method rather than use dark frames stored in Solarsoft. \item[/quiet] Stops \eisprep\ from popping up widget information messages during operation. \item[/save] Tells \eisprep\ to save the level-1 data into a FITS file in the current working directory, and also save the error arrays into a FITS file. These FITS files have the same name as the level-0 FITS file, except `l0' is replaced with `l1' and `er' for the intensity and error arrays, respectively. \item[/retain] Retains pixels with DN values with negative or zero values -- see Sect.~\ref{sect.dc} for more details. %% \item[/correct\_sensitivity] Applies a correction to the output %% intensities due to the decreasing EIS sensitivity with time. \end{description} \emph{Please read Sect.~\ref{sect.radcal} for information on how to implement the latest radiometric calibration.} The structure of the level-1 FITS file is identical to the level-0 FITS file except that the arrays that originally contained data numbers are modified to contain calibrated intensities. In addition, \eisprep\ also computes the 1$\sigma$ error on each intensity value and the arrays for these are stored in a separate FITS file. A complete set of keywords for \eisprep\ is given in Appendix~\ref{sect.keywords} \section{Missing data and eis\_prep} An important issue that users of \eisprep\ need to be aware of is the concept of `missing data'. For any EIS data-set there will be pixels within the data arrays that will not be usable for science purposes. They arise from a variety of causes, including saturated data, detector artifacts (dust, warm pixels, hot pixels), cosmic ray hits and missing data packets. One of the key jobs of \eisprep\ is to flag these pixels for the user. All of the flagged pixels are referred to as `missing data'. After \eisprep\ has completed its processing, each pixel in the data arrays will have an intensity value and an error value. The intensities and errors and stored in two separate files (see the next section). All missing data are given a value of -100 in the error file \emph{only}. The intensities of missing pixels are interpolated from neighboring pixels. At first glance, then, users may believe that missing pixels when viewed in the intensity arrays produced by \eisprep\ are normal and so suitable for science analysis. Since they do not have realistic errors assigned to them, however, then this is not the case. Some routines in the EIS software tree (for example eis\_auto\_fit) automatically check for pixels with errors set to -100 and will not include them in the processing. An alternative interpolation method (described in EIS Software Note \#13) for EIS data is available that assigns a realistic error bar to missing data so that the pixels can be used in science analysis. The method is not the default option in \eisprep\ as it erases the `memory' of where the missing pixels are (since they are no longer flagged with -100 values), but it can be initiated by using the keyword /refill. The interpolation method can also implemented as a post-\eisprep\ step by using the routine eis\_getwindata. For example: % \begin{verbatim} windata=eis_get_windata(l1name, 195.12, /refill) \end{verbatim} % where l1name is the name of the level-1 FITS file and 195.12 is the selected wavelength. \section{What does \eisprep\ do?}\label{sect.whatdo} The eis\_prep routine in Solarsoft takes the EIS level-0 FITS files and produces as output a level-1 FITS file. In addition to converting the measured CCD signal into calibrated intensity units, a key part of eis\_prep is to flag bad data points. These can arise through pixel saturation, cosmic ray hits, or simply defective pixels on the CCD. The bad points are referred to as missing pixels. The central outputs of EIS PREP are two level-1 FITS files, one containing calibrated intensities at each pixel, and the other containing error bars on these intensities. Note that the missing pixels are assigned a value of -100 in the error file, not the intensity file. The bad points in the intensity file are replaced with the median of the neighbouring pixels. The sequential steps performed by \eisprep\ are as follows. UPDATE: on 2019 December 4 an update to \eisprep\ was performed that modified the order of the steps below, introducing a new step for removing residual warm pixels, and modified the cosmic ray removal method. Please Check EIS Software Note No.~24 for further details. The sections below describe the \eisprep\ steps \emph{prior} to the change. The previous version of \eisprep\ can be run by giving the /original keyword. \subsection{Step 1: flagging saturated data, 0~DN and 2048~DN values}\label{sect.sat} There are three intensity values that need to be flagged as missing. The EIS CCDs have a 14 bit dynamic range and so saturation occurs at 16,383 data numbers (DN). Missing data packets can result in values of 0~DN, which are not real because normal pixels will always have a pedestal DN value of around 500~DN even if there are no counts. A further bad pixel value is 2048~DN. It is not known how this value arises but, when seen, an entire block of data will be set to this value. Since 2048~DN is potentially a real data value, then \eisprep\ will only flag this value if the entire Y-column on the detector takes the value. \subsection{Step 2: dark current and pedestal subtraction}\label{sect.dc} In the raw data, the spectra are found to sit on a background of around 500 DN that arises principally from the CCD bias (or pedestal), and secondly from the CCD dark current. Before launch it was expected that the bias and dark current could be subtracted together from the science data by using a dark frame (i.e., an exposure with the shutter closed), however tests showed that this was not successful mainly because the dark current is found to vary through the satellite's orbit (due to the CCD temperature varying). The most commonly-used method for estimating the bias and dark current is to use the science exposures themselves and this is enabled by the /default keyword. If /default is not specified, then the user must choose a dark frame either through the dc\_file= keyword, or via a widget selection window. For EIS data obtained with the 1\arcsec\ and 2\arcsec\ slits, there are two methods for estimating the bias and dark current that are employed in \eisprep, and the one used depends on the size of the wavelength windows used for the observation. Most rasters take narrow wavelength windows of size 16--32 pixels centered on specific emission lines. For these windows, each exposure is extracted and the 2\%\ of the detector pixels that have the lowest DN values are isolated. The median DN value of these 2\% pixels is then set to be the background level and it is subtracted from the DN values of each pixel. The maximum size of an EIS wavelength window is 1024 pixels and in this case a different method is applied. The reason for this lies in the properties of the EIS CCDs. EIS has two CCDs (for the two wavelength channels) and each has a width of 2048 pixels. The electronics method of reading the CCDs is such that each CCD can be effectively treated as if it is two CCDs of 1024 pixels stuck together. This means that the maximum size of a wavelength window is 1024 pixels, and in this case the window occupies the complete width of the ``effective CCD'' (one half of the actual CCD). Therefore if a window of 1024 pixels is selected to observe one half of the EIS SW channel, then it will always observe the complete spectrum in this half of the channel -- its position can not be adjusted. For this reason, the method for determining the pedestal in this case is to identify a narrow wavelength region within the window that is free of emission lines, and then take the median intensity value in this region. This value is then the pedestal value for the wavelength window. Table~\ref{tbl.pedestal} shows the spectral regions that are used to define the pedestal value for the four sets of 1024 wide pixel windows that are currently implemented in \eisprep\ (as of October 2012). Note that the 1\arcsec\ and 2\arcsec\ spectra are slightly offset from each other on the EIS detector, hence the fixed pixel ranges correspond to slightly different wavelength ranges for the two slits. \begin{table}[h] \caption{Spectral regions used to derive CCD pedestal values for 1024 pixel windows.} \begin{center} \begin{tabular}{cccc} \noalign{\hrule} \noalign{\smallskip} \noalign{\hrule} \noalign{\smallskip} %% table headings here &&\multicolumn{2}{c}{Wavelengths / \AA} \\ \cline{3-4} \noalign{\smallskip} Channel & Pixels & 1\arcsec & 2\arcsec \\ \noalign{\hrule} \noalign{\smallskip} %% table entries here SW 1 & 39--84 & 167.10--168.10 & 167.27--168.28 \\ SW 2 & 944--989 & 210.10--211.10 & 210.27--211.28 \\ LW 1 & 39--84 & 246.59--247.59 & 246.77--247.77 \\ LW 2 & 926--971 & 289.11--290.11 & 289.29--290.29 \\ \noalign{\hrule} \end{tabular} \end{center} \label{tbl.pedestal} \end{table} Generally, when users select a window of 1024 pixels they will do so for each of the four ``effective CCDs'', and such data-sets are usually referred to as full CCD data-sets. However, it is possible for users to combine a 1024 pixel window with narrower windows. In this case a potential anomaly can occur whereby the 1024 pixel window may appear to have a significantly different continuum level compared to the other windows after running \eisprep\ -- this is because of the different methods of estimating the pedestal. % For full CCD spectra a different approach is used. Sections of the CCD have been identified where there are no (or very weak) emission lines and so it can be assumed that the background here represents a measure of the CCD background. The regions are then averaged to yield the background level which is subtracted from the spectra. It is to be noted that both of the pedestal removal methods will yield some pixels with negative DN values. For window data it will only be 1\% of pixels because of the method used, but for full CCD spectra it can be up to 50\% of pixels if the raster contains very weak emission (e.g., coronal holes or off-limb spectra). The default mode for \eisprep\ sets pixels with zero or negative DN values to be missing data. This is because it is not possible to assign a photon noise error to the data points. (Note that negative DN values are expected since, if there are zero solar counts, then the uncertainty in the CCD counts will be 0 $\pm$ read noise.) By setting the /RETAIN keyword in \eisprep, pixels with zero or negative DN values will not be flagged as missing. The errors for the pixels will be set to the dark current error estimate (see Step 6). For EIS slot data (obtained with the 40\arcsec\ and 266\arcsec\ slits), fixed values for the pedestal and dark current are set at 549, 500, 556 and 547~DN for the four CCD sectors identified in Table~\ref{tbl.pedestal}. \subsection{Step 3: cosmic ray removal} Anomalously bright pixels are found on the EIS CCD images that arise from ‘hot pixels’, ‘warm pixels’ and cosmic rays. The cosmic ray removal is performed by EIS DESPIKE, a wrapper routine that calls NEW SPIKE, a routine developed for removing cosmic rays from SOHO/Coronal Diagnostic Spectrometer (CDS) data-sets \citep{cds_sw46,2000A&A...362L..21P}. For CDS data processing it was typical for not only the identified CCD pixels to be flagged, but also the nearest-neighbour pixels on the CCD. This is because there is often residual signal from the cosmic ray next to the brightest pixels. EIS sees significantly less cosmic rays than CDS apart from during the approximately 5 minute passes through the South Atlantic Anomaly. As the most useful function of EIS DESPIKE was actually to flag warm pixels, and since warm pixels are only single pixel events, then the nearest-neighbour option is usually switched off for EIS. It is to be noted that the NEW SPIKE routine was designed to be cautious when removing cosmic rays from line profiles, thus possibly artificially enhancing the emission line intensities at these locations. \subsection{Step 4: flagging hot and warm pixels}\label{sect.hotwarm} Both hot pixels and warm pixels are single pixels that have anomalously high DN values. A hot pixel is defined to be one that yields 25,000 electrons pixel-1 s-1 at room temperature (a specification from the CCD manufacturer). Pixels that fall below this threshold but are still clearly identified as being anomalous when inspecting the data are referred to as ‘warm’ pixels. Separate maps of the locations of hot and warm pixels are generated by the EIS team approximately every week following inspection of 100 s dark exposures and they are stored in Solarsoft. The pixel maps that are closest in time to the science observation are used by EIS\_PREP to mark the hot and warm pixels as missing data. From 4 October 2022 onwards, the EIS team are now using 150~s dark exposures rather than 100~s exposures, as these enable 10\%\ more warm pixels to be identified. The 100~s exposures are continued but on a monthly basis in order to maintain continuity with the original dataset. Warm pixels have a significant impact on the analysis of EIS data and users are recommended to read EIS Software Note \#6, `Warm and hot pixels on the EIS CCDs', and EIS Software Note \#13, `Interpolating missing pixels in EIS data'. \subsection{Step 5: flagging dusty pixels} The next step for EIS PREP is to flag the pixels affected by dust on the CCD. Several small pieces of dust accumulated on the CCD before launch and are found to completely block the solar signal on the CCD at their locations. They are fixed in position and cover less than 0.1\% of the CCD, however two of the pieces do affect the strong lines Fe XI 188.23, 188.30 and Fe XII 193.51 such that the lines can not be used over 15--30" spatial ranges in solar-Y. \subsection{Step 6: radiometric calibration}\label{sect.radcal} The final step of EIS PREP is to convert DN values into intensities in units \ecssa. The errors on the intensities are computed assuming photon statistics together with an error estimate for the dark current. This varies from 2.24 to 2.37 DN for the four CCD quadrants. To return units of photons~cm$^{-2}$~s$^{-1}$~sr$^{-1}$, the keyword \textbf{/photons} should be set. To not perform radiometric calibration at all, then the keyword \textbf{/noabs} should be used. This leaves the intensity in DN units. \citet{2016ApJ...820...63B} found that leaving the intensities in DN units results in slightly narrower widths when fitting emission lines with Gaussians. Please check Appendix~\ref{sect.noabs} for further information about the \textbf{/noabs} keyword. The intensity units can be checked by using the \textsf{getunit} object method. The call recommended in Sect.~\ref{sect.call} results in the original laboratory calibration to be applied to the data. However, it is known that the calibration has changed since launch and two alternative prescriptions were presented by \citet{2013A&A...555A..47D} and \citet{2014ApJS..213...11W}. \emph{It is generally recommended to apply one of these corrections during the data analysis rather than at the eis\_prep stage.} One example is when using the eis\_auto\_fit suite of software (EIS Software Note \#16). After fitting the lines, the intensity array is generated with: % \begin{verbatim} IDL> int=eis_get_fitdata(fitdata,/int,calib=calib) \end{verbatim} % where calib=1 implements the \citet{2013A&A...555A..47D} calibration, and calib=3 implements the \citet{2014ApJS..213...11W} calibration. The EIS Wiki page at % \begin{verbatim} http://solarb.mssl.ucl.ac.uk/eiswiki/Wiki.jsp?page=EISCalibration \end{verbatim} % has further information on implementing the calibration. Prior to the \citet{2013A&A...555A..47D} and \citet{2014ApJS..213...11W} work, users were recommended to use the /correct\_sensitivity keyword when calling eis\_prep. Please see Appendix~\ref{sect.corrsens} for more information on this keyword. \section{Calculating intensities and errors} The level-0 FITS file contains intensities stored as `data numbers', or DN, and the main job of \eisprep\ is to convert these to calibrated intensities with associated 1$\sigma$ errors. The steps in this process are as follows. The DN value, $D$, for a given pixel consists of three components: \begin{equation} D=D_{\rm ped}+D_{\rm dc}+D_{\rm photon} \end{equation} where $D_{\rm ped}$ is the CCD pedestal, $D_{\rm dc}$ is due to the dark current, and $D_{\rm photon}$ is the signal from the incident photons. An average value of $D_{\rm ped}+D_{\rm dc}$ is estimated from the data themselves through one of the methods described in Sect.~\ref{sect.dc}. We refer to this value as $D_{\rm sub}$. The number of photons in any pixel on the detector is then given by: \begin{equation}\label{p_defn.eq} P = (D-D_{\rm sub}) g \left( {\lambda\over {\rm \AA}}\right) {3.65\over 12398.5} \end{equation} where $g$ is the CCD gain, 3.65 eV is the energy to produce an electron-hole pair in silicon, and 12398.5 is the conversion factor for eV into \AA. The photon statistics noise is given by $\sqrt P$, however the error on the dark current estimate must be added in quadrature to this: \begin{equation} \sigma_P= \sqrt{ P + \sigma_{\rm dc}^2} \end{equation} where $\sigma_{\rm dc}$ is the dark current uncertainty. The latter has been estimated directly from EIS dark frames and takes a value 2.24--2.37~DN (as of 2009 January 5), depending on the CCD quadrant. The value $\sigma_{\rm dc}$ is then derived from these values using Eq.~\ref{p_defn.eq}. To convert from photons to \ecssa\ requires a number of factors including exposure time, telescope effective area and slit size. Note that effective area itself consists of a number of factors including filter transmissions, coating reflectivities, telescope size and CCD quantum efficiency. The effective area for a given wavelength can be obtained with the IDL routine EIS\_EFF\_AREA(). Writing these factors as simply $f(\lambda)$, the calibrated intensities are $I=f(\lambda)P$. The 1$\sigma$ error on $I$ is then given by \begin{equation} {\sigma_I \over I} = {\sigma_P \over P} \end{equation} or \begin{equation} \sigma_I=f(\lambda) \sigma_P \end{equation} % A pixel with associated wavelength, \lam, registers a data number % value of $d_{\lambda}$. The data number value is converted to a photon % value by the expression % \begin{equation} % p_{\lambda}=k\,d_{\lambda}\lambda % \end{equation} % where $k$ is a constant \bibliographystyle{aasjournal} \bibliography{eis_swnote_01} %$ \newpage \appendix \section{Full set of \eisprep\ keywords}\label{sect.keywords} \begin{description} \item[noabs] Do not perform absolute calibration, i.e., the intensity arrays are left in data number units. \item[photons] Instead of \ecssa\ the output intensities are given in units of photons~cm$^{-2}$~s$^{-1}$~sr$^{-1}$~\AA$^{-1}$. \item[refill] This uses the method described in Software Note \#13 to replace missing pixels with a 'sensible' intensity value. Generally the user is recommended not to use this keyword with \eisprep\ but use it instead with one of the post-processing routines such as \textsc{EIS\_GETWINDATA}. \item[nocr] Switch off cosmic ray detection routine. \item[nohp] Switch off hot pixel detection routine. \item[nowp] Switch off warm pixel detection routine. \item[nodp] Switch off dusty pixel detection routine. \end{description} \section{Comment on /noabs keyword}\label{sect.noabs} If the /noabs keyword is set, then the ``intensity'' arrays returned by \eisprep\ have DN units. Technically a 1-sigma error can be assigned to these values by computing photon statistic errors and converting back to DN values, but \eisprep\ does not do this. However, \eisprep\ \emph{does} produce an error FITS file when /noabs is set. The error values in this FITS file are not correct and should not be used, however the error arrays do contain the positions of missing pixels by flagging them with -100. This is important when using the /refill keyword (either for \eisprep\ or \verb|eis_getwindata|). The /refill keyword interpolates missing values, and this works correctly if /noabs is set. However, the interpolated error values should again not be used. \section{The /correct\_sensitivity keyword}\label{sect.corrsens} This is no longer recommended to be used, but we give details on it here for archive purposes. EIS Software Note \#2 described changes in the EIS sensitivity with time and this was implemented with the /correct\_sensitivity keyword for eis\_prep. The sensitivity decay was modeled as an exponential decay of the form $\exp(-t/\tau)$ where $t$ is the time of the observation in days since the Hinode launch (2006 September 22 21:36~UT), and $\tau$ is the decay time in days. The value of $\tau$ can be obtained by doing: % \begin{verbatim} IDL> cal=obj_new('eis_cal') IDL> print,cal->gettau_sensitivity() \end{verbatim} % Note that the sensitivity decay is modeled as being the same for all wavelengths. The crucial advances of the \citet{2013A&A...555A..47D} and \citet{2014ApJS..213...11W} implementations were to derive wavelength-dependent changes with time. \section{Document modification history} \emph{version 3.8}: Updated Sect.~\ref{sect.hotwarm} to add information about new 150~s dark exposures; updated Sect.~\ref{sect.radcal} to mention /photons and /noabs keywords. \emph{version 3.7}: Updated Sect.~\ref{sect.sat} to mention 0 and 2048~DN values; updated Sect.~\ref{sect.whatdo} to mentioned EIS Note No.~24. \emph{version 3.6}: Updated recommendation for radiometric calibration (Sects.~\ref{sect.call}, \ref{sect.radcal} and App.~\ref{sect.corrsens}). \emph{version 3.5}: Added Sect.~\ref{sect.noabs}. \emph{version 3.4}: Sect.~\ref{sect.dc} has been expanded. \emph{version 3.3}: The /correct\_sensitivity keyword has been restored as the implementation is now working correctly. \emph{Version 3.2}: Changed document title. Removed /correct\_sensitivity from the recommended eis\_prep calling sequence (Sect.~\ref{sect.call}) as there is a bug in the current implementation. Also changed l1name to l0name in the IDL call line. \end{document}