\documentclass[11pt]{article} \usepackage[dvips]{graphicx} %% FOR INCLUDING GRAPHICS \usepackage{epsf} %% ---------------------------------------------------------------------- %% Add specials here %% ---------------------------------------------------------------------- \def\ion#1#2{#1\,{\sc #2}\relax} %% %% ---------------------------------------------------------------------- %% Modify: %% Note number, version number and date %% Title %% Author and affiliation %% ---------------------------------------------------------------------- \setlength{\oddsidemargin}{0in} % 1 inch (0 + 1) from left. \setlength{\topmargin}{-0.5in} % 1 inch (1.5 - 0.5) from top. \setlength{\textwidth}{6.5in} % 1 inch (8.5 - 1 - 6.5) from right. \setlength{\textheight}{9in} % 1 inch (11 - 1 - 9) from bottom. \addtolength{\parskip}{0.5\baselineskip} \newcommand{\lam}{$\lambda$} \newcommand{\arcsec}{$^{\prime\prime}$} % The environment centerpage centers text vertically on the page. This is % useful within a titlepage environment. \newenvironment{centerpage}{\mbox{} \protect\vspace*{\fill}}{ \protect\vspace*{\fill} \mbox{} \protect\\ \mbox{}} \begin{document} \bibliographystyle{plain} % Define bibliography style. \begin{titlepage} \hrule \noindent\textbf{EUV IMAGING SPECTROMETER}\\ \vspace{-0.7\baselineskip}\\ {\noindent\Huge\bf Hinode} \vspace{2mm} \hrule \vspace{3mm} \centerline{\bf EIS SOFTWARE NOTE No. 24} \vspace{3mm} \hrule \noindent Version 1.2 \hfill 17 December 2019 \vspace{2mm} \hrule \begin{centerpage} \begin{center} {\Large\bf EIS\_PREP: removing residual warm pixels and cosmic rays}\\ \mbox{}\\ \mbox{}\\ Peter Young \\ NASA Goddard Space Flight Center \\ Code 671 \\ Greenbelt, MD 20771, USA \end{center} \end{centerpage} \end{titlepage} %% ---------------------------------------------------------------------- %% The document! %% ---------------------------------------------------------------------- \section{Overview} This document describes procedures to remove residual warm pixels and cosmic rays from EIS data as part of the calibration process implemented in the routine EIS\_PREP. These procedures were introduced in 2019, and they required some changes to how EIS\_PREP operates. The new procedures apply to narrow slit data only. Sect.~\ref{sect.slot} discusses the slot data. \section{EIS\_PREP procedures prior to 2019}\label{sect.intro} The default procedure for EIS\_PREP prior to 2019 was to do the following sequence: % \begin{enumerate} \item Flag cosmic rays using the routine EIS\_DESPIKE (/nocr). \item Flag hot pixels using hot pixel maps stored in Solarsoft (/nohp). \item Flag warm pixels using warm pixel maps stored in Solarsoft (/nowp). \item Flag dusty pixels using dust pixel maps stored in Solarsoft (/nodp). \end{enumerate} % Further details are given in EIS Software Notes Nos.~1 and 6. The keywords indicated in the brackets above are used to switch off the steps when calling EIS\_PREP. Inspection of the level-1 data produced by EIS\_PREP shows that many warm pixels are not removed, either because they fall below the threshold used in creating the warm pixel maps, or because the warm pixels were not present in the maps (which are obtained on a weekly basis). I refer to these as \emph{residual warm pixels} (RWPs). Examples from a data-set beginning at 12:07~UT on 19-Mar-2016 are shown in Figure~\ref{fig.residual}. Time--Y images from the level-1 \ion{Fe}{ix} \lam197.86 window are shown; the left panel is from wavelength pixel 11 (within the line), and the right panel from pixel 20 (in the continuum). Significant resdiual warm pixels can be seen in each. Potentially the RWPs could be removed by the EIS despiking routine, but since this routine is optimized to remove spikes that occur in individual exposures then it is not ideal. In addition, RWPs are often adjacent to regular, stronger WPs, and since the despiking routine is applied prior to warm pixel removal then it is not able to flag the RWP in this case. In addition to the RWP problem, it was also noticed that the despiking routine often flags many pixels that are clearly just noise. Figure~\ref{fig.despike} shows an example from the data-set beginning at 12:07~UT on 19-Mar-2016. Wavelength pixel 17 from the \ion{Fe}{ix} \lam197.86 window was chosen, and the images show the time--Y slice prior to cosmic ray cleaning (left panel) and the locations of the flagged cosmic rays (right panel). \begin{figure}[t] \centerline{\epsfxsize=5.5in\epsfbox{plot_residual_example.eps}} \caption{Time--Y images for two wavelength pixels in the \ion{Fe}{ix} \lam197.86 window are shown. Residual warm pixels are the narrow horizontal lines in the images.} \label{fig.residual} \end{figure} \begin{figure}[h] \centerline{\epsfxsize=5.5in\epsfbox{plot_despike_example.eps}} \caption{The left panel shows an image from an EIS data-set prior to cosmic ray cleaning. The right panel shows the pixels (black) that were flagged as cosmic rays by EIS\_DESPIKE.} \label{fig.despike} \end{figure} \section{New procedures for EIS\_PREP} The revised operating sequence for EIS\_PREP is as follows: % \begin{enumerate} \item Flag dusty pixels using dust pixel maps stored in Solarsoft (/nodp). \item Flag hot pixels using hot pixel maps stored in Solarsoft (/nohp). \item Flag warm pixels using warm pixel maps stored in Solarsoft (/nowp). \item Flag residual warm pixels with EIS\_PREP\_RESIDUAL\_WPS (/norwp). \item Flag cosmic rays using the routine EIS\_PREP\_CLEAN\_CRS (/nocr). \end{enumerate} % An important distinction between dusty pixels and the other types is that EIS\_PREP does not try to interpolate the intensity over the dusty regions from neighboring regions. For this reason it is now moved to the first step. The other four steps are now performed consecutively on each data window in one block of code, whereas previously each had a separate block of code. The reason for this is the data interpolation. For example, consider a hot pixel directly adjacent in the Y-direction to a warm pixel. After the hot pixel has been flagged, the software tries to replace the intensity with a median of the two neighboring pixels. Since the warm pixel has not yet been flagged (since this is a later step in the code) then it is possible that the hot pixel is replaced with the warm pixel intensity. With the new code, the positions of the four types of missing data get stored in a single array, and then the interpolation is applied to this array. This greatly reduces the number of data artifacts seen in the level-1 data. (It is worth highlighting that the interpolated intensity array produced by EIS\_PREP should not be used for data analysis, only quicklook purposes. The flagged pixels should either be ignored, i.e., treated as missing, or replaced using the /refill keyword.) \section{How to call EIS\_PREP with the new method}\label{sect.call} EIS\_PREP is called in exactly the same manner as before, with the recommended calling sequence (see also the EIS Wiki): % \begin{verbatim} IDL> eis_prep, l0name, /refill, /quiet, /save, /default \end{verbatim} A new keyword, /original, has been added which will run EIS\_PREP exactly how it was prior to the implementation of the new method (which was introduced on 2019 December 4). The earlier code was renamed EIS\_PREP\_ORIGINAL and thus /original switches to calling this routine. If you want to swich off the RWP or CR removal steps from the new EIS\_PREP, then use the keywords /norwp and /nocr. \section{How to check the RWP and CR removal}\label{sect.check} The key difference with the previous version of EIS\_PREP is that the cosmic ray removing routine, EIS\_DESPIKE, is no longer used, being replaced by EIS\_PREP\_CLEAN\_CRS, and an additional routine, EIS\_RESIDUAL\_WPS, is used to flag residual warm pixels that were not previously removed. To test how well the EIS\_RESIDUAL\_WPS and EIS\_PREP\_CLEAN\_CRS routines work for a given data-set, the following procedure can be run. First, EIS\_PREP is run but with the /norwp, /nocr and /noabs keywords set: % \begin{verbatim} IDL> eis_prep,l0name,/quiet,/save,/default,/retain,/norwp,/nocr,/noabs \end{verbatim} Now read a data window from the level-1 file: % \begin{verbatim} IDL> wd=eis_getwindata(l1name,wvl) \end{verbatim} The windata structure can then be ``cleaned'' with: % \begin{verbatim} IDL> wdout=eis_clean_windata(wd,swtch=swtch) \end{verbatim} % where SWTCH is an array containing the locations of the pixels that have been cleaned, i.e., residual warm pixels and cosmic rays removed. A GUI-based routine can then be used to browse the before-and-after arrays, with the flagged pixels shown: % \begin{verbatim} IDL> eis_spike_viewer,wd,wdout,swtch \end{verbatim} % The routine shows a slice through the intensity arrays of WD (left) and WDOUT (middle), and the SWTCH array (right). The latter is an array of 0's and 1's of same size as the intensity arrays, with 1 indicating a flagged pixel. There are options to show exposures ($\lambda$ vs.\ Y) or images (X vs.\ Y). The `Zoom' and `Pixel' options control the behavior of the left mouse button on the images. For the former, clicking-and-dragging creates a rubber-band box allowing you to zoom in to the images. The latter allows you to click on individual pixels, and the three bottom panels show cuts through this pixel in the $\lambda$, X and Y directions. \section{Slot data}\label{sect.slot} The slot data (i.e., the 40\arcsec\ and 266\arcsec\ slits) are not processed for residual warm pixels, nor are cosmic rays removed with EIS\_PREP\_CLEAN\_CRS. Instead, the old routine EIS\_DESPIKE is used but a new keyword /CLEAN\_EXP has been added to force the routine to use CLEAN\_EXPOSURE. \section{Removing residual warm pixels} The criteria for flagging a warm pixel are applied to a 2D image of wavelength $\times$ solar-Y (i.e., an EIS exposure). Any warm, hot, dusty or saturated pixels previously identified by EIS\_PREP are set to the missing value ($-100$). A comparison of the pixel's intensity with the median of surrounding pixels is the basis of the method. A feature of warm pixels is that they affect every exposure in the raster, and so the routine looks for pixels that are consistently flagged in a majority of the exposures. If one is found, then the pixel will be flagged in all of the exposures, irrespective of whether the pixel in an individual exposure was flagged. The method is complicated by the missing pixels, which potentially introduce a bias in the median value that can affect whether a pixel is flagged or not. To illustrate the potential problems, consider the case where a pixel lies in the wing of a strong line, say at the full-width at half-maximum intensity level. The basic check is to compare the pixel with the median of the surrounding 3x7 pixel block. Because of the strong slope in the line intensity across the three wavelength pixels, the median would essentially come from the central 1x7 block. However, consider the case where there are, e.g., 5 missing pixels in the 1x7 block on the high intensity side. Potentially the median pixel would then come from the low intensity 1x7 block, which is much weaker than the central 1x7 block. The pixel then has a much higher intensity than this median pixel and so it gets flagged, even though it may have a normal intensity. This specific problem is dealt with by, in addition to checking the 3x7 block, the routine also checks against the median of the central 1x7 block. A pixel is flagged if it satisfies both criteria. However, there is then potentially a further problem. Suppose the intensity in the Y-direction is monotonically increasing from bottom to top. Further suppose that the top 3 pixels are all missing. The median is obtained from the three low intensity pixels and there is a good chance that the central pixel is flagged by accident. The routine checks for this by requiring there be at least one pixel on both the top and bottom side of the pixel. (It should be noted that if the median is computed from an even number of elements, then IDL always returns the higher number rather than the lower number. For example, for [1,2,3,4] the median is 3, not 2. This helps reduce the number of spurious flagged pixels when there are missing data.) Three checks are done on RWPs based on the pixel intensity, $I$, (given in data numbers, DN). A limit of $L$~DN separates weak and strong pixels, and a further limit of 1000~DN separates strong from very strong pixels. %% Three categories of warm pixel are considered: (i) intensity $< L$~DN %% (weak), (ii) intensity $\ge L$~DN (strong), and (iii) intensity $\ge 1000$~DN %% (very strong). The limit, $L$, is defined by first taking a histogram of the image with values between $-30$ and 50~DN. We then set $MVL$ (\verb|max_var_low| in the code) to be the DN value at which the histogram peaks. If this value is less than 6, then it is set to 6 (this value is a heritage value from EIS\_DESPIKE). $L$ is then set to $(MLV/1.5)^2$. For further details we refer the reader to the IDL code, which is extensively commented. The various parameters and checks were arrived at by extensive study of a number of data-sets. If you use EIS\_SPIKE\_VIEWER to look at dark areas of the detector then you will find the routine usually flags the strongest RWPs, but there often remain many very weak RWPs. Although the eye can pick these out fairly easily they are typically only a few DN in strength, and trying to flag them will often result in false positives. RWPs occurring in line profiles are more difficult to flag due to the risk of removing real data, and so less are flagged compared to dark areas. The routine can have problems when multiple warm or hot pixels are found in adjacent pixels. If these are present in the WP/HP maps, then they are flagged as missing and there may not be enough good pixels remaining to identify any RWPs. \section{Removing cosmic rays} The procedure for CRs is similar to that for RWPs in that pixels are identified based on comparisons with the median value of pixels in the neighborhood of a single exposure. Note that EIS\_DESPIKE performed checks in the exposure dimension (time) in addition to the 2D detector image. This method is generally more effective for accurately identifying CRs, but was not used here as there are many EIS data-sets for which the raster step size is larger than the slit width (so-called ``picket-fence'' rasters). There is then the risk of false-positive identifications due to large, real intensity variations between slit positions with a large spatial separation. The checks performed for flagging CRs are performed in EIS\_CR\_CHECKER, and there are five checks in all. One is performed on weak intensity pixels, the rest on strong intensity pixels. The value of the pixel's 3x7 median is used to determine if the pixel lies within an emission line. If yes, then the routine also checks the 1x7 median as described in the RWP section above. The locations of the weak and strong intensity CRs are stored separately by EIS\_CR\_CHECKER. EIS\_PREP\_CLEAN\_CRS then flags neighbors of the strong CRs, but not for the weak CRs (which generally do not extend into neighboring pixels). EIS\_PREP\_CLEAN\_CRS then performs another pass over the data using EIS\_CR\_CHECKER, which picks up residual CRs from the large events. \section{Known issues} \begin{itemize} \item Warm pixels and cosmic rays occurring in line profiles are more difficult to flag due to the risk of removing real data. \item Warm and hot pixels can be found in adjacent pixels, sometimes 3, 4 or 5 together. If some of these are removed using the WP and HP maps, leaving missing data, then the routine will often fail to remove a RWP simply because there are no good data nearby. \item Using the spike viewer software, many very weak warm pixels can be seen in dark areas with the eye. Generally these are not flagged because the signal is too weak. \end{itemize} \begin{figure}[t] \centerline{\epsfxsize=5.5in\epsfbox{plot_noresidual_example.eps}} \caption{The same data as shown in Figure~\ref{fig.residual} are displayed, but after processing with the new version of EIS\_PREP.} \label{fig.noresidual} \end{figure} \section{Results}\label{sect.results} In Sect.~\ref{sect.intro}, data from 2016 was shown that illustrated the problems with residual warm pixels and cosmic ray removal. Figure~\ref{fig.noresidual} shows the same plots as Figure~\ref{fig.residual} but after applying the new version of EIS\_PREP. It can be seen that the strong WP in the left panel is no longer present, and also several of the stronger WPs in the right panel have been removed. Figure~\ref{fig.despike} showed that the EIS\_DESPIKE routine originally used by EIS\_PREP flagged many low intensity pixels as CRs. With the new version, the region shown in Figure~\ref{fig.despike} has no pixels flagged as CRs. \begin{figure}[h] \centerline{\epsfxsize=\textwidth\epsfbox{20191125_image_comparison.eps}} \caption{Images in three emission lines from the 25-Nov-2019 12:13~UT dataset. The top row shows results from the old EIS\_PREP data and the bottom row shows results from the new EIS\_PREP data.} \label{fig.20191125} \end{figure} Figure~\ref{fig.20191125} compares images formed in three emission lines (\ion{Fe}{xii} \lam195.12, \ion{Fe}{ix} \lam197.86 and \ion{Si}{vii} \lam275.36) using the previous version of ES\_PREP, and the new version. The data-set is from 2019 November 25 at 12:13~UT, and the images were formed by averaging the seven wavelength pixels centered on the lines' rest wavelengths. Only a fraction of the Y field-of-view is shown. The data were processed using the /refill option, and any remaining missing pixels were filled using EIS\_PREP\_MED\_FILL. The lines were chosen as they are examples of strong, medium and weak strength lines. The \lam195.12 images are almost identical. The \lam197.86 images show one quite strong WP that is mostly removed with the new method, and other weaker WPs that have been removed (e.g., the two around Y=180--185). The \lam275.36 images show one strong WP at Y=170 that is mostly removed, and a number of cosmic rays around Y=230 (these are probably remnants of a WP that were partially removed with the previous method. The \lam275.36 image shows that many weak warm pixels remain in the data with the new method. The number is exaggerated in this plot since the data are averaged over seven images. Potentially more of these WPs could be removed by adjusting the detection threshold. Figure~\ref{fig.20191125-gauss} compares Gaussian fits to \ion{Fe}{ix} \lam197.86, with the top row again showing the results from the old version of EIS\_PREP and the bottom row showing the new version. The fits were performed with EIS\_AUTO\_FIT, and the left column shows integrated intensity, and the right column shows Doppler shifts. It can be seen that the WP highlighted in Figure~\ref{fig.20191125} makes an effect on the Gaussian fits. Otherwise the intensity and velocity images are very similar, with slightly less artifacts in the velocity image. \begin{figure}[h] \centerline{\epsfxsize=\textwidth\epsfbox{20191125_fe9_fits.eps}} \caption{Fit results for \ion{Fe}{ix} \lam197.86 from the 25-Nov-2019 12:13~UT dataset. The top row shows results from the old EIS\_PREP data and the bottom row shows results from the new EIS\_PREP data. The left column shows integrated intensity, and the right column shows line-of-sight velocity scaled between $\pm$~20~km~s$^{-1}$.} \label{fig.20191125-gauss} \end{figure} \newpage \appendix \section{Document history} \noindent\emph{Ver.~1.2, 17-Dec-2019}. Added Sect.~\ref{sect.slot} to mention slot data. \noindent\emph{Ver.~1.1, 11-Dec-2019.} Added further examples to Sect.~\ref{sect.results}. Added Sect.~\ref{sect.call}, and renamed Sect.~\ref{sect.check}. \noindent\emph{Ver.~1.0, 5-Dec-2019.} First release. \end{document}