Acta Polytechnica doi:10.14311/AP.2014.54.0275 Acta Polytechnica 54(4):275–280, 2014 © Czech Technical University in Prague, 2014 available online at http://ojs.cvut.cz/ojs/index.php/ap USING PHOTOPLETHYSMOGRAPHY IMAGING FOR OBJECTIVE CONTACTLESS PAIN ASSESSMENT Marcus Koenya, ∗, Nikolai Blanika, Xinchi Yua, Michael Czaplikb, Marian Waltera, Rolf Rossaintb, Vladimir Blazeka, Steffen Leonhardta a Chair for Medical Information Technology, Helmholtz-Institute, RWTH Aachen University, Pauwelsstrasse 20, 52074 Aachen, Germany b Department of Anesthesiology, University Hospital Aachen, Pauwelsstrasse 30, 52074 Aachen, Germany ∗ corresponding author: koeny@hia.rwth-aachen.de Abstract. This work presents an extension to the known Analgesia Nociception Index (ANI), which provides an objective estimation of the current depth of analgesia. An adequate “measure” would facilitate so-called balanced anesthesia. Generally, ANI is computed using heart rate variability or rather beat-to-beat intervals based on an electrocardiogram (ECG). There are clinical situations where no ECG monitoring is available or required, but only photoplethysmography (PPG), e.g., in some cases in postoperative care or pain therapy. In addition, a combination of PPG and ECG for obtaining beat-to-beat intervals may lead to increased robustness and reliability for dealing with artifacts. This work therefore investigates the computation of ANI using standard PPG. In addition, new methods and opportunities are presented using contactless PPG imaging (PPGI). PPGI®enables contactless PPG recordings for deriving beat-to-beat intervals as well as analysis of local perfusion and wounds. Keywords: anesthesiology, analgesia, nociception, pain, heart rate variability, ECG, PPG, image based PPG, PPGI. 1. Introduction This work discusses an extension to the photoplethys- mogram (PPG) [1] based analgesia nociception index (ANI) for image based PPG (PPGI®). One of the major tasks of the anesthesiologist during surgical interventions is to maintain adequate narcosis. The dosage of drugs leading to suppression of the pa- tient’s consciousness and the sensation of pain must be adapted to the current surgical progress, and also indi- vidually for each patient with regard to his premedical history and current health state. Some events, e.g., skin incisions, require deeper analgesia. Increases of blood pressure and/or heart rate can indicate inadequate narcosis. However, it is often particularly challenging to discriminate pain from insufficient sedation. An adequate assessment of the depth of analgesia would support the anesthesiologist in balancing the narcosis properly. Unfortunately, a reliable objective measurement of pain intensity is not yet available, since pain is an individual sensation leading to large inter-patient vari- ability. Awake patients are usually asked to estimate their pain intensity on a given scale (e.g., from 1 to 10), using e.g., the visual analogue scale (VAS) [2]. However, this procedure is not feasible with unconscious or uncooperative patients. Several new methods are currently under investigation which aim to quantify the depth of analgesia. One is the Surgical Stress Index (SSI) which has been especially developed for analgesia monitoring during surgical interventions, and is based on an analysis of the PPG measured by a finger clip [3]. SSI is currently available as an extension module for GE Healthcare monitors to assess the analgesia or the stress level during surgical interventions. Another method is based on skin conductance mea- surements [4]. The number of fluctuations of the skin conductance (NFSC) is associated with the patient’s pain sensation. Analyses of skin conductance are mostly used in postoperative clinical trials under various cir- cumstances [5, 6]. A promising new method is ANI. The ANI index is based on an analysis of the ECG signal [7], especially an analysis of heart rate variability (HRV) or the beat- to-beat interval [8]. 1.1. Heart Rate Variability Heart frequency is related to the activity of the sympa- thetic and parasympathetic nervous system. Variations of frequency are affected by the autonomous nervous system (ANS) which, for example, reacts to external influences, such as stress or pain. Since the patient is unconscious during surgical interventions, it can be as- sumed that in this situation, pain is the most significant stress factor. In studies, a relatively high correlation between analgesic drugs and HRV has been confirmed [7]. The spectrum of the HRV can be divided into four parts, which are associated with different sources [8, 9]. ANI uses the HF frequency range between 0.15 Hz and 0.5 Hz, which is associated with the correlation between breathing and heart rhythm [7] or the respiratory sinus arrhythmia. 275 http://dx.doi.org/10.14311/AP.2014.54.0275 http://ojs.cvut.cz/ojs/index.php/ap M. Koeny, N. Blanik, X. Yu et al. Acta Polytechnica 1.2. Analgesia Nociception Index The following steps of ANI computation are summarized from [7]. The first step is to determine the beat-to-beat intervals from the ECG signal. Then a special filter is used to remove extra systoles and artifacts from the beat-to-beat interval series [10]. After resampling, the signal is normalized over a time period of 64 seconds and the signal is band-pass filtered between 0.15 and 0.5 Hz. Finally, the parasympathetic tone is computed using an area-under-the-envelope algorithm of the filtered beat- to-beat series curve. ANI is now computed, according to Equation 1, where AUCmin is determined as the minimum area of the envelopes of the 64 s interval divided into 16 s parts. Finally ANI is determined as described in [7]: ANI = 100 · (α · AUCmin + β) 12.8 (1) One advantage of the system is that no additional sensor is necessary. According to the manufacturer, ANI is also valid for awake patients in postoperative care [7], for example in the recovery room or in intensive care. 1.3. Use of PPG PPG is an optical volumetric measurement. It is rou- tinely used during surgical interventions to determine oxygen saturation (SpO2). In most cases, it is measured using a finger clip. Figure 4 shows an ECG and the inherent PPG signal. Of course, the ECG and PPG correlate well. Therefore, the beat-to-beat intervals should also correlate, and the authors assume the PPG signal can be used to determine the ANI index using the PPG signal. A very promising approach would be to fuse ECG and PPG in order to improve the accuracy and the robustness of ANI computation. In addition, the source can be switched if the quality of one of the signals is not sufficient or is provided with artifacts. 1.4. Image based PPG Image-based PPG technology (photoplethysmography imaging, PPGI) was first introduced in 1998 by RWTH Aachen University, Aachen, Germany [11]. A PPGI signal is recorded using visible and/or near infrared camera technology focused on the skin surface. Similar to classical PPG, PPGI detects minor changes in light intensity originating from modulated perfusion of the skin tissue. Hence, these changes — although not visible to the human eye — possess similar information to a PPG signal. The signal can be used to estimate heart rate and heart rate variability.[11]. Since, camera sensor and measurement location are now not fixed together rigidly, relative movements be- tween the observed object and the camera may become an issue in the assessment of PPGI sequences. To pre- vent such movement artifacts, either image processing algorithms for movement compensation must be applied, Anesthesia Workstation HR 83 LL Lead Off NIBP 120 80 Propofol 15 ml/h Infusion Pump Patient Monitor HR 83 NIBP 120 80 Surgical Events Anesthesia Events Anesthesia Machine Propofol 15 ml/h Infusion Pump Propofol 15 ml/h Infusion Pump Figure 1. Setup of the system in the University Hospi- tal in Aachen for recording vital signs during surgical interventions. or the measurement setup must guarantee a motion- free measurement scenario, for example by appropriate fixation of the observed body part. To extract a PPG signal from a PPGI video sequence, every single pixel of the sensitive camera chip senses like a classical PPG sensor. In addition, inside the recorded video frame, a region of interest (ROI) can be defined in which for every frame the mean gray values of all containing pixels are calculated, representing the mean regional PPG value at that particular moment. This enables totally contact-free, non-obtrusive mon- itoring of skin perfusion and further derivable vital parameters. The local distribution within the observed body surface can be estimated for all findings resulting in functional mapping of the observed parameter with spatial resolution [12]. We acknowledge that part of the work presented in this paper has been previously published in [1]. 2. Materials and Methods 2.1. Data recordings in the University Hospital in Aachen The ECG and PPG signals used to develop and evalu- ate the described algorithms were recorded over a time period of two months anonymously at the University Hospital of Aachen, Germany, after approval by the local ethics committee. The patients received either total intravenous anesthesia or balanced anesthesia with volatile anesthetics. In addition to the ECG and PPG signal, all available vital signs from the patient moni- tor, the anesthesia machine and syringe pumps were recorded. In addition, important anesthesia-associated and surgical events, such as skin incision or manually administered drugs were recorded by a dedicated ob- 276 vol. 54 no. 4/2014 Using PPG Imaging for Objective Contactless Pain Assessment Figure 2. Steps of the beat-to-beat interval calculation, modified after [14]. server. Based on these data, influences on the ANI could be examined and correlations could be analyzed, cf. Figure 1. The system used for data acquisition was developed as an “anesthesia workstation” in context of a research project [13] on modular networking in the operating room. Using the PC data connection with the MP70 patient monitor (Philips AG), a Primus or Cato anes- thesia machine (Draeger Medical, Luebeck) and syringe pumps (BBRaun, Melsungen), data were recorded dur- ing the surgical intervention and were prepared for further visualization and analysis in Matlab (The Math- works Company). Using the same system, during an animal trial, the vital signs of a young pig (“Deutsche Landrasse”) were recorded as a reference. Using an additional PC, which was connected to the PPGI cam- era (AVT, Pike 210B), the PPGI video was recorded synchronously, using a self-developed software tool with Matlab. Due to restrictions of the recording system, three minutes could be recorded continuously. The resolution of the camera was set for this recording to 640 × 480 pixels, with a sample rate of 15 frames per second. 2.2. Preprocessing the PPG Signal Because the ANI was originally developed using the ECG derived beat-to-beat intervals, the most important step was to adapt the beat-to-beat interval computation to the PPG signal. The first step is to estimate the beat- to-beat intervals from the PPG signal. This was done using a modified version of the PPG-adapted ADAPIT algorithm [14] with the following steps, according to Figure 2: (1.) A median filter of length 550 ms is applied to the original signal (Figure 2.1) to determine the DC offset and trend of the PPG signal (Figure 2.2). (2.) The median filtered signal is subtracted from the original signal in order to eliminate the DC offset and trend of the PPG signal (Figure 2.3). Figure 3. Selected example of a PPGI image. (3.) To filter out invalid peaks, two thresholds are calculated in a seven-second-long window (Figure 2.3): T1 = 2 × standard deviation, T2 = 3 × standard deviation. The first estimation of peaks results of peaks greater than T2. (4.) The third threshold is T3 = 0.8 × standard devia- tion of the resulting signal. (5.) All peaks not within the threshold are invalid. (6.) A standard peak detection algorithm is used to detect the maxima of the resulting signal (Figure 2.4). (7.) Figure 2.5 shows the detected peaks can be seen together with the original signal. Now the distance between the peaks can be deter- mined in order to estimate the ANI, as described in Section 1.2 and [7]. 2.3. Preprocessing the PPGI Signal First, the PPGI signal must be extracted from the PPGI video signal. In the context of the animal trial, this was done setting up a fixed region of interest (ROI), as shown in Figure 3. A fixed ROI can be used, because the pig is sedated and the camera is mounted in a fixed position. Therefore, movement artifacts are suspended. The video signal was filtered within the region of interest using a 2D median filter. The extracted PPGI signal has a sample rate of 15 samples per second, as this is the frame rate of the PPGI camera. For heart rate variability computation, exact determination of the HRV is essential. Therefore, 15 samples per second are not sufficient and the signal is interpolated to 125 Hz, which is equal to the sample rate of the measured PPG signal. A simple spline interpolation was used, because this matches the characteristics of the PPG signal much 277 M. Koeny, N. Blanik, X. Yu et al. Acta Polytechnica -1 0 1 2 3 4 5 6 ECG signal PPG signal time [s] Figure 4. Example of an ECG and a PPG signal with a time-delay. Figure 5. Interpolated vs. non-interpolated PPGI signal. better than a linear interpolation. Figure 5 shows the original extracted signal and the up-sampled signal. The interpolated PPGI signal is handled like the regular PPG signal for further processing steps. The PPGI signal was at least limited to three minutes, due to restrictions of the recording system used in the study. ANI computation is not reliable in this context. In addition, the pig was ventilated with up to 40 breaths per minute, while the filter of the HRV is set to 0.15– 0.5 Hz, according to ANI specification. Thus, the ANI would not be valid. Therefore, in the following results only the RR-intervals are considered and are compared with the ECG and PPG RR-intervals. 3. Results 3.1. Comparison of ECG, PPG and PPGI Signals It is clearly understandable that the heart rates of the different signals correlate. Of course, there is a short time-delay between the R-peaks of the ECG and the maximum of the PPG signal, see Figure 4. There are various reasons for the short time delay. Physiologically, the time delay is caused by the pulse transition time from the heart to the fingers. This delay is dependent on cardiovascular factors, e.g., blood pressure. The other time delay is due to technical reasons. Depending on the device, the time delay of the signal filter of each signal differs. There can also be a time shift, if the two signals are acquired by different devices. For example, the ECG is received from a Figure 6. HRV signals of ECG, PPG and PPGI. patient monitor and the PPG from a special optimized PPG device. For computation of the HRV, the time delay can be ignored, because it is only up to one second under normal conditions. This does not influence ANI computation. The PPGI signal is also delayed. In addition to the delay caused by signal processing and filtering, the delay is dependent on the same physiological factors as the regular PPG signal. Of course, it also depends on the region that is recorded or that the ROI is focused on. 3.2. Comparing Heart Rate Variability In addition to the time delay between the ECG signal and the PPG signal, the sample times of the ECG and PPG signals, are relevant for the precision of the measurement. Because the R-peak is a short sharp peak, it can be altered and the detected maximum is not precisely the real maximum. An example is shown in Figure 4. State of the art devices for high quality HRV analysis use a sample rate of up to 1000 Hz [8] to ensure high precision HRV computation. The patient monitor used in this study has a sample rate of 250 Hz for the ECG signal and 125 Hz for the PPG signal. This is standard for currently used patient monitors. The PPG signal is much smoother than the ECG signal, see Figure 3. For the lower sample frequency, the error of the peak detection is larger. Additional care must therefore be taken in peak detection and in threshold adjustment of this algorithm. In addition, the 278 vol. 54 no. 4/2014 Using PPG Imaging for Objective Contactless Pain Assessment -1 0 1 2 0 25 50 75 1. 2. 0 2 43. 43:50 44:02 44:14 44:26 44:38 44:50 0 25 50 75 time (mm:ss) 4. Figure 7. (1) ECG signal. (2) ECG-based ANI. (3) PPG signal. (4) PPG-based ANI. dicrotic notch can influence the peak detection of the ADAPIT algorithm. The non-interpolated PPGI signal has a sample rate of 15 Hz. This sample rate is not sufficient to determine the peaks of the PPGI wave accurately. Using the interpolation method described in Section 2.3, the PPGI curve is smoother and is up-sampled to a standard PPG sample rate of 125 Hz. Unfortunately, interpolation is only an approach and does not represent the real PPGI signal. The error is therefore larger than the error of the PPG signal. Figure 6 shows the RR-intervals from the synchro- nized PPG and PPGI signals. The increase is caused by inaccurate determination of the RR-intervals, caused by the interpolation and the lower sampling rate. The big spikes are generated by artifacts of the peak detection algorithm. Of course this deviation influences the ANI, as is discussed in the following Section. 3.3. Comparing Indices The differences mentioned above in ECG- and PPG- based beat-to-beat interval calculation result in a de- viation between the two ANIs. As shown in Figure 7, there is an offset between the different ANI indices, resulting in a changed distribution, see Figure 8. The distribution is computed over 20 surgical interventions, using Matlab software. Hence, the ANI is lower and if the HRV increases the PPG-ANI is lower than the ECG- ANI. However, both ANIs can be merged, resulting in a single more reliable index. For example, the PPG signal is free of artifacts during cautery, which however makes the ECG signal useless. ANI is computed over a time of about one minute, and even a short cautery can influence ANI for a minute. An analysis of 10 surgical interventions using autocorrelation shows a correlation from 0.92 to 0.98 of the beat-to-beat intervals for the time delay. The ANI correlates from 0.90 to 0.97. It can be assumed that most of the deviation results from errors in peak detection and possible misdetected peaks caused by artifacts. Unfortunately, the ANI of the ani- mal trial cannot be computed, because the respiration Figure 8. Distribution of (1) ECG-ANI and (2) PPG- ANI. The distribution is computed based on 40 hours of surgical interventions. rate was set up to 40 per minute. This exceeds the limit of the filter between 0.15–0.5 Hz and produces an invalid ANI index. 4. Discussion and Outlook PPG-based ANI aims to improve the assessment of analgesia during surgical interventions. With the PPG signal, an additional source for ANI computation is available, which can be chosen if the ECG signal is not available or not valid, for example during electro-surgical procedures. Both indices (ECG-ANI and PPG-ANI) could in future be combined to one index to improve ANI. ANI is therefore likewise usable in postoperative care with awake patients. The PPG-based version is an alternative if no ECG is derived. Additionally, it can be combined with other methods [3] for postoperative pain and stress analysis. Further work needs to be done to adapt the scaling factors (α and β) of the ANI to the offset in RR-interval variation. Furthermore, histogram transformation can be used to translate the PPG-ANI to the distribution of the ECG-ANI, compared to the results in Figure 8. The indices can be fused, for example using artifact detection algorithms or other strategies for selection 279 M. Koeny, N. Blanik, X. Yu et al. Acta Polytechnica between ECG or PPG. Additional information about current surgical procedures from other devices in a networked operating room [13] can make artifact detec- tion more reliable. For example, if the system could get information that an electro surgical procedure has started, the source of ANI computation can automat- ically be switched from ECG and PPG. This would prevent complex analysis and artifact detection. Further research needs to be done on PPGI-based RR interval computation and PPGI-ANI. First, the sample rate needs to be increased, and automatic determina- tion of the region of interest should be implemented. This should be done by focusing the PPGI camera on a smaller field of view, or by setting an ROI directly in the camera. Then the frame-rate can be increased, leading to a higher sample-rate of the resulting PPGI signal. A higher sample rate improves the RR-interval detec- tion. The improved PPGI system should be verified in a human trial. In addition, histogram transforma- tion can be used to match the PPG-based ANI to the ECG-ANI. Contactless measurement of the PPG wave- form offers new opportunities. For example, during postoperative pain therapy, the PPGI method offers a very comfortable method for assessing the ANI, as the patient does not have to be connected to an ECG or PPG device. In addition, the camera-based pro- cedure enables contactless measurements on various regions of the body. For example, the PPG signal can be measured synchronously on the face and on the hand. This enables the physician to focus on different regions and consider local physiological phenomena, e.g., local vasomotion. Another opportunity can be the use of PPGI in wound diagnosis [15]. For example, the PPGI camera, optionally in combination with an thermo infrared camera, can be focused on a wound. This can deliver information about local PPG correlated phenomena and the difference in thermography caused by inflammation. This information can be used for improved wound diagnosis, possibly leading to improved wound treatment. Acknowledgements This work forms part of the OR.NET project, which is funded by the German Federal Ministry of Education and Research (BMBF). The authors would like to thank all partners for the efficient collaborative work. The human and animal trials have been approved by the local ethics committee. References [1] M. Koeny, X. Yu, M. Czaplik, et al. Computing the Analgesia Nociception Index Based on PPG Signal Analysis. In Poster conference. 2013. [2] S. L. Collins, R. A. Moore, H. J. McQuay. The Visual Analogue Pain Intensity Scale: What is Moderate Pain in Millimetres? Pain 72(1):95–97, 1997. [3] M. Huiku, K. Uutela, M. Van Gils, et al. Assessment of Surgical Stress During General Anaesthesia. British Journal of Anaesthesia 98(4):447–455, 2007. doi:10.1093/bja/aem004. [4] T. Ledowski, J. Bromilow, J. Wu, et al. The Assessment of Postoperative Pain by Monitoring Skin Conductance: Results of a Prospective Study. Anaesthesia 62(10):989– 993, 2007. doi:10.1111/j.1365-2044.2007.05191.x. [5] T. Ledowski, J. Bromilow, M. Paech, et al. Monitoring of Skin Conductance to Assess Postoperative Pain Intensity. British Journal of Anaesthesia 97(6):862–865, 2006. doi:10.1093/bja/ael280. [6] M. Czaplik, C. Hübner, M. Köny, et al. Acute Pain Therapy in Postanesthesia Care Unit Directed by Skin Conductance: A Randomized Controlled Trial. PloS one 7(7):e41758, 2012. doi:10.1371/journal.pone.0041758. [7] R. Logier, M. Jeanne, A. Dassonneville, et al. PhysioDoloris: A Monitoring Device for Analgesia/Nociception Balance Evaluation using Heart Rate Variability Analysis. In Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE, pp. 1194–1197. IEEE, 2010. doi:10.1109/IEMBS.2010.5625971. [8] M. Malik, J. Bigger, A. Camm, et al. Heart Rate Variability. Standards of measurement, physiological interpretation, and clinical use, Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. Circulation 93(5):1043– 1065, 1996. doi:10.1161/âĂŃ01.CIR.93.5.1043. [9] S. Akselrod, D. Gordon, J. Madwed, et al. Hemodynamic Regulation: Investigation by Spectral Analysis. American Journal of Physiology-Heart and Circulatory Physiology 249(4):H867–H875, 1985. [10] R. Logier, J. De Jonckheere, A. Dassonneville. An Efficient Algorithm for RR Intervals Series Filtering. In Engineering in Medicine and Biology Society, 2004. IEMBS’04. 26th Annual International Conference of the IEEE, vol. 2, pp. 3937–3940. IEEE, 2004. doi:10.1109/IEMBS.2004.1404100. [11] V. Blazek, U. Schultz-Ehrenburg. Frontiers in computer-aided visualization of vascular functions. In Fortschrittberichte VDI, pp. 117–124. VDI Verlag Duesseldorf, 1998. [12] T. Wu, V. Blazek, H. J. Schmitt. Photoplethysmography Imaging: a new Noninvasive and Noncontact Method for Mapping of the Dermal Perfusion Changes. In EOS/SPIE European Biomedical Optics Week, pp. 62–70. International Society for Optics and Photonics, 2000. doi:10.1117/12.407646. [13] M. Koeny, J. Benzko, M. Czaplik, et al. Getting Anesthesia Online: The smartOR Network. International Journal on Advances in Internet Technology 5(3 and 4):114–125, 2012. [14] C. Yu, Z. Liu, T. McKenna, et al. A method for automatic identification of reliable heart rates calculated from ECG and PPG waveforms. Journal of the American Medical Informatics Association 13(3):309–320, 2006. doi:10.1197/jamia.M1925. [15] U. Schultz-Ehrenburg, V. Blazek. Value of Quantitative Photoplethysmography for Functional Vascular Diagnostics. Skin Pharmacol Appl Skin Physiol 14(5):316–323, 2001. 280 http://dx.doi.org/10.1093/bja/aem004 http://dx.doi.org/10.1111/j.1365-2044.2007.05191.x http://dx.doi.org/10.1093/bja/ael280 http://dx.doi.org/10.1371/journal.pone.0041758 http://dx.doi.org/10.1109/IEMBS.2010.5625971 http://dx.doi.org/10.1161/​01.CIR.93.5.1043 http://dx.doi.org/10.1109/IEMBS.2004.1404100 http://dx.doi.org/10.1117/12.407646 http://dx.doi.org/10.1197/jamia.M1925 Acta Polytechnica 54(4):275–280, 2014 1 Introduction 1.1 Heart Rate Variability 1.2 Analgesia Nociception Index 1.3 Use of PPG 1.4 Image based PPG 2 Materials and Methods 2.1 Data recordings in the University Hospital in Aachen 2.2 Preprocessing the PPG Signal 2.3 Preprocessing the PPGI Signal 3 Results 3.1 Comparison of ECG, PPG and PPGI Signals 3.2 Comparing Heart Rate Variability 3.3 Comparing Indices 4 Discussion and Outlook Acknowledgements References