Available online at www.HighTechJournal.org HighTech and Innovation Journal Vol. 3, No. 3, September, 2022 319 ISSN: 2723-9535 DeepImageTranslator V2: Analysis of Multimodal Medical Images using Semantic Segmentation Maps Generated through Deep Learning En Zhou Ye 1, En Hui Ye 1, Maxime Bouthillier 2 , Run Zhou Ye 3, 4* 1 Blyth Academy, Ottawa, Ontario, Canada. 2 Département de Radiologie, Radio-oncologie et Médecine Nucléaire, Faculté de Médecine, Université de Montréal, Montréal, Québec, Canada. 3 Department of Radiation Medicine Program, Princess Margaret Cancer Centre, University Health Network, Toronto, Canada. 4 Division of Endocrinology, Department of Medicine, Centre de Recherche du Centre Hospitalier Universitaire de Sherbrooke, Université de Sherbrooke, Sherbrooke, Québec, Canada. Received 13 March 2022; Revised 22 July 2022; Accepted 04 August 2022; Published 01 September 2022 Abstract Introduction: Analysis of multimodal medical images often requires the selection of one or many anatomical regions of interest (ROIs) for extraction of useful statistics. This task can prove laborious when a manual approach is used. We have previously developed a user-friendly software tool for image-to-image translation using deep learning. Therefore, we present herein an update to the DeepImageTranslator V2 software with the addition of a tool for multimodal medical image segmentation analysis (hereby referred to as the MMMISA). Methods: The MMMISA was implemented using the Tkinter library; backend computations were implemented using the Pydicom, Numpy, and OpenCV libraries. We tested our software using 4188 slices from whole-body axial 2-deoxy-2-[18F]-fluoroglucose-position emission tomography/ computed tomography scans ([¹⁸F]-FDG-PET/CT) of 10 patients from the American College of Radiology Imaging Network-Head and Neck Squamous Cell Carcinoma (ACRIN-HNSCC) database. Using the deep learning software DeepImageTranslator, a model was trained with 36 randomly selected CT slices and manually labelled semantic segmentation maps. Utilizing the trained model, all the CT scans of the 10 HNSCC patients were segmented with high accuracy. Segmentation maps generated using the deep convolutional network were then used to measure organ specific [¹⁸F]-FDG uptake. We also compared measurements performed using the MMMISA and those made with manually selected ROIs. Results: The MMMISA is a tool that allows user to select ROIs based on deep learning-generated segmentation maps and to compute accurate statistics for these ROIs based on coregistered multimodal images. We found that organ-specific [¹⁸F]-FDG uptake measured using multiple manually selected ROIs is concordant with whole-tissue measurements made with segmentation maps using the MMMISA tool. Keywords: Multimodal Medical Imaging; Semantic Segmentation; Regions of Interest; Deep Learning; Software; PET/CT Imaging. 1. Introduction Analysis of multimodal medical images (e.g., position emission tomography/magnetic resonance imaging [PET/MRI] and PET/computed tomography [PET/CT]) often requires the selection of one or many anatomical regions of interest (ROIs) for extraction of useful statistics [1-7]. The use of spherical or ellipsoid ROIs may be sufficient for large organs such as the liver and large muscle groups. However, for organs/tissues with complex shapes (e.g., the intestines and adipose tissues), manual ROI segmentation is not a scalable approach. One possible method is the use of * Corresponding author: run.zhou.ye@usherbrooke.ca http://dx.doi.org/10.28991/HIJ-2022-03-03-07  This is an open access article under the CC-BY license (https://creativecommons.org/licenses/by/4.0/). © Authors retain all copyrights. https://creativecommons.org/licenses/by/4.0/ https://orcid.org/0000-0002-2131-8988 https://orcid.org/0000-0002-0333-1767 HighTech and Innovation Journal Vol. 3, No. 3, September, 2022 320 deep learning for automated segmentation. Nevertheless, most deep learning pipelines for semantic image segmentation generate color-coded segmentation maps stored as image files, while most free software programs for medical image analysis (e.g., 3D-Slicer, OsiriX Lite, and AMIDE) cannot use these files to generate ROI statistics of multimodal images stored as DICOM files. We have previously developed a user-friendly software tool for image-to-image translation using deep learning (DeepImageTranslator, released at: https://sourceforge.net/projects/deepimagetranslator/) [8]. Therefore, we present herein an update to the DeepImageTranslator software with the addition of a tool for multimodal medical image segmentation analysis (hereby referred to as the MMMISA). We then demonstrate the use of the program for the measurement of 2-deoxy-2-[18F]fluoroglucose ([¹⁸F]-FDG) uptake by the lungs and subcutaneous adipose tissue using whole-body [¹⁸F]-FDG-PET/CT scans from the ACRIN-HNSCC-FDG-PET/CT database [9-11]. Furthermore, we also compare measurements performed using the MMMISA and those made with manually selected ROIs. 2. Methods 2.1. Development of the MMMISA Program The MMMISA program presented herein was written in Python 3.8 and distributed under the GNU General Public License (version 3.0). The graphical user interface was developed using the Tkinter library, which is the most commonly used Python package for graphical user interface creation. Image analysis algorithms were implemented using the Pydicom, Numpy, and OpenCV libraries. The program is included as part of version 2 of the DeepImageTranslator software (https://sourceforge.net/projects/deepimagetranslator/) and is also available as a standalone program (https://sourceforge.net/projects/mmmisa/) for Windows. The source codes are available at: (https://github.com/runzhouye/MMMISA). 2.2. PET/CT Image Dataset Whole-body CT and FDG-PET images from 10 patients (numbers 001, 002, 003, 007, 008, 010, 012, 018, 019, and 027—which were the first 10 whole-body scans with the same windowing and devoid of significant radiographic artifacts) were downloaded from the ACRIN-HNSCC-FDG-PET/CT (ACRIN 6685) database [9, 10] via the Cancer Institute Archive [11]. We arbitrarily chose the ACRIN-HNSCC-FDG-PET/CT database since it was one of the few databases that contain coregistered whole-body PET and CT scans. 2.3. Manual Extraction of Multimodal Image Data CT and FDG-PET images were loaded into the AMIDE software [12]. For each patient, 11 spherical ROIs (10 mm diameter) in the subcutaneous adipose tissue and 3 spherical ROIs (50 mm diameter) in the lungs were drawn at different axial positions based on whole-body CT images. ROI statistics were subsequently generated for the coregistered PET images. 2.4. Semantic Image Segmentation Thirty-six axial slices were randomly chosen from the 4188 axial CT images from the 10 patients for manual semantic segmentation with the GIMP (GNU Image Manipulation Program) software of the background, lungs, bones, brain, subcutaneous and visceral adipose tissue, and other soft tissues by labelling these regions in black (RGB=[0,0,0]), yellow (RGB=[255,255,0]), white (RGB=[255,255,255]), cyan (RGB=[0,255,255]), red (RGB=[255,0,0]), green (RGB=[0,255,0]), and blue (RGB=[0,0,255]). Thirty-six training samples were considered more than sufficient since we have previously shown that models can be trained to achieve high accuracy with as little as 17 images [13]. CT image- segmentation map pairs were then loaded into the DeepImageTranslator software to train a deep convolutional neural network as previously described in Ye et al. [13] with 1000 training epochs. The final model was used to perform automatic semantic segmentation of the 4188 axial CT images from the 10 patients in less than 10 minutes. The generalizability of such an approach for automated segmentation has previously been shown to be excellent [13]. 2.5. Automated Extraction of Multimodal Image Data For each patient, the original PET/CT scans were loaded into the MMMISA program along with the semantic segmentation maps produced by the convolutional neural network. In this study, we chose to extract FDG uptake from the lungs and subcutaneous adipose tissue by extracting regions of the model-generated segmentation maps containing yellow and red pixels, respectively using the MMMISA program. Lower and upper color threshold were set at (R,G,B) = (150,150,0) and (R,G,B) = (255,255,150), respectively, for the lungs, and (R,G,B) = (150,0,0) and (R,G,B) = (255,150,150), respectively, for the subcutaneous adipose tissue. ROI statistics were then generated for the FDG-PET scans using the MMMISA software. https://sourceforge.net/projects/deepimagetranslator/ https://sourceforge.net/projects/mmmisa/ https://github.com/runzhouye/MMMISA HighTech and Innovation Journal Vol. 3, No. 3, September, 2022 321 2.6. Statistical Analyses Statistical analyses were carried out using GraphPad Prism version 9. Pearson’s R values were computed for the correlation between organ-specific FDG uptakes measured using multiple manually selected ROIs and FDG uptake determined using deep learning-generated segmentation maps. 3. Results 3.1. The MMMISA Plugin for the DeepImageTranslator The MMMISA program is included in version 2 of the DeepImageTranslator and is also available as a standalone software. The main window (Figure 1) allows for the user to visualize single- and dual-modality images written in the standard DICOM (Digital Imaging and Communications in Medicine) file format, the most commonly used file format in medical imaging. When images from a second modality are loaded into the program, they are automatically matched, along with the corresponding segmentation map, to the image of the first modality that is being currently displayed. When necessary, the program also performs image registration of modality 2 images based on modality 1 images through translation and/or stretching such that objects in both image sets overlap. This allows for simultaneous visualization of both image sets and segmentation maps. Figure 1. Main window of MMMISA, showing (from left to right), modality 1 (CT) images, segmentation maps generated with convolutional neural network, and modality 2 (PET) images A second, ROI selection, window (Figure 2) displays user settings for the extraction of ROIs based on pixel colors of the segmentation maps. Specific regions of the color-coded semantic segmentation maps can be extracted by setting lower and upper thresholds for the red, green, and blue color component values using the ROI selection window. The user can also choose to only include the left or right side of the patient for analysis, which can be useful in order to exclude the strong signals from of certain radiotracers injected into the left or right arm. When the “Apply” button is pressed, ROIs are generated based on the color thresholds using the segmentation maps and applied to corresponding slices of modality 1 and 2 images. The cropped images are then displayed in the main window for visualization. Figure 2. ROI selection window and main window with updated modality 1 and 2 images HighTech and Innovation Journal Vol. 3, No. 3, September, 2022 322 When “Save analysis” is selected, data will be extracted from modality 1 and 2 images, including the name of the scan, time at which each slice was produced, position of image slices, total area of the ROIs on each slice, total pixel values in the ROIs, average and standard deviation of values of pixels inside the ROIs, and pixel size. Results are then written in an excel file and stored under the user-designated directory. 3.2. Semantic Segmentation of PET/CT Images Segmentation results for images outside of the training set obtained with the convolutional neural network trained using the DeepImageTranslator were illustrated in Figure 3. Our final model was able to accurately segment the lungs, brain, bone, subcutaneous and visceral adipose tissue, and other soft tissues. Figure 3. Pairs of CT images outside of the training set and sematic segmentation maps generated with deep convolutional neural network. The background, lungs, bones, brain, subcutaneous and visceral adipose tissue, and other soft tissues were labelled in black, yellow, white), cyan, red, green, and blue, respectively. Increase in number of manually selected ROIs increases accuracy of organ-specific FDG uptake approximations compared to true organ-specific FDG uptake measured using deep learning-generated segmentation maps. Next, we tested the concordance of organ-specific FDG uptake measured using multiple manually selected ROIs versus FDG uptake determined using deep learning-generated segmentation maps. In general, regardless of the number of ROIs used, manually measured FDG uptake in the lungs and subcutaneous adipose tissue was well correlated with that calculated with segmentation maps using the MMMISA program (Figure 4). For subcutaneous adipose tissue FDG uptake, the correlation coefficient and the -log of the P-value increased sharply once values from more than 4 ROIs were combined (Figure 4-a). The increase in measurement accuracy (determined by the correlation coefficient) through increasing numbers of manually selected ROIs plateaued after more than 8 ROIs were used. Nevertheless, the P-value of the correlation between manual measurement and that using segmentation maps continued to decrease when more ROIs were used (Figure 4-b). Similar results were obtained for the measurement of FDG uptake in the lungs (Figures 4- c and 4-d). HighTech and Innovation Journal Vol. 3, No. 3, September, 2022 323 Figure 4. Correlation coefficient (A and C) and P-value (B and D) for the association between organ-specific FDG uptake measured using multiple manually selected ROIs and FDG uptake determined using deep learning-generated segmentation maps, as a function of number of manually selected ROIs, for the subcutaneous adipose tissue (A and B) and lungs (C and D). ROI: region of interest. 4. Summary and Conclusion In recent years, numerous open-source software tools have been reported in the field of medical image processing [14-18]. One growing area of development is the popularization of deep learning methods through the creation of user- friendly tools with a graphical interface. Nevertheless, most deep learning pipelines for semantic image segmentation generate color-coded segmentation maps stored as image files, while most free software programs for medical image analysis cannot use these files to generate ROI statistics of multimodal images stored as DICOM files. Nonetheless, selection of ROIs is an important aspect of in vivo metabolic studies involving PET/CT imaging [19- 22]. In particular, measurements of volume and radiotracer uptake of adipose tissues of different regions may prove to be important for future studies on the metabolic syndrome, as hypertrophic obesity is related to changes in adipose tissue distribution and alterations in metabolic endpoints [23, 24]. Therefore, we have presented herein an update to the DeepImageTranslator software [8] by including a tool for multimodal medical image segmentation analysis based on semantic segmentation maps generated using a deep convolutional neural network. Our program can be accessed through a graphical interface and allows users to extract ROI statistics of multimodal images (e.g., PET/CT and PET/MRI) based on color-coded semantic segmentation maps. We showed that organ-specific FDG uptake measured using multiple manually selected small, spherical ROIs is concordant with whole-tissue measurements made with segmentation maps using the MMMISA program. Furthermore, we found that increase in number of manually selected ROIs increases the accuracy of organ specific FDG uptake approximations. Therefore, our pipeline constitutes a simple, automated, and scalable approach to obtain ROI statistics using multimodal scans. Although the accuracy of the neural network would never surpass the accuracy of manually labelled segmentation maps used for model training, our approach would eventually greatly simplify the task of researchers and radiologists performing whole-body semantic segmentation of multimodal tomography data. 5. Declarations 5.1. Author Contributions Software development: E.Z.Y., E.H.Y., and R.Z.Y.; Statistical analyses: E.H.Y.; Interpretation and manuscript drafting: E.Z.Y., M.B., and R.Z.Y. All authors have read and agreed to the published version of the manuscript. 5.2. Data Availability Statement  The source code for the DeepImageTranslator is publicly available at: https://github.com/runzhouye/MMMISA.  The compiled standalone software is available for Window10 at: https://sourceforge.net/projects/deepimagetranslator/ https://sourceforge.net/projects/mmmisa/  The datasets generated during and/or analyzed during the current study are available at: https://doi.org/10.6084/m9.figshare.16800925 HighTech and Innovation Journal Vol. 3, No. 3, September, 2022 324 5.3. Funding This work was funded by the Canadian Institutes of Health Research (CIHR Funding Reference Number: 202111FBD-476587-76355). 5.4. Ethical Approval All analyses were conducted on the public ACRIN 6685 dataset in accordance with the TCIA (The Cancer Imaging Archive) Data Usage Policy. 5.5. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. 6. References [1] Lensen, K. J. D. F., Van Sijl, A. M., Voskuyl, A. E., Van Der Laken, C. J., Heymans, M. W., Comans, E. F. I., Nurmohamed, M. T., Smulders, Y. M., & Boellaard, R. (2017). Variability in quantitative analysis of atherosclerotic plaque inflammation using 18F-FDG PET/CT. PLoS ONE, 12(8), 181847. doi:10.1371/journal.pone.0181847. [2] Cao, Y., Gathaiya, N., Han, Q., Kemp, B. J., & Jensen, M. D. (2019). Subcutaneous adipose tissue free fatty acid uptake measured using positron emission tomography and adipose biopsies in humans. American Journal of Physiology. Endocrinology and Metabolism, 317(2), E194–E199. doi:10.1152/ajpendo.00030.2019. [3] Noll, C., Montastier, É., Amrani, M., Kunach, M., Frisch, F., Fortin, M., Bouffard, L., Dubreuil, S., Phoenix, S., Cunnane, S. C., Guérin, B., Turcotte, E. E., Laville, M., & Carpentier, A. C. (2020). Seven-day overfeeding enhances adipose tissue dietary fatty acid storage and decreases myocardial and skeletal muscle dietary fatty acid partitioning in healthy subjects. American Journal of Physiology - Endocrinology and Metabolism, 318(2), E286–E296. doi:10.1152/ajpendo.00474.2019. [4] Hojjati, M., Badve, C., Garg, V., Tatsuoka, C., Rogers, L., Sloan, A., Faulhaber, P., Ros, P. R., & Wolansky, L. J. (2018). Role of FDG-PET/MRI, FDG-PET/CT, and Dynamic Susceptibility Contrast Perfusion MRI in Differentiating Radiation Necrosis from Tumor Recurrence in Glioblastomas. Journal of Neuroimaging, 28(1), 118–125. doi:10.1111/jon.12460. [5] Montastier, É., Zhou Ye, R., Noll, C., Bouffard, L., Fortin, M., Frisch, F., Phoenix, S., Guérin, B., Turcotte, É. E., Lewis, G. F., & Carpentier, A. C. (2021). Increased postprandial nonesterified fatty acid efflux from adipose tissue in prediabetes is offset by enhanced dietary fatty acid adipose trapping. American Journal of Physiology - Endocrinology and Metabolism, 320(6), E1093– E1106. doi:10.1152/AJPENDO.00619.2020. [6] Labbé, S. M., Grenier-Larouche, T., Noll, C., Phoenix, S., Guérin, B., Turcotte, E. E., & Carpentier, A. C. (2012). Increased myocardial uptake of dietary fatty acids linked to cardiac dysfunction in glucose-intolerant humans. Diabetes, 61(11), 2701–2710. doi:10.2337/db11-1805. [7] Ye, R. Z., Montastier, É., Noll, C., Frisch, F., Fortin, M., Bouffard, L., ... & Carpentier, A. C. (2022). Total Postprandial Hepatic Nonesterified and Dietary Fatty Acid Uptake Is Increased and Insufficiently Curbed by Adipose Tissue Fatty Acid Trapping in Prediabetes With Overweight. Diabetes, 71(9), 1891-1901. doi:10.2337/db21-1097. [8] Sourceforge Website (2022). DeepImageTranslator: a free, user-friendly tool for image translation using deep-learning and its applications in CT image analysis. Available online: https://sourceforge.net/projects/deepimagetranslator/ (accessed on February 2022). [9] Kinahan, P., Muzi, M., Bialecki, B., Herman, B., & Coombs, L. (2019). Data from the ACRIN 6668 Trial NSCLC-FDG- PET. Cancer Imaging Arch, 10. [10] Lowe, V. J., Duan, F., Subramaniam, R. M., Sicks, J. R. D., Romanoff, J., Bartel, T., Jian, Q., Nussenbaum, B., Richmon, J., Arnold, C. D., Cognetti, D., & Stack, B. C. (2019). Multicenter trial of [18F]fluorodeoxyglucose positron emission tomography/computed tomography staging of head and neck cancer and negative predictive value and surgical impact in the n0 neck: Results from acrin 6685. Journal of Clinical Oncology, 37(20), 1704–1712. doi:10.1200/JCO.18.01182. [11] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The cancer imaging archive (TCIA): Maintaining and operating a public information repository. Journal of Digital Imaging, 26(6), 1045–1057. doi:10.1007/s10278-013-9622-7. [12] Loening, A. M., & Gambhir, S. S. (2003). AMIDE: A Free Software Tool for Multimodality Medical Image Analysis. Molecular Imaging, 2(3), 131–137. doi:10.1162/153535003322556877. [13] Ye, R. Z., Noll, C., Richard, G., Lepage, M., Turcotte, É. E., & Carpentier, A. C. (2022). DeepImageTranslator: A free, user- friendly graphical interface for image translation using deep-learning and its applications in 3D CT image analysis. SLAS Technology, 27(1), 76–84. doi:10.1016/j.slast.2021.10.014. HighTech and Innovation Journal Vol. 3, No. 3, September, 2022 325 [14] Lessard, R., Tremblay, N. M., Plourde, M. É., & Guillot, M. (2020). An open-source software for monitoring intrafraction motion during external beam radiation therapy based on superimposition of contours of projected ROIs on cine-MV images. Journal of Applied Clinical Medical Physics, 21(8), 173–182. doi:10.1002/acm2.12940. [15] Yushkevich, P. A., Pashchinskiy, A., Oguz, I., Mohan, S., Schmitt, J. E., Stein, J. M., Zukić, D., Vicory, J., McCormick, M., Yushkevich, N., Schwartz, N., Gao, Y., & Gerig, G. (2019). User-Guided Segmentation of Multi-modality Medical Imaging Datasets with ITK-SNAP. Neuroinformatics, 17(1), 83–102. doi:10.1007/s12021-018-9385-x. [16] Bücking, T. M., Hill, E. R., Robertson, J. L., Maneas, E., Plumb, A. A., & Nikitichev, D. I. (2017). From medical imaging data to 3D printed anatomical models. PLoS ONE, 12(5), 178540. doi:10.1371/journal.pone.0178540. [17] Debus, C., Floca, R., Ingrisch, M., Kompan, I., Maier-Hein, K., Abdollahi, A., & Nolden, M. (2019). MITK-ModelFit: A generic open-source framework for model fits and their exploration in medical imaging - design, implementation and application on the example of DCE-MRI. BMC Bioinformatics, 20(1), 31. doi:10.1186/s12859-018-2588-1. [18] Razeghi, O., Solís-Lemus, J. A., Lee, A. W. C., Karim, R., Corrado, C., Roney, C. H., de Vecchi, A., & Niederer, S. A. (2020). CemrgApp: An interactive medical imaging application with image processing, computer vision, and machine learning toolkits for cardiovascular research. SoftwareX, 12, 100570. doi:10.1016/j.softx.2020.100570. [19] Croteau, E., Lavallée, É., Labbe, S. M., Hubert, L., Pifferi, F., Rousseau, J. A., Cunnane, S. C., Carpentier, A. C., Lecomte, R., & Bénard, F. (2010). Image-derived input function in dynamic human PET/CT: Methodology and validation with 11C-acetate and 18F- fluorothioheptadecanoic acid in muscle and 18F-fluorodeoxyglucose in brain. European Journal of Nuclear Medicine and Molecular Imaging, 37(8), 1539–1550. doi:10.1007/s00259-010-1443-z. [20] Lee, S. Y., Jee, W. H., Yoo, I. R., Jung, J. Y., Im, S. A., Chung, Y. G., & Kang, J. H. (2019). Comparison of 3T diffusion- weighted MRI and 18F-FDG PET/CT in musculoskeletal tumours: Quantitative analysis of apparent diffusion coefficients and standardized uptake values. British Journal of Radiology, 92(1102), 20181051. doi:10.1259/bjr.20181051. [21] Richard, G., Noll, C., Archambault, M., Lebel, R., Tremblay, L., Ait‐Mohand, S., Guérin, B., Blondin, D. P., Carpentier, A. C., & Lepage, M. (2020). Contribution of perfusion to the 11 C-acetate signal in brown adipose tissue assessed by DCE‐MRI and 68Ga-DOTA PET in a rat model. Magnetic Resonance in Medicine, 85(3), 1625–1642. Portico. doi:10.1002/mrm.28535. [22] Carreau, A. M., Noll, C., Blondin, D. P., Frisch, F., Nadeau, M., Pelletier, M., Phoenix, S., Cunnane, S. C., Guérin, B., Turcotte, E. E., Lebel, S., Biertho, L., Tchernof, A., & Carpentier, A. C. (2020). Bariatric surgery rapidly decreases cardiac dietary fatty acid partitioning and hepatic insulin resistance through increased intra-abdominal adipose tissue storage and reduced spillover in type 2 diabetes. Diabetes, 69(4), 567–577. doi:10.2337/db19-0773. [23] Ye, R. Z., Richard, G., Gévry, N., Tchernof, A., & Carpentier, A. C. (2022). Fat Cell Size: Measurement Methods, Pathophysiological Origins, and Relationships with Metabolic Dysregulations. Endocrine Reviews, 43(1), 35–60. doi:10.1210/endrev/bnab018. [24] Carpentier, A.C. (2021). 100thanniversary of the discovery of insulin perspective: Insulin and adipose tissue fatty acid metabolism. American Journal of Physiology - Endocrinology and Metabolism (AJPENDO), 320(4), 653–670. doi:10.1152/AJPENDO.00620.2020.