Diabetes and Retinopathy [1 ed.] 0128174382, 9780128174388

Table of contents :
Cover
Diabetes and Retinopathy
Copyright
Contributors
1
Complementary capabilities of photoacoustic imaging to existing optical ocular imaging techniques
References
2
Intraretinal fluid map generation in optical coherence tomography images
Introduction
Optical coherence tomography: Background and significance
The classical segmentation approach
Fluid identification by means of a regional analysis
ROI extraction
Image sampling
Classification
Binary map creation
Color map creation
Discussion and conclusions
Acknowledgments
References
3
Fully automated identification and clinical classification of macular edema using optical coherence tomography ...
Background and significance
Computational identification and characterization of the MEs
Region of interest delimitation
Identification of the different types of macular edema
Results and discussion
Conclusions
Acknowledgments
References
4
Optimal surface segmentation with subvoxel accuracy in spectral domain optical coherence tomography images
Introduction
Methods
Problem formulation and energy function
Original formulation in regularly sampled space
Formulation in irregularly sampled space to achieve subvoxel accuracy
Graph construction
Intracolumn edges
Intercolumn edges
Intersurface edges
Surface recovery from minimum s-t cut
Experimental methods
Data
Workflow
Experiment for subvoxel accuracy
Experiment for super-resolution accuracy
Cost function design
Gradient vector flow
Parameter setting
Results
Results for subvoxel accuracy
Results for super-resolution accuracy
Discussion and conclusions
References
5
Analysis of optical coherence tomography images using deep convolutional neural network for maculopathy grading
Introduction
Macular edema
Age-related macular degeneration
Central serous chorioretinopathy
Retinal imaging modalities
Optical coherence tomography
Dataset description
TU-Net: A deep CNN architecture for maculopathy grading
Preprocessing
Proposed TU-Net architecture
Results and discussion
Conclusion
References
6
Segmentation of retinal layers from OCT scans
Introduction
Method
Joint MGRF-based macula: Centred image segmentation
Shape model Psp(m)
Appearance model
3D retinal layers segmentation
Experimental results
Conclusion
References
7
Low-complexity computer-aided diagnosis for diabetic retinopathy
Introduction
Related work
Low-complexity CNN for diabetic retinopathy
Mathematical model and architecture
Evaluation metrics
Model hyperparameters
Experimental results
Discussion
Conclusion
References
8
Ophthalmic optical coherence tomography angiography in diabetes
Introduction
Qualitative and quantitative changes in ophthalmic blood vessels in diabetes detected by OCTA
OCTA in patients with diabetes without clinically apparent DR
OCTA in patients with nonproliferative DR
OCTA in patients with proliferative DR
OCTA in patients with diabetic macular edema
Effect of various treatments of DR and DME to OCTA parameters
OCTA at peripapillary area and optic nerve head in diabetes
Relevance of findings of OCTA studies concerning pathophysiology of DR
Conclusion
References
9
Early detection of diabetics using retinal OCT images
Introduction
Traditional methods for early detection and assessment of diabetes
Optical coherence tomography
Early detection of diabetes using retinal OCT images
Related work on early detection of diabetes using OCT images
Proposed work
Automatic segmentation of retinal OCT images
Feature extraction
Classification of diabetes using an RF classifier
Experimental results
Discussion and conclusions
Acknowledgment
References
10
A noninvasive approach for the early detection of diabetic retinopathy
Introduction
Methods
Results
Validation
Conclusion
References
Index

Citation preview

Diabetes and Retinopathy

Diabetes and Retinopathy Edited by Ayman S. El-Baz

University of Louisville, Louisville, KY, United States

Jasjit S. Suri

AtheroPoint, Roseville, CA, United States

Elsevier Radarweg 29, PO Box 211, 1000 AE Amsterdam, Netherlands The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom 50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States © 2020 Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions. This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein). Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility. To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein. Library of Congress Cataloging-in-Publication Data A catalog record for this book is available from the Library of Congress British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-12-817438-8 For information on all Elsevier publications visit our website at https://www.elsevier.com/books-and-journals

Publisher: Stacy Masucci Acquisitions Editor: Tari K. Broderick Editorial Project Manager: Samantha Allard Production Project Manager: Maria Bernard Cover Designer: Matthew Limbert Typeset by SPi Global, India

Contributors Michael D. Abra`moff Department of Electrical and Computer Engineering; Department of Biomedical Engineering; Department of Ophthalmology and Visual Sciences, Carver College of Medicine, University of Iowa, Iowa City, IA, United States Gary Abrams Department of Ophthalmology, Visual and Anatomical Sciences, Wayne State University School of Medicine, Detroit, MI, United States Muhammad Usman Akram Department of Computer & Software Engineering, National University of Sciences and Technology, Islamabad, Pakistan Yasmina Al Khalil Department of Electrical and Computer Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates Marah Alhalabi Department of Electrical and Computer Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates Imran Basit Department of Ophthalmology, Armed Forces Institute of Ophthalmology, Rawalpindi, Pakistan Etsuo Chihara Sensho-kai Eye Institute, Uji, Kyoto, Japan Galina Dimitrova Department of Ophthalmology, City General Hospital “8th of September”, Skopje, North Macedonia Ayman El-Baz Bioengineering Department, University of Louisville, Louisville, KY, United States Adel Elmaghraby Computer Science and Computer Engineering Department, University of Louisville, Louisville, KY, United States Marı´a Isabel Ferna´ndez Ophthalmological Institute Go´mez-Ulla and Department of Ophthalmology, University Hospital of Santiago de Compostela, Santiago de Compostela, Spain Luay Fraiwan Department of Electrical and Computer Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates Winston Furtado Bioengineering Department, University of Louisville, Louisville, KY, United States

ix

x

Contributors

Mohammed Ghazal Department of Electrical and Computer Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates; Bioengineering Department, University of Louisville, Louisville, KY, United States Guruprasad Giridharan Bioengineering Department, University of Louisville, Louisville, KY, United States Francisco Go´mez-Ulla Ophthalmological Institute Go´mez-Ulla and Department of Ophthalmology, University Hospital of Santiago de Compostela, Santiago de Compostela, Spain Anju Goyal Department of Ophthalmology, Visual and Anatomical Sciences, Wayne State University School of Medicine, Detroit, MI, United States Taimur Hassan Department of Computer & Software Engineering, National University of Sciences and Technology, Islamabad, Pakistan; Center for Cyber-Physical Systems, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates Ashraf Khalil Computer Science Department, College of Engineering, Abu Dhabi University, Abu Dhabi, United Arab Emirates Ashraf Khallaf Bioengineering Department, University of Louisville, Louisville, KY, United States Dipen Kumar Wayne State University School of Medicine, Detroit, MI, United States Ali H. Mahmoud Bioengineering Department, University of Louisville, Louisville, KY, United States Rayyan Manwar Department of Biomedical Engineering, Wayne State University, Detroit, MI, United States Joaquim de Moura Department of Computer Science; CITIC-Research Center of ˜ a, Spain Information and Communication Technologies, University of A Corun˜a, A Corun Jorge Novo Department of Computer Science; CITIC-Research Center of Information and ˜ a, Spain Communication Technologies, University of A Corun˜a, A Corun Marcos Ortega Department of Computer Science; CITIC-Research Center of Information ˜ a, Spain and Communication Technologies, University of A Corun˜a, A Corun Manuel G. Penedo Department of Computer Science; CITIC-Research Center of ˜ a, Spain Information and Communication Technologies, University of A Corun˜a, A Corun Gabriela Samagaio CITIC-Research Center of Information and Communication ˜ a, A Corun ˜ a, Spain Technologies; Department of Computer Science, University of A Corun Harpal Sandhu Department of Ophthalmology and Visual Sciences; Department of Ophthalmology, School of Medicine, University of Louisville, Louisville, KY, United States

Contributors

xi

Shlomit Schaal Ophthalmology and Visual Sciences Department, University of Massachusetts Medical School, Worcester, MA, United States Mohamed Shaban Electrical and Computer Engineering, University of South Alabama, Mobile, AL, United States Abhay Shah Department of Electrical and Computer Engineering, University of Iowa, Iowa City, IA, United States Ahmed Shalaby Bioengineering Department, University of Louisville, Louisville, KY, United States Ahmed A. Sleman Bioengineering Department, University of Louisville, Louisville, KY, United States Ahmed Soliman Bioengineering Department, University of Louisville, Louisville, KY, United States Jasjit S. Suri AtheroPoint LLC; Global Biomedical Technologies, Inc., Roseville, CA; Department of Electrical Engineering, Idaho State University, Pocatello, ID, United States Fatma Taher College of Technological Innovation, Zayed University, Dubai, United Arab Emirates Alan Truhan Wayne State University Physician Group, Kresge Eye Institute, Detroit, MI, United States Pla´cido L. Vidal Department of Computer Science; CITIC-Research Center of Information ˜ a, A Corun ˜ a, Spain and Communication Technologies, University of A Corun Xiaodong Wu Department of Electrical and Computer Engineering; Department of Radiation Oncology, University of Iowa, Iowa City, IA, United States

1

Complementary capabilities of photoacoustic imaging to existing optical ocular imaging techniques Dipen Kumara, Anju Goyalb, Alan Truhanc, Gary Abramsb, Rayyan Manward W AY N E ST A TE U NI VE RS I TY S CH O OL O F M E DI CI NE , DE TRO I T, MI, UN I TE D S TA T ES DEPARTMENT OF OPHTHALMOLOGY, V ISUAL AND ANATOMICAL SCIENC ES, WAYNE STATE UNIVERSITY SCHOOL OF MEDICINE, DETROIT, MI, UNITED STATES c WAYNE STATE UNIVERSITY P HY S I CIA N G RO UP , K RE S G E E YE IN S T IT UT E, D ET ROI T, M I, U N IT ED ST AT ES d DEPARTMENT OF BIOMEDICAL ENGI NEERING, WAYNE STATE UNIV ERSI TY , DE TRO I T, MI, UNI TE D STAT ES a

b

Since 1886, when the first picture of the human retina was taken, ocular imaging has played a crucial role in the diagnosis and management of ophthalmic diseases [1]. One of the biggest contributors to the advancement of ocular imaging is the adoption of optical imaging techniques. Optical imaging is a method of looking into the body in a noninvasive way, like X-rays. However, unlike radiological imaging techniques that use ionizing radiation, optical imaging uses light and the properties of photons to produce detailed images ranging from structures as small as cells and molecules to structures as large as tissues and organs. There are plenty of advantages of using optical imaging compared to radiological imaging. For one, optical imaging is much safer for patients since it uses nonionizing radiation to excite electrons without causing damage. Additionally, since it is fast and safe, optical imaging can be used to monitor acute and chronic diseases, as well as treatment outcomes. Optical imaging is also useful for imaging soft tissue since different types of tissues absorb and scatter light differently. Finally, optical imaging can advantageously use varying colors of light to see and measure multiple properties of tissues at a time. Therefore, it is no surprise that the optical imaging modalities of fundus photography in the 1920s [2], scanning laser ophthalmoscope (SLO) imaging in 1981 [3] and optical coherence tomography (OCT) in 1991 [4] have touted a “golden age” in ophthalmic imaging applications [5]. Although these technologies have advanced the field of ocular imaging and are commonly used in clinical practice, they are not without their flaws. A new technology, photoacoustic imaging, has been shown to have promising features that could make it the next major imaging technique in ophthalmology. Additionally, photoacoustic imaging (PAI) can combine with preexisting optical microscopic imaging modalities to achieve multimodal imaging of the eye. In this chapter, we present a brief Diabetes and Retinopathy. https://doi.org/10.1016/B978-0-12-817438-8.00001-8 © 2020 Elsevier Inc. All rights reserved.

1

2

Diabetes and Retinopathy

overview of fundus photography, SLO, and OCTwhile discussing the potential of PAI as the next major ocular imaging modality. First introduced in 1920 and extensively used since 1960, fundus photography continues to be a staple technique in ophthalmology [2]. Initially 35 mm film was the standard for fundus photography but it has long been replaced by digital acquisition [5]. Fundus photography works in a similar manner as an indirect ophthalmoscope. Light is focused by a series of lenses on a ring-shaped aperture, which then is passed into a central aperture to form a ring which then passes through the camera objective lens and cornea to illuminate the retina. The reflected light from the retina then passes through a dark hole in the annulus formed by the illumination system previously described. There is minimal reflection of the light source in the captured image because the light rays of the two systems are independent. A picture can then be taken by using one mirror to interrupt the light from the illumination system so that the light from a flash bulb can pass into the eye. Another mirror drops at the same time in front of the observation telescope to direct the reflected light onto film or a digital charge-coupled device (CCD). Monochromatic light can also be used rather than white light since monochromatic light allows for increased contrast of anatomical details of the fundus [6]. Normally fundus photography can only capture a small field of view (FOV) while the pupil is dilated, but it can be increased with a small aperture stop at the cost of resolution [2]. The maximum field of view is 50 degrees but it can be increased to 60 degrees if using a mydriatic camera [2]. Additionally, by using a special Montage software, individual images can be put together to form a collage that can cover up to 110 degrees [2]. Furthermore, fundus photography can be combined with wide angle imaging to achieve a field of view between 45 and 140 degrees, but there is proportionally less retinal magnification [5]. The main advantages of fundus photography are ease of use, full color, low cost compared to other imaging techniques, and high patient compliance [2]. Currently, fundus photography is used to monitor the progression of diseases like diabetic retinopathy, age-related macular degeneration (ARMD), glaucoma, and neoplasms of the eye [5]. In SLO, the retina is scanned in a rectangular pattern of parallel scanning lines followed by the electron beam on a TV or computer screen (raster pattern) [2] using a monochromatic, narrow laser beam. The beam is usually deflected using one slow vertical and one fast horizontal galvanometer scanner [7]. By modulating the scanning beam, projection of graphics in the raster is achieved. Since it uses a raster pattern, early SLOs could output to a TV monitor and be recorded on videotapes. The SLO has been further improved by combining it with other technologies. Confocal scanning laser ophthalmoscope (cSLO) combines the principles of confocal imaging to increase contrast and depth resolution. Confocal microscopy was invented in 1955 by Marvin Minsky [8]. Confocal microscopy uses a pinhole (confocal filter), which is in an optically conjugate plane in front of a detector and point illumination to remove out-of-focus signal [2]. Much of the light that is reflected is blocked by the pinhole since light is only reflected by structures closer to the focal plane. Two-dimensional (2D) imaging occurs in a raster pattern over the specimen but three-dimensional (3D) imaging is possible by changing the axial resolution.

Chapter 1 • Complementary capabilities of photoacoustic imaging

3

By increasing the numerical aperture or decreasing the diameter of the pinhole one can increase the depth. One can then scan many thin sections through a sample which can be combined with SLO to allow cSLO to acquire depth information [9]. An improvement to cSLO is multispectral SLOs that use multiple lasers of different wavelengths. These lasers tend to be coaxial via a couple of dichroic combining mirrors and the goal is to introduce color to match images from fundus photography. The lasers are either multiplexed or fired simultaneously on an X-Y scanning mirror that causes the light to focus on a square area of several millimeters on the retina. The reflected light then traverses to a beam splitter that directs a portion of the light to the detector [2]. Multispectral SLOs are used for retinal vessel oximetry, reflectometry, angioscotometry, and fundus perimetry [10–14]. Overall, cSLO is advantageous compared to previous imaging techniques since it allows for better images, patient comfort, video capability, and the ability to image pupils that do not dilate well. It has been shown to be effective in detecting biomarkers for diabetic retinopathy [15], age-related macular degeneration [16], scanning the nerve head in glaucoma [17], and imaging the retinal nerve fiber layer (RNFL) [18]. The most common use of the SLO is with ultrawide-field imaging of 200 degrees using the Optos System. This uses an SLO with an ellipsoidal lens to visualize the peripheral retina. About 82% of the retina can be imaged. Advantages include low light level for patient comfort and good images can often be obtained without dilation of the pupil. Fundus autofluorescence, fluorescein angiography, and indocyanine green angiography can be done with the Optos system. A more recent adaptation to SLO is adaptive optics SLO (AOSLO). Adaptive optics was a technology originally created for astronomy that has been combined with SLO to reduce the effects of wavefront distortions caused by optical aberrations. This is done by measuring the wavefront distortions and compensating for them by using devices such as a deformable mirror [19]. These distortions diminish the quality of the image being reflected by the eye which prevented microscopic resolution of structures such as capillaries and cells [3]. AOSLO most commonly uses a Shack-Hartmann sensor to measure these distortions by calculating the local phase errors in the wavefront. A phase modulator, such as a deformable mirror, can be used to correct these errors since the phase errors can be used to reconstruct the wavefront which in turn can control the deformable mirror. Another aspect to have a high magnification of small structures is image stabilization. Recently, eye tracking and stimulus delivery method have been implemented in AOSLO to achieve it [20]. OCT is a noninvasive, micron level, high-resolution imaging technique based on the principle of Michelson interferometer that provides real-time images of the retina. As is with Michelson interferometer, an interference pattern is produced by splitting light into two arms: a sample arm from scanning the retina and a reference arm from a mirror. These arms are then recombined by semitransparent mirrors and redirected to a photodetector or camera [21]. If the interference is constructive between the two arms, the signal is strong at the detector and if they are destructive, the signal is weak at the detector. A reflectivity profile, also called an A-scan, can be gathered by scanning the mirror in the reference arm which contains information on spatial dimensions and location of

4

Diabetes and Retinopathy

the structures in the retina. A cross-sectional tomograph, otherwise known as a B-scan, can be obtained by combining a series of A-scans. OCT uses low-coherence interferometry as opposed to conventional interferometry that uses long coherence length [22]. Lowcoherence interferometry uses low-coherence light which is light that consists of a broad range of frequencies rather than just a single frequency. The broadband light allows for low-coherence interferometry to shorten the interference to micrometers, perfect for its usage in ophthalmology. Additionally, it should be noted that OCT usually utilizes near-infrared (NIR) light since the relatively long wavelength allows for NIR to penetrate deeper than cSLO into scattering media like the retina. Since its inception in 1991, OCT has made huge advancements and improvements to increase the rate of imaging and resolution of OCT. Time domain OCTs (TD OCTs) have largely been replaced by spectral domain or Fourier-domain OCT (SD-OCT) since current state-of-the-art ones can produce between 40 and 70,000 A-scans per minute, which is much faster than TD OCTs [5]. The major advantages of it being faster are that the scan takes less time and it is less impacted by artifacts and aberrations caused by blinking or eye movement [5]. Like SLO, OCT has been combined with adaptive optics (AO-OCT) to decrease the aberrations caused by imperfections in the curvature of the cornea and lens [23]. Also, AO-OCT has the advantage of higher axial resolution compared to AO-SLO [23]. OCT used to be limited by the fact that it could not be used for blood flow analysis due to a poor delineation of blood vessels from the scattering of light as erythrocytes move through them [24]. However, three types of OCT have shown promise in this regard: Doppler OCT, OCT angiography (OCTA), and visible light OCT (vis-OCT). Doppler OCT combines OCT with the principles of the Doppler effect which results in improved resolution and sensitivity that allows for the evaluation of blood flow, the volume of retinal and choroidal vasculature, abnormalities in choroidal vasculature [25], and abnormalities in retinal and choroidal vessels [26]. OCTA came about due to the improvements in OCT sensitivity and speed over the years which has led to better delineation of blood vessels [27]. OCTA compares consecutive B-scans taken at rates of several hundred Hz. The advantages of OCTA are that it does not require the use of fluorescein dyes such as sodium fluorescein and indocyanine green [28], the ability for repeated scans, and the ability to analyze flow in a specific axial location of the retina or choroid [29]. Vis-OCT, which uses visible light rather than NIR, has also recently gained attention due to better axial resolution than NIR-based OCTs and better image contrast due to scattering properties of tissues in visible light, albeit at the cost of image depth [30]. On top of visualizing 3D retinal structure, vis-OCT can quantify blood oxygen saturation (sO2) in retinal circulation [25]. Due to its ability to show cross sections of tissue layers at micrometer resolution, OCT is heavily used in ophthalmology as a method to assess structural changes in the retina in diseases such as diabetic retinopathy, vein occlusion, age-related macular degeneration, glaucoma, multiple sclerosis, and other diseases that have ocular sequelae. OCT is very sensitive in detecting macular edema and is more accurate than clinical examination. OCT has significantly reduced false positive referrals for diabetic macular edema (DME) during diabetic screenings [31]. Additionally, OCT has given insight into abnormalities at the juncture between vitreous and the macula

Chapter 1 • Complementary capabilities of photoacoustic imaging

5

in patients with DME which could influence management and prognosis [32]. Furthermore, OCT is also useful in the early detection of uveitic macular edema [33] with the identification of specific OCT patterns associated with the disease [34]. Another disease that OCT is used for is ARMD. Fluorescein angiography has been largely replaced by OCT as the imaging method for monitoring ARMD treatment and the need for further anti-VEGF treatment [35]. OCT is also heavily used in cases of glaucoma. Glaucoma progression is associated with RNFL and ganglion cell thinning [36], so OCT can be used for glaucoma detection and progression [37]. While most of OCT technology is focused on imaging of the retina or pathologies related to the retina, enhanced-depth imaging OCT (EDIOCT) can evaluate choroidal thickness and posterior segment inflammatory disorders [5]. Aside from monitoring the choroid, it has been shown to be useful in monitoring other ocular inflammatory diseases such as Vogt Koyanagi Harada disease [38], sarcoidosis [39], birdshot chorioretinopathy [40], and infectious choroiditis [41]. However, OCT has been well established in ophthalmology; it has also been used in other medical disciplines such as dermatology [42–55]. While fundus photography, SLO, and OCT are still consistently used today in ophthalmology, they are not without their problems and limitations. To start, fundus photography requires pupil dilation with short-acting mydriatic drops which can cause discomfort for patients [5]. There have been recent advancements in cameras that do not require mydriatic drops but these can be affected by media opacity, such as cataracts, so mydriatic cameras are still the cameras of choice. Mydriatic cameras are especially desired if there is a need to image the periphery of the retina [56]. Even more so than discomfort to patients, these technologies suffer from a lack of quantitative data, lack of ability to take photographs of high quality, poor depth resolution, difficulty in comparing serial photographs, and the need to subject patients to high-intensity light to illuminate the retina [2]. As for SLO, one of the limitations is that involuntary eye movements affect image quality. A solution to this is tracking SLO (TSLO) which uses a high-speed retinal tracker to significantly improve image quality [56]. Another limitation of SLO is that current commercial SLOs, such as Optos or the Heidelberg wide lens, do not provide images of the eye from ora to ora [57]. Additionally, there is a distortion of the image on the periphery of the image since it is taking a 2D image of a 3D globe [58]. Also, the measurements of the eye, such as distance and area, may not be the actual dimensions of the eye since it does not standardize the image to any axis of the eye [5]. Artifacts on the image can also be caused by several things: eyelashes, cataracts, intraocular lens implants, pigments in the anterior segment of the eye, and vitreous opacities to name a few [59]. Furthermore, the cost of equipment and maintenance of SLO can be a large barrier [5]. Finally, there are the limitations of OCT. OCT by itself is unable to measure sO2 and RPE melanin. While OCTA exists, it is restricted by its limited field of view, lack of information on fill or flow speed, and motion artifacts [60]. Vis-OCT suffers from limited image depth and can cause discomfort for eye imaging [23]. Finally, since all three techniques are optical scattering-based modalities, measurements of blood oxygen saturation in the eye are affected by light scattering, and fundus photography and SLO also need to use contrast agents to measure them [61].

6

Diabetes and Retinopathy

When light is received by the eye, it is processed by both the retinal pigment epithelium (RPE) and the retina which consumes a large amount of oxygen and energy [62]. Therefore, the retina needs supporting vasculature which it has from retinal and choroidal circulation. Normally these vasculature systems bring oxygen and nutrients to the retina [63], and studies have shown that variations in the sO2 and RPE melanin play a role in ocular diseases such as diabetic retinopathy [64], glaucoma [65], retinal venous occlusion [66], and ARMD [67]. Thus, there has been an increased effort in the past decade to quantify the sO2 and RPE melanin concentration in the eye. Fortunately, both blood and melanin, within the visible light spectral range, have high optical absorption coefficients which allow them to be measured [68]. PAI has been shown to measure optical absorption properties of both blood and melanin in a noninvasive and precise way in other locations of the body [69, 70]. Therefore, PAI is a recent technology for ophthalmology due to its potential clinical use in measuring retinal and choroidal sO2 and the RPE melanin. Photoacoustic imaging has been well studied in several preclinical imaging applications [71–80]. It is based on the photoacoustic effect, which is the generation of ultrasound waves due to the absorption of light and thermal expansion [81]. The primary PAI technique is photoacoustic tomography (PAT). PAT starts by using a laser to illuminate and excite the sample where short (nanosecond) laser pulses are used that satisfy the stress and thermal confinements. The sample then exhibits photoacoustic effect as it absorbs energy from the laser which results in heat emission, transient thermoelastic expansion, and leads to generation of ultrasound wave [69, 70]. The generated acoustic wave is detected by ultrasound transducers and recorded as a function of time which then is converted based on the sound speed in the sample into a one-dimensional depth-resolved image, also called an A-line. By aligning the A-lines based on their spatial location, a transverse linear scan of the point laser illumination on the sample can make a 2D image. From there a 2D raster scan of the point of illumination creates a 3D image. PAT can be categorized as photoacoustic computed tomography (PACT) or photoacoustic microscopy (PAM). PACT uses an array of ultrasonic transducers (multiple single element, linear, phased, ring, circular, or spherical arrays) to detect PA waves emitted from an object at multiple view angles [82] while PAM uses the raster scanning method [83]. Even though a higher penetration depth can be achieved using PACT, it comes at the expense of coerce resolution, system, and computational costs [84]. On the other hand, higher resolution PAM systems can be classified based on their spatial resolution or the type of scanning they use with limited penetration depth. For spatial resolution, PAM systems can be either acoustic resolution where the imaging resolution is based on the focus of the ultrasonic detector [85] or optical resolution where the resolution is determined by the optical focal spot [86]. As for the scanning classifications, there is mechanical scanning which simultaneously translates the optical illumination and ultrasound detection for volumetric imaging [87] and optical scanning where there is a set of galvanometers which maintain the ultrasound detection stationary while they scan a focused optical illumination [87]. Currently, PA is capable of imaging structures in both the anterior and posterior segments of the eye. Originally, it had been used to examine ocular structures such as the iris or retinal vasculature qualitatively

Chapter 1 • Complementary capabilities of photoacoustic imaging

7

[88], but current PAI focuses on the quantification of properties like sO2 [87] or retinal oxygen metabolic rate (rMRO2) [89] in the eye. The major structure that PAI currently focuses on in the anterior segment of the eye is the iris, specifically the red blood cells in the microvasculature and melanin of the iris [62]. While both mechanical-scanning acoustic-resolution PAM (AR-PAM) and optical-resolution PAM (OR-PAM) have been used to image the iris, only mechanical-scanning OR-PAM has been able to obtain high-resolution images of iris microvasculature [90]. The system works by focusing laser illumination light onto the iris microvasculature using a microscope objective lens [91]. A water tank is placed over the subject’s eye so that a focused ultrasonic detector can receive the ultrasonic signals emitted from the iris [91]. Additionally, sO2 of the iris microvasculature can be measured by using two excitation wavelengths that have different oxy-hemoglobin and deoxy-hemoglobin absorption coefficients [91]. Iris melanin has also been measured by PAI using mechanical-scanning OR-PAM [91]. However, unlike the sO2 of the iris microvasculature only qualitative measuring of iris melanin has been performed [84]. Instead of the iris, the focus of PAI in the posterior segment is the red blood cells in the retinal and choroidal microvasculature along with melanin in the RPE [62]. Both mechanical-scanning OR-PAM and AR-PAM have been used to image the posterior segment of the eye [92], but the resolution is too low to visualize the microvasculature in AR-PAM [92] and in OR-PAM the lens attenuates the ultrasonic signals resulting in reduced signal-to-noise ratio (SNR) of the images [93]. To overcome this, optical-scanning PA microscopy (OS-PAM) was developed [87]. Unlike mechanicalscanning OR-PAM, OS-PAM uses a pulsed laser coupled to a 1  2-single-mode optical fiber [87]. One of the outputs allowed for the compensation of laser intensity variation, while the other was directed to the cornea using a pair of galvanometer mirrors and a pair of telescope lenses [87]. Additionally, OS-PAM uses an ultrasonic needle transducer to detect PA waves, thus eliminating the need for a water tank [62]. Moreover, the needle prevents major signal attenuation resulting in high SNR images [62]. Lastly, although it has not been used in PAI of the eye, contrast agents improve PA image quality [94] and extend the scope of PAI to the genetic and molecular level [95]. Some of the contrast agents used like Evans blue [96], indocyanine green [97], and nanoparticles [98] are already common ophthalmic contrast agents thus inviting the possibility of using them with ocular PAM. Unfortunately, while PAI shows a lot of promise as an upcoming ocular imaging modality, it is relatively new and has many limitations that need to be addressed before the clinical translation. First, photoacoustic signal detection requires physical contact with the eye. Whether it is a water tank or a needle transducer with ultrasonic gel, both cause patient discomfort and are not suitable for clinical settings [62]. Additionally, physical motion for saccades or head movement can disrupt PAI. While there have been strides taken to fix this problem, there are many concerns about the performance stability and detection sensitivity with these noncontact PA methods [62]. Second, OS-PAM still requires extended imaging depth for both the retina and the choroid, high resolution for RPE melanin, and fast imaging speeds to reduce motion artifacts. For depth, optical

8

Diabetes and Retinopathy

clearing agents could be used [99] but they are not usable for in vivo imaging and NIR light could be used but the high-power excitation is a safety concern [100]. For improving the resolution of PA, one could potentially increase the lateral resolution by using the synthetic aperture technique [101]. As for axial resolution, a broad ultrasonic bandwidth does increase the axial resolution; however, higher sensitivity in OS-PAM is achieved with a narrower bandwidth [102]. A balance needs to be determined to maximize both axial resolution and detection sensitivity. Finally, higher imaging speed could reduce motion artifacts and while increasing the laser repetition rate can increase imaging speed, it is limited by the ultrasound propagation time from the posterior eye. Lastly, before PAI can be clinically adapted, it requires numerous animal studies to confirm the longitudinal performance stability of PA measurements in the eye. Furthermore, there is limited knowledge about PAI for the early detection of ocular disease [62]. Finally, studies have shown that visual stimulation of the retina can result in changes to retinal vessel diameter, blood flow, and sO2 [103]. Therefore, further studies are needed to shed light on the effect of visual light illumination on OS-PAM accuracy. The biggest advantage of OS-PAM is that multimodal imaging is achievable by combining OS-PAM with other imaging modalities. The development of multimodal microscopic imaging techniques has become increasingly important in the biomedical community as it provides comprehensive physiological information about biological tissues [104]. In the case of ocular imaging, most optical image modalities work by detecting the scattering of light reflected from the eye or fluorescent light stimulated in the sample. The problem is that these modalities require the back-traveling of photons from the sample, so they cannot measure the optical absorption. Therefore, OS-PAM complements these modalities well because it is currently the only optical absorption-based imaging modality [84]. Thus, by combining the two, one can get anatomical information, like cellular layer organization of the retina, from preexisting ocular imaging techniques and molecular information, like sO2, from OS-PAM which gives a quantitative, holistic image of the eye. OS-PAM can be combined with autofluorescence imaging [105], fluorescein angiography [65], SLO [84], and most importantly OCT. OCT adds to OS-PAM by allowing for detailed, high-resolution, retinal and choroidal structural information [105]. Additionally, by using repeated OCT scanning, complete retinal vasculature mapping is possible [106]. Furthermore, OCT can quantitatively measure retinal blood flow rate and velocity by detecting the Doppler phase shifts produced by moving blood [107]. Finally, OCT can be used to guide OS-PAM so that an area of interest on posterior segment can be imaged [108]. Ocular imaging has come a long way since the first image of the retina in 1886. The addition of optical imaging modalities to ophthalmology has introduced faster and more precise methods for physicians to monitor and diagnose ocular pathologies. While fundus photography, SLO, and OCT have advanced ocular imaging to a large degree, they have clear limitations in being optical scattering-based imaging modalities as presented in Table 1. Therefore, the introduction of photoacoustic imaging to ophthalmology could lead to the development of a novel, stand-alone modality and/or a complimentary modality to OCT and SLO that could advance the field of ocular imaging.

Chapter 1 • Complementary capabilities of photoacoustic imaging

9

Table 1 List of ophthalmological imaging modalities and their applications, advantages, and limitations. Technology

Applications

Advantages

Limitations

Fundus photography

Retinal fundus imaging, diabetes, ARMD, glaucoma, neoplasms of the eye

SLO

Retinal vessel oximetry, reflectometry, angioscotometry, fundus perimetry diabetic retinopathy, age-related macular degeneration, scanning the nerve head in glaucoma, and imaging the retinal nerve fiber layer Macular edema, macular degeneration, glaucoma, multiple sclerosis sO2 and RPE imaging

Quick and simple technique to master, true view of the retina, observes a larger retinal field at any one time compared with ophthalmoscopy, high patient compliance, able to monitor progression of diseases, and low cost compared to other imaging modalities High lateral resolution, fast imaging, high-quality images, patient comfort, and video capability

Image produced is 2D, difficulty observing and assessing abnormalities due to lack of depth appreciation on images, less magnification and image clarity, conditions such as cataracts reduce image clarity, artifact errors may produce unusual images Low depth resolution, high maintenance cost, affected by motion artifacts, distortion of image at the periphery and light scattering affects sO2

High lateral and depth resolution

Based OCT has poor delineation of blood vessels and limited field of view in OCT angiography Only optical absorption imaging, currently requires physical contact, needs more testing before clinically available

OCT

PAOM

Optical absorption based, medium depth perception, and multimodal imaging with other modalities

References [1] A. Taruttis, V. Ntziachristos, Advances in real-time multispectral optoacoustic imaging and its applications, Nat. Photonics 9 (4) (2015) 219. [2] B.I. Gramatikov, Modern technologies for retinal scanning and imaging: an introduction for the biomedical engineer, Biomed. Eng. Online 13 (1) (2014) 52. [3] R.H. Webb, G.W. Hughes, Scanning laser ophthalmoscope, IEEE Trans. Biomed. Eng. 7 (1981) 488–492. [4] D. Huang, et al., Optical coherence tomography, Science 254 (5035) (1991) 1178–1181. [5] A. Bajwa, R. Aman, A.K. Reddy, A comprehensive review of diagnostic imaging technologies to evaluate the retina and the optic disk, Int. Ophthalmol. 35 (5) (2015) 733–755. [6] D.Y. Lin, et al., The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: a comparison with ophthalmoscopy and standardized mydriatic color photography, Am J. Ophthalmol. 134 (2) (2002) 204–213. [7] R.H. Webb, Optics for laser rasters, Appl. Opt. 23 (20) (1984) 3680–3683.

10

Diabetes and Retinopathy

[8] M. Minsky, Memoir on inventing the confocal scanning microscope, Scanning 10 (4) (1988) 128–138. [9] P. Vieira, et al., Tomographic reconstruction of the retina using a confocal scanning laser ophthalmoscope, Physiol. Meas. 20 (1) (1999) 1. [10] P. Vieira, et al., True colour imaging of the fundus using a scanning laser ophthalmoscope, Physiol. Meas. 23 (1) (2001) 1. [11] A.E. Elsner, et al., Reflectometry with a scanning laser ophthalmoscope, Appl. Opt. 31 (19) (1992) 3697–3710. [12] A. Remky, E. Beausencourt, A.E. Elsner, Angioscotometry with the scanning laser ophthalmoscope. Comparison of the effect of different wavelengths, Invest. Ophthalmol. Vis. Sci. 37 (11) (1996) 2350–2355. [13] A. Lompado, et al., Multispectral confocal scanning laser ophthalmoscope for retinal vessel oximetry, in: Spectral Imaging: Instrumentation, Applications, and Analysis, International Society for Optics and Photonics, 2000. [14] A. Remky, et al., Blue-on-yellow perimetry with a scanning laser ophthalmoscope: small alterations in the central macula with aging, JOSA A 18 (7) (2001) 1425–1436. [15] W. Wykes, A. Pyott, Y. Ferguson, Detection of diabetic retinopathy by scanning laser ophthalmoscopy, Eye 8 (4) (1994) 437. [16] A. Manivannan, et al., Clinical investigation of an infrared digital scanning laser ophthalmoscope, Br. J. Ophthalmol. 78 (2) (1994) 84–90. € u € rk, Comparison of spectral-domain optical coherence tomography [17] G. Seymenog˘lu, E. Bas¸ er, B. Ozt and Heidelberg retina tomograph III optic nerve head parameters in glaucoma, Ophthalmologica 229 (2) (2013) 101–105. [18] E.W. Chan, et al., Diagnostic performance of the ISNT rule for glaucoma based on the Heidelberg retinal tomograph, Transl. Vis. Sci. Technol. 2 (5) (2013) 2. [19] J. Liang, D.R. Williams, D.T. Miller, Supernormal vision and high-resolution retinal imaging through adaptive optics, JOSA A 14 (11) (1997) 2884–2892. [20] S.A. Burns, et al., Large-field-of-view, modular, stabilized, adaptive-optics-based scanning laser ophthalmoscope, JOSA A 24 (5) (2007) 1313–1326. [21] C.A. Puliafito, et al., Imaging of macular diseases with optical coherence tomography, Ophthalmology 102 (2) (1995) 217–229. [22] A. Fercher, K. Mengedoht, W. Werner, Eye-length measurement by interferometry with partially coherent light, Opt. Lett. 13 (3) (1988) 186–188. [23] M. Pircher, R.J. Zawadzki, Review of adaptive optics OCT (AO-OCT): principles and applications for retinal imaging, Biomed. Opt. Express 8 (5) (2017) 2536–2562. [24] W. Drexler, et al., Optical coherence tomography today: speed, contrast, and multimodality, J. Biomed. Opt. 19 (7) (2014) 071412. [25] J.A. Izatt, et al., In vivo bidirectional color Doppler flow imaging of picoliter blood volumes using optical coherence tomography, Opt. Lett. 22 (18) (1997) 1439–1441. [26] R.A. Leitgeb, et al., Real-time measurement of in vitro flow by Fourier-domain color Doppler optical coherence tomography, Opt. Lett. 29 (2) (2004) 171–173. [27] M. Ang, et al., Optical coherence tomography angiography: a review of current and future clinical applications, Graefes Arch. Clin. Exp. Ophthalmol. 256 (2) (2018) 237–245. [28] S.S. Gao, et al., Optical coherence tomography angiography, Invest. Ophthalmol. Vis. Sci. 57 (9) (2016) OCT27–OCT36. [29] P.A. Keane, S.R. Sadda, Retinal imaging in the twenty-first century: state of the art and future directions, Ophthalmology 121 (12) (2014) 2489–2500.

Chapter 1 • Complementary capabilities of photoacoustic imaging

11

[30] X. Shu, L.J. Beckmann, H.F. Zhang, Visible-light optical coherence tomography: a review, J. Biomed. Opt. 22 (12) (2017) 121707. [31] H. Koizumi, M.C. Pozzoni, R.F. Spaide, Fundus autofluorescence in birdshot chorioretinopathy, Ophthalmology 115 (5) (2008) e15–e20. [32] T. Otani, S. Kishi, Y. Maruyama, Patterns of diabetic macular edema with optical coherence tomography, Am J. Ophthalmol. 127 (6) (1999) 688–693. [33] A. Hassenstein, A.A. Bialasiewicz, G. Richard, Optical coherence tomography in uveitis patients, Am J. Ophthalmol. 130 (5) (2000) 669–670. [34] N.N. Markomichelakis, et al., Patterns of macular edema in patients with uveitis: qualitative and quantitative assessment using optical coherence tomography, Ophthalmology 111 (5) (2004) 946–953. [35] I. Krebs, et al., Activity of neovascular lesions treated with bevacizumab: comparison between optical coherence tomography and fluorescein angiography, Graefes Arch. Clin. Exp. Ophthalmol. 246 (6) (2008) 811–815. [36] I.I. Bussel, G. Wollstein, J.S. Schuman, OCT for glaucoma diagnosis, screening and detection of glaucoma progression, Br. J. Ophthalmol. 98 (Suppl. 2) (2014) ii15–ii19. [37] J.W. Jeoung, et al., Macular ganglion cell imaging study: glaucoma diagnostic accuracy of spectraldomain optical coherence tomography, Invest. Ophthalmol. Vis. Sci. 54 (7) (2013) 4422–4429. [38] I. Maruko, et al., Subfoveal choroidal thickness after treatment of Vogt–Koyanagi–Harada disease, Retina 31 (3) (2011) 510–517. [39] Y.S. Modi, et al., Multimodal imaging of sarcoid choroidal granulomas, J. Ophthalmic Inflamm. Infect. 3 (1) (2013) 58. [40] P.A. Keane, et al., Characterization of birdshot chorioretinopathy using extramacular enhanced depth optical coherence tomography, JAMA Ophthalmol. 131 (3) (2013) 341–350. [41] D. Goldenberg, et al., Vitreal, retinal, and choroidal findings in active and scarred toxoplasmosis lesions: a prospective study by spectral-domain optical coherence tomography, Graefes Arch. Clin. Exp. Ophthalmol. 251 (8) (2013) 2037–2045. [42] A. Hojjatoleslami, M.R.N. Avanaki, OCT skin image enhancement through attenuation compensation, Appl. Opt. 51 (21) (2012) 4927–4935. [43] M.R.N. Avanaki, et al., Spatial compounding algorithm for speckle reduction of dynamic focus OCT images, IEEE Photon. Technol. Lett. 25 (15) (2013) 1439–1442. [44] S. Hojjatoleslami, M. Avanaki, A.G. Podoleanu, Image quality improvement in optical coherence tomography using Lucy–Richardson deconvolution algorithm, Appl. Opt. 52 (23) (2013) 5663–5670. [45] M.R. Avanaki, et al., Quantitative evaluation of scattering in optical coherence tomography skin images using the extended Huygens–Fresnel theorem, Appl. Opt. 52 (8) (2013) 1574–1580. [46] M.R.N. Avanaki, et al., Investigation of basal cell carcinoma using dynamic focus optical coherence tomography, Appl. Opt. 52 (10) (2013) 2116–2124. [47] M.R.N. Avanaki, A. Hojjat, A.G. Podoleanu, Investigation of computer-based skin cancer detection using optical coherence tomography, J. Mod. Opt. 56 (13) (2009) 1536–1544. [48] S. Adabi, et al., Universal in vivo textural model for human skin based on optical coherence tomograms, Sci. Rep. 7 (1) (2017) 1–11. [49] M.R.N. Avanaki, A. Hojjatoleslami, Skin layer detection of optical coherence tomography images, Optik 124 (22) (2013) 5665–5668. [50] S. Adabi, et al., Optical coherence tomography technology and quality improvement methods for optical coherence tomography images of skin: a short review, Biomed. Eng. Comput. Biol. 8 (2017). 1179597217713475.

12

Diabetes and Retinopathy

[51] A. Taghavikhalilbad, et al., Semi-automated localization of dermal epidermal junction in optical coherence tomography images of skin, Appl. Opt. 56 (11) (2017) 3116–3121. [52] M. Faiza, et al., High-resolution wavelet-fractal compressed optical coherence tomography images, Appl. Opt. 56 (4) (2017) 1119–1123. [53] M.R. Avanaki, A. Podoleanu, En-face time-domain optical coherence tomography with dynamic focus for high-resolution imaging, J. Biomed. Opt. 22 (5) (2017) 056009. [54] Z. Turani, et al., Optical radiomic signatures derived from optical coherence tomography images to improve identification of melanoma, Cancer Res. 79 (8) (2019) 2021–2030. [55] S. Adabi, et al., An overview of methods to mitigate artifacts in optical coherence tomography imaging of the skin, Skin Res. Technol. 24 (2) (2018) 265–273. [56] T.J. Bennett, C.J. Barry, Ophthalmic imaging today: an ophthalmic photographer’s viewpoint—a review, Clin. Exp. Ophthalmol. 37 (1) (2009) 2–13. [57] D.X. Hammer, et al., Compact scanning laser ophthalmoscope with high-speed retinal tracker, Appl. Opt. 42 (22) (2003) 4621–4632. [58] B. Chou, Limitations of the panoramic 200 Optomap, Optom. Vis. Sci. 80 (10) (2003) 671–672. [59] R.F. Spaide, Peripheral areas of nonperfusion in treated central retinal vein occlusion as imaged by wide-field fluorescein angiography, Retina 31 (5) (2011) 829–837. [60] R.W. Dunphy, et al., Structural features anterior to the retina represented in panoramic scanning laser fundus images, Ophthalmic Surg. Lasers Imaging Retina 39 (2) (2008) 160–163. [61] M. Zhang, et al., Projection-resolved optical coherence tomographic angiography, Biomed. Opt. Express 7 (3) (2016) 816–828. [62] W. Liu, H.F. Zhang, Photoacoustic imaging of the eye: a mini review, Photo-Dermatology 4 (3) (2016) 112–123. [63] D.-Y. Yu, S.J. Cringle, Oxygen distribution and consumption within the retina in vascularised and avascular retinas and in animal models of retinal disease, Prog. Retin. Eye Res. 20 (2) (2001) 175–208. [64] P.A. Campochiaro, Molecular pathogenesis of retinal and choroidal vascular diseases, Prog. Retin. Eye Res. 49 (2015) 67–81. [65] S.H. Hardarson, E. Stefa´nsson, Retinal oxygen saturation is altered in diabetic retinopathy, Br. J. Ophthalmol. 96 (4) (2012) 560–563. [66] O.B. Olafsdottir, et al., Retinal oximetry in primary open-angle glaucoma, Invest. Ophthalmol. Vis. Sci. 52 (9) (2011) 6409–6413. [67] S.H. Hardarson, E. Stefansson, Oxygen saturation in central retinal vein occlusion, Am J. Ophthalmol. 150 (6) (2010) 871–875. [68] T.T. Berendschot, et al., Influence of lutein supplementation on macular pigment, assessed with two objective techniques, Invest. Ophthalmol. Vis. Sci. 41 (11) (2000) 3322–3326. [69] S.L. Jacques, Optical properties of biological tissues: a review, Phys. Med. Biol. 58 (11) (2013) R37. [70] M. Xu, L.V. Wang, Photoacoustic imaging in biomedicine, Rev. Sci. Instrum. 77 (4) (2006) 041101. [71] M. Nasiriavanaki, et al., High-resolution photoacoustic tomography of resting-state functional connectivity in the mouse brain, Proc. Natl. Acad. Sci. 111 (1) (2014) 21–26. [72] J. Yao, et al., Noninvasive photoacoustic computed tomography of mouse brain metabolism in vivo, NeuroImage 64 (2013) 257–266. [73] M. Mozaffarzadeh, et al., Linear-array photoacoustic imaging using minimum variance-based delay multiply and sum adaptive beamforming algorithm, J. Biomed. Opt. 23 (2) (2018) 026002. [74] J. Xia, et al., Wide-field two-dimensional multifocal optical-resolution photoacoustic-computed microscopy, Opt. Lett. 38 (24) (2013) 5236–5239.

Chapter 1 • Complementary capabilities of photoacoustic imaging

13

[75] A.-R. Mohammadi-Nejad, et al., Neonatal brain resting-state functional connectivity imaging modalities, Photoacoustics 10 (2018) 1–19. [76] S. Mahmoodkalayeh, et al., Low temperature-mediated enhancement of photoacoustic imaging depth, Sci. Rep. 8 (1) (2018) 4873. [77] N. Meimani, et al., A numerical analysis of a semi-dry coupling configuration in photoacoustic computed tomography for infant brain imaging, Photo-Dermatology 7 (2017) 27–35. [78] R. Manwar, M. Hosseinzadeh, A. Hariri, K. Kratkiewicz, S. Noei, R. Mohammad, N. Avanaki, Photoacoustic signal enhancement: towards utilization of low energy laser diodes in real-time photoacoustic imaging, Sensors 18 (10) (2018) 3498. [79] L. Mohammadi, H. Behnam, M. Nasiriavanaki, Modeling skull’s acoustic attenuation and dispersion on photoacoustic signal, in: Photons Plus Ultrasound: Imaging and Sensing, International Society for Optics and Photonics, 2017. [80] M. Zafar, et al., Development of low-cost fast photoacoustic computed tomography: system characterization and phantom study, Appl. Sci. 9 (3) (2019) 374. [81] J. Yao, et al., Label-free oxygen-metabolic photoacoustic microscopy in vivo, J. Biomed. Opt. 16 (7) (2011) 076003. [82] A. Fatima, et al., Review of cost reduction methods in photoacoustic computed tomography, PhotoDermatology 15 (2019) 100137. [83] S. Hu, et al., Functional transcranial brain imaging by optical-resolution photoacoustic microscopy, J. Biomed. Opt. 14 (4) (2009) 040503. [84] B.T. Cox, et al., Quantitative spectroscopic photoacoustic imaging: a review, J. Biomed. Opt. 17 (6) (2012) 061202. [85] L.V. Wang, L. Gao, Photoacoustic microscopy and computed tomography: from bench to bedside, Annu. Rev. Biomed. Eng. 16 (2014) 155–185. [86] M. Xu, L.V. Wang, Universal back-projection algorithm for photoacoustic computed tomography, Phys. Rev. E 71 (1) (2005) 016706. [87] W. Xing, et al., Integrated optical-and acoustic-resolution photoacoustic microscopy based on an optical fiber bundle, Opt. Lett. 38 (1) (2013) 52–54. [88] K. Maslov, et al., Optical-resolution photoacoustic microscopy for in vivo imaging of single capillaries, Opt. Lett. 33 (9) (2008) 929–931. [89] W. Song, et al., Integrating photoacoustic ophthalmoscopy with scanning laser ophthalmoscopy, optical coherence tomography, and fluorescein angiography for a multimodal retinal imaging platform, J. Biomed. Opt. 17 (6) (2012) 061206. [90] S. Jiao, et al., Photoacoustic ophthalmoscopy for in vivo retinal imaging, Opt. Express 18 (4) (2010) 3967–3972. [91] S.N. Hennen, et al., Photoacoustic tomography imaging and estimation of oxygen saturation of hemoglobin in ocular tissue of rabbits, Exp. Eye Res. 138 (2015) 153–158. [92] S. Hu, et al., Label-free photoacoustic ophthalmic angiography, Opt. Lett. 35 (1) (2010) 1–3. [93] J.M. Thijssen, H.J.M. Mol, M.R. Timmer, Acoustic parameters of ocular tissues, Ultrasound Med. Biol. 11 (1) (1985) 157–161. [94] G.P. Luke, D. Yeager, S.Y. Emelianov, Biomedical applications of photoacoustic imaging with exogenous contrast agents, Ann. Biomed. Eng. 40 (2) (2012) 422–437. [95] W. Li, X. Chen, Gold nanoparticles for photoacoustic imaging, Nanomedicine (London, England) 10 (2) (2015) 299–320. [96] J. Yao, et al., Evans blue dye-enhanced capillary-resolution photoacoustic microscopy in vivo, J. Biomed. Opt. 14 (5) (2009) 054049.

14

Diabetes and Retinopathy

[97] G. Kim, et al., Indocyanine-green-embedded PEBBLEs as a contrast agent for photoacoustic imaging, J. Biomed. Opt. 12 (4) (2007) 1–8. [98] X. Yang, et al., Nanoparticles for photoacoustic imaging, Wiley Interdiscip. Rev. Nanomed. Nanobiotechnol. 1 (4) (2009) 360–368. [99] Y. Zhou, J. Yao, L.V. Wang, Optical clearing-aided photoacoustic microscopy with enhanced resolution and imaging depth, Opt. Lett. 38 (14) (2013) 2592–2595. [100] P. Hai, et al., Near-infrared optical-resolution photoacoustic microscopy, Opt. Lett. 39 (17) (2014) 5192–5195. [101] M.-L. Li, et al., Improved in vivo photoacoustic microscopy based on a virtual-detector concept, Opt. Lett. 31 (4) (2006) 474–476. [102] T. Ma, et al., Systematic study of high-frequency ultrasonic transducer design for laser-scanning photoacoustic ophthalmoscopy, J. Biomed. Opt. 19 (1) (2014) 16015. [103] W. Song, et al., Multimodal photoacoustic ophthalmoscopy in mouse, J. Biophotonics 6 (6–7) (2013) 505–512. [104] S.Z. Jiao, F. Hao, Multimodal microscopy for comprehensive tissue characterizations, in: Advanced Biophotonics: Tissue Optical Sectioning, CRC Press, 2016, pp. 475–505. [105] E.J. Ferna´ndez, et al., Ultrahigh resolution optical coherence tomography and pancorrection for cellular imaging of the living human retina, Opt. Express 16 (15) (2008) 11083–11094. [106] Y. Jia, et al., Split-spectrum amplitude-decorrelation angiography with optical coherence tomography, Opt. Express 20 (4) (2012) 4710–4725. [107] W. Liu, et al., Measuring retinal blood flow in rats using Doppler optical coherence tomography without knowing eyeball axial length, Med. Phys. 42 (9) (2015) 5356–5362. [108] H.F. Zhang, C.A. Puliafito, S. Jiao, Photoacoustic ophthalmoscopy for in vivo retinal imaging: current status and prospects, Ophthalmic Surg. Lasers Imaging 42 (2011) S106–S115.

Chapter 1 • Complementary capabilities of photoacoustic imaging

15

Dipen Kumar is currently a second-year medical student at Wayne State School of Medicine in Detroit, Michigan. He graduated from the University of Illinois at UrbanaChampaign in 2017 majoring in bioengineering with a minor in chemistry. His interests include ophthalmology, surgical medicine, and the applications of technology in medicine.

Dr. Anju Goyal is an associate professor of Ophthalmology and director of Medical Student Education at Kresge Eye Institute, the Department of Ophthalmology of Wayne State University School of Medicine. She completed her Ophthalmology residency, followed by a Glaucoma fellowship, at Wayne State University. Upon graduation in 2005, Dr. Goyal became an active member of the faculty, serving as director of the Residents’ Clinic. In 2011, Dr. Goyal became the director of the Ophthalmology Clinical Elective for Wayne State University School of Medicine students. She has developed multiple innovative programs for student education and mentorship that are locally and nationally recognized. As principal investigator on her grant Vision Detroit, Dr. Goyal worked to unite needed ophthalmic care and health-care education for Detroit’s underserved community. Dr. Goyal is a peer-elected member of the Association of University Professors in Ophthalmology Medical Student Educator’s Council and is actively involved in joint committee work with the American Academy of Ophthalmology. Additionally, she serves as a National Eye Institute’s National Eye Health Education Program Strategic Planning Committee Member. Dr. Goyal is a member of the Alpha Omega Alpha Medical Honor Society, the Wayne State University School of Medicine Alumni Board of Governors, recipient of the Wayne State University School of Medicine’s College Teaching Award, the American Academy of Ophthalmology Secretariat Award and Distinguished Service Award, the Lawrence M. Weiner, MD Award for Teaching and Service, the Lawrence Stocker, MD Award for Compassionate Medicine, Hour Detroit magazine’s Top Doc Awards, and the Kresge Eye Institute Distinguished Alumni Award.

16

Diabetes and Retinopathy

Alan Truhan is from Detroit and went to Wayne State University he has a BFA with a focus on photography. He worked at both Kresge Eye Institute in Detroit and Northwestern Memorial Hospital in Chicago. He is now the manager of the Imaging and Testing Department at Kresge/WSUPG.

Dr. Gary Abrams is a professor of Ophthalmology and director of the Ligon Research Center of Vision at the Kresge Eye Institute of Wayne State University. He received his medical education at the University of Oklahoma and completed a residency in Ophthalmology at the Medical College of Wisconsin and a fellowship in vitreoretinal surgery at the Bascom Palmer Eye Institute of the University of Miami. He was Chair of Ophthalmology at the Kresge Eye Institute of Wayne State University for 17 years and held prior faculty appointments at the Bascom Palmer Eye Institute and the Medical College of Wisconsin. Dr. Abrams has published more than 200 articles and book chapters. He received the 2005 Paul Kayser International Award in Retina Research (Schepens International Society), the 2015 Pyron Award in Retina Research (American Society of Retina Specialists), and the Secretariat and Life Achievement Honor Awards from the American Academy of Ophthalmology. He was president of the Association for Research in Vision and Ophthalmology (ARVO) and the Association of University Professors of Ophthalmology (AUPO). He served as chairman of the ARVO Foundation for Eye Research, was a member of the Executive Committee of the Club Jules Gonin, and currently serves on the Editorial Board of the journal Retina.

Chapter 1 • Complementary capabilities of photoacoustic imaging

Rayyan Manwar received his PhD from University of Windsor, Windsor, Ontario in 2017. His bachelor’s is in Electrical and Electronic Engineering from Islamic University of Technology (IUT), Gazipur, Bangladesh in 2011. Currently, he is a postdoctoral fellow at OPIRA Lab, Wayne State University, Detroit, MI. His research interests include MEMS-based design, fabrication, and characterization, photoacoustic and ultrasound imaging.

17

2

Intraretinal fluid map generation in optical coherence tomography images Pla´cido L. Vidala,b, Joaquim de Mouraa,b, Jorge Novoa,b, Manuel G. Penedoa,b, Marcos Ortegaa,b a

˜ A, A C ORUN˜ A, SPAIN DEPARTMENT OF COMP UTER SCIENCE, UNIVERSITY O F A CORUN CIT IC -R ESE AR CH CENT ER OF I N FOR MAT ION AND C OMMUNICAT ION TE CHNOLOGIES , UNIVER SIT Y O F A CORUN˜ A, A C ORUN˜ A, SPAIN

b

1 Introduction Computer-aided diagnosis (CAD) has become, thanks to the advancements in computing and the transference to other fields, one of the main tools in the assistance of the diagnosis process. These systems not only reduce the time that an expert clinician needs to assess a given pathology and perform their diagnostic, but also allow them to make decisions based on an independent system. This means isolating the results from the subjectivity of the human expert as well as increasing the repeatability of the results, reducing this way the error rate in a critical field as is the medical one [1–3]. Currently, the eye fundus represents one of the most studied parts of the human body, as it represents the internal part that is easiest to access and analyze. Additionally, pathologies not only from the visual system, but also from other systems can be detected that contribute to the functionality of this organ. This can be easily seen in the retina, the neurosensory part of the eye, as it comprises both the vascular and the neural system. Thus, pathologies that directly or indirectly affect processes involving these two systems may leave their footprint in the retinal structures. As an example of relevant diseases of the nervous system that leave their footprint in the retina, we can find Parkinson [4, 5], Alzheimer [6, 7], and multiple sclerosis [8, 9] among others of similar severity and prevalence. Regarding diseases of the vascular system, pathologies like hypertension [10, 11], diabetes [12], and atherosclerosis [13, 14] among others also degenerate the retinal structures and can be detected by means of the retinal analysis. Given the great amount of relevant diseases with a significant impact on the quality of life of the patients that can be detected and studied in this organ and the ease to access and analyze its constituent structures, the eye fundus has sensibly gained importance among the proposal of new CAD methodologies. Diabetes and Retinopathy. https://doi.org/10.1016/B978-0-12-817438-8.00002-X © 2020 Elsevier Inc. All rights reserved.

19

20

Diabetes and Retinopathy

2 Optical coherence tomography: Background and significance Currently, one of the main means to analyze the ocular structures is the optical coherence tomography (OCT) [15, 16]. This medical image modality is able to generate crosssectional images of the living tissue in a noninvasive way. To do so, the device projects a low-coherence beam of light toward the target tissues and, by means of analyzing the phase and amplitude of the reference light source compared to the reflected one [17], they create a representation of these tissues. This medical imaging technique, despite its lower tissue penetration capabilities, is able to offer images with a superior resolution (in the order of microns) with respect to other capture techniques such as ultrasounds or MRI. This enhanced resolution makes this modality perfect for the analysis of the complex eye structures. The OCT scans can be divided into three different types: A-Scan, B-Scan, and C-Scan (Fig. 1). An A-Scan represents the reflected amplitudes along the axis of the propagated light, a B-Scan is a series of A-Scans that create the 2D cross-sectional image of the analyzed tissues, and a C-Scan is a group of B-Scans that allow to obtain a 3D view of the covered section. To improve the quality of the images, these OCT systems usually scan the same section multiple times. In this way, and by means of proprietary algorithms, they generate a final OCT scan with better quality and less noise. This strategy requires the patient remain still during a longer capturing process, which in some cases and despite the modern automatic alignment techniques implemented in the devices is not possible. In these cases, lowering the quality scans is the only viable option. In these particular cases and to improve the quality of the images despite the capture conditions, automatic denoising methodologies are often added as a preprocessing step in these CAD systems [18].

FIG. 1 Different parts of an OCT volume. Each B-Scan is accompanied with the corresponding near-infrared reflectance retinography of the eye fundus indicating the section of the retina the OCT scans were taken from.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

21

Thanks to this medical imaging technology, multiple retinal structures and their respective pathological variants can be detected. Some of the most common structures that can be identified and analyzed with this medical image modality are the optic disc [19, 20] (representing the point of exit of the optic nerve), the retinal histological structure (the disposition and organization of the different retinal components), the vessel tree [21, 22] (representing an extension of the eye vascular system that helps to nourish the retina), and the choroid [23, 24] (the main vascular layer of the eye). All these structures can be used as reference to detect multiple pathologies like the epiretinal membrane [25, 26] with the histological analysis and the vascular tortuosity [27, 28] or relevant markers for biometric identification/image registry [29] with the vascular tree analysis. In this chapter, we focus on the B-Scan OCT images of the retina, where we can identify three main regions, as shown in Fig. 2: the vitreous humor, the aforementioned choroid, and the retinal tissues. The vitreous humor represents the internal fluid of the eyeball. Due to its low-density fluid nature it is represented as an homogeneous dark zone. The choroid, on the other hand, represents the vascular layer of the eye that supplies the retina and thus presents tubular-like patterns. Finally, the retina is represented as a layered structure where, thanks to the OCT image modality, the different functional structures of its cells can be identified. In this chapter, we only consider only the innerlimiting membrane (ILM) and the retinal pigmented epithelium (RPE), as they constitute the layers that define the limiting borders of the retina, the region of interest (ROI) of the scope of this chapter.

FIG. 2 Three main regions identified in an OCT B-Scan of the retina: the vitreous humor, the choroid, and the retina itself (delimited in green by the two limiting ILM and RPE layers).

22

Diabetes and Retinopathy

Herein, we further explain the detection of the macular edema (ME) [30], pathology that is the result of intraretinal fluid accumulations that render progressively blind the afflicted [31, 32] and the two main currents followed nowadays: the classical segmentation and the recently proposed alternative paradigm based on a regional analysis. These fluid accumulations, part of the effects of diseases like diabetes and age-related macular degeneration, conform one of the main causes of blindness in developed countries.

3 The classical segmentation approach The first works centered in solving the issue of the fluid detection used direct segmentation approaches. As an example, one of these studies proposed by Wilkins et al. [33] shows a procedure that was posteriorly followed by the majority of subsequent studies: an initial preprocessing of the image to unify the characteristics of the samples to ease the posterior workload of the algorithm (in most of the cases this consists in a denoising step), the segmentation algorithm itself that creates an initial candidate list, and a final filtering process that reduces the number of false positives (FPs) that the methodology generated. In this particular case, Wilkins et al. apply a series of filters to correct the irregularities of the OCTimage like a median filter to suppress noise and a bilateral filter to smooth the image preserving the edges. Then, the authors apply a threshold based on a value that was obtained empirically and filter the FPs based on the size and gray level of the candidate regions. The main issue with this methodology is that the majority of its rules are based on values obtained empirically, being dependent on the dataset used by Wilkins et al. This strategy of using empirically determined thresholds was followed in other studies like the one by Roychowdhury et al. [34], where they improved the detections with constraints to filter the FPs based also on the cyst disposition among the retinal layers, shape, and intensity constraints. As an alternative to these empirically defined thresholds, Gonza´lez et al. [35] and €us et al. [36] were the first to adapt clustering methods to create a segmentation Mattha of these cystoid fluid accumulations. Gonza´lez et al. applied a region flooding and merging to generate a considerable set of candidate pixel clusters followed by the use of texture, shape, location, and size features to filter the irrelevant candidates. On the other hand, €us et al. used a k-means to obtain the initial candidates and segmented the FPs Mattha with a kNN classifier using as features the minimal distance to the bottom retinal boundary and their shape (using Hu moments). Additionally, to filter the FPs product of shadows, they also applied an A-Scan filtering to remove the artifacts that are the product of the vessel shadows via an analysis of the gray-level intensity histogram. Similarly, Chiu et al. [37] proposed a methodology that also uses texture and intensity features like in the studies conducted Gonza´lez et al. but improving this classification with a graph theory and dynamic programming framework that finds the retinal layers and helps to isolate the class boundaries. Other studies like the proposals of Rashno et al. [38, 39] transform the images to an alternative domain (in this case, neutrosophic, based on the proposal of Guo and Cheng [40]) where other approaches can be used to segment the fluid bodies in this alternative point of view.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

23

Other studies have expanded these ideas to the 3D domain. Some of the studies have used the C-Scan information to aid in the preprocessing, like the study performed by Esmaeili et al. that used this extra dimension to improve the denoising steps. Others, like the proposal of Wang et al. [41], create an initial segmentation and propagate the results through the rest of B-Scans of the OCT volume. In the same way, others have expand classical approaches like the clustering techniques that were previously used to the 3D domain, as seen in the study conducted by Wang et al. [42]. Similarly, there are some studies that also expand texture-based proposals to the 3D domain with a voxel analysis. An example of these studies can be the ones performed by Chen et al. [43] and Xu et al. [44], that define representative features from the different successive B-Scans and perform the aforementioned voxel classification. Nonetheless, despite the 3D domain considered in these studies, all these methodologies also use a preprocessing and FP filtering steps. Unsupervised techniques have also been used, as proposed by Montuoro et al. [45] where they use principal component analysis to find the image features instead of manually defining them, and also posteriorly use graph theory strategies and layer information to refine the 3D results. Finally, recent proposals applied deep learning-based techniques to find the precise segmentation of these cystoid fluid leakages. Many of them proposed networks based on medical imaging specific neural networks like the studies by Girish et al. [46], Lee et al. [47], Venhuizen et al. [48], and Roy et al. [49], that used modified architectures based on the U-Net model proposed by Ronneberger et al. [50].

4 Fluid identification by means of a regional analysis All the previously detailed approaches present the same issue: while they attain the desired defined segmentation of these fluid accumulations, there are several cases where this segmentation simply does not exist. These cystoid bodies may appear mixed with the normal tissues of the retina or even with other pathological structures that are common for the same diseases that provoke the leakages. In Fig. 3, two examples of fluid regions with different complications for a precise segmentation can be appreciated. Both images show how not all the cysts present a defined cellular barrier. Given this diffuse transition between the pathological region and the normal tissue, the only accurate way to analyze

FIG. 3 Example sections of images from two different OCT devices that present different complications for a precise fluid segmentation.

24

Diabetes and Retinopathy

this data is via a regional analysis and a representation that allows different levels of fluid presence in the same region. Given that the experts were able to differentiate where the fluid region was present but not their exact limits, Moura et al. [51, 52] proposed a way to study these areas that overcame the segmentation complications. Instead of achieving an accurate segmentation, these studies proposed to analyze regions of a given window size using texture and intensity features, demonstrating the feasibility of the regional analysis. In fact, in this way, the interexpert disagreement product of the diffuse borders of these complex regions is diminished and the cystoid areas can be properly detected independently of their underlying problems. In this chapter, we further revise the methodology that was specifically created to take advantage of this regional analysis [53], and how to use it to generate a robust and coherent representation of the identified fluid regions that can be used by clinicians to assess pathologies like the aforementioned ones. This methodology proposes the use of two different representation maps for the assessment of the B-Scans: a binary map and a color map. The methodology steps (shown in Fig. 4) needed for the creation of both maps are presented hereafter, including a brief explanation of the nuances of the design decisions.

FIG. 4 Main steps for the creation of the fluid maps.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

25

4.1 ROI extraction As explained in the introduction, the eyeball presents two main structures that can appear in the retinal OCT images that are used in this study apart from the retina: the vitreous humor and the choroid. The vitreous humor represents a fluid region with properties similar to the cystoid fluid and that are not present anywhere else in the retina. Given this, excluding the vitreous humor allows the use of common fluid characteristics between the vitreous humor and the fluid leakages without fear of FPs in this extraretinal region. On the other hand, as the choroid represents a vascular layer, several circular patterns that are also present in the fluid accumulations can be seen in this region. Thus, excluding it as with the vitreous humor lets the classifier use these patterns in the retinal region. Moreover, as the algorithm performs a complete sampling of the image, reducing the region to analyze also results in a significant computational optimization when generating the maps (Fig. 5). To remove these retinal layers, the algorithm uses the vertical gradients of the pixels as weights for the Dijkstra algorithm [54] basing our approach on the proposal of Chiu et al. [55]. This algorithm first finds the ILM and IS-OS layers to, subsequently, use these layers as landmarks to progressively find the adjacent layers close to them. In this way, the ILM and RPE layers of the retina that limit the desired ROI are obtained.

4.2 Image sampling As the methodology performs a regional analysis, the image needs to be analyzed coherently. Thus, the sampling algorithm divides the image into overlapping windows. As shown in Fig. 6, the shape of the retina is not uniform and, because of this, many samples will partially fall outside the ROI. In the same figure, the solution proposed by Vidal et al. [53] is presented. As the methodology uses texture features that require the study of the spatial relations of the pixels in the image, the only way to prevent inclusion of external patterns in the process is to directly remove these unwanted zones. To do so, an algorithm that finds the biggest

FIG. 5 OCT A-Scan original image and resulting ROI with the choroid and the vitreous humor already removed.

26

Diabetes and Retinopathy

FIG. 6 Example of sample that partially falls outside the ROI, and the result of the maximum extracted subsample. The area outside the ROI is represented in red and the valid retinal region in green.

rectangular section exclusively formed by ROI pixels is applied. In this way, if an extracted sample contains vitreous humor or choroidal tissue, they are directly removed while most of the relevant patterns are preserved.

4.3 Classification To classify the samples, a feature vector of texture and intensity features common in the medical field is extracted from each analyzed window. Several considerations were taken into account when selecting the set of features that better described both the retinal and the pathological region. Also, different feature selection techniques were applied to select the most relevant markers. First, as the cystoid regions present a clear disposition to have darker values in the samples with respect to other normal or pathological structures (like drusen or normal retinal tissues), several intensity-based features like (but not limited to) the mean, maximum, minimum, and standard distribution were extracted to give the model the capability to analyze this point of view. Additionally, as samples falling in limiting regions, contain both

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

27

cystoid and noncystoid regions, would present an irregular distribution of the gray levels, features describing this gray-level distribution like representative percentiles, obliquity, kurtosis, etc., were also included in the feature set. Another important factor differentiating the fluid regions with respect to other retinal structures is the orientation and organization of their borders. While normal retinal layers usually present horizontal-like patterns (as expected from a layered structure) and shadows casted by vessels and other dense structures mostly present completely vertical patterns (as they are artifacts product of the light beam direction during the A-Scan creation), the cystoid fluid bodies do not have a defined shape or follow these constraints. These cystoid bodies, as they conform a fluid accumulation, present a circular shape in the OCT cross-sectional section and have gradients in all angles. Thus, a descriptor of these gradient orientations could greatly aid the identification of these regions and as such the Histogram of Oriented Gradients (HOG) [56] and Gabor [57, 58] features describing these gradient orientations are used. The HOG descriptors, as its name implies, provide a representation of the texture using the orientations of its gradients as principal feature. In a similar way, Gabor filters are composed of a set of convolutional masks that measure the response of a texture to a pattern at a certain frequency and orientation. On the other hand, as these regions are composed of low-density fluid leakages, they present an homogeneous dark texture that greatly contrasts with the patterns that are present in normal retinal tissues. To analyze these patterns, two representative texture descriptors are used, describing the spatial relationship between the pixels of a texture. The gray-level cooccurrence matrix (GLCM) [59] and the local binary patterns (LBP) [60]. The GLCM texture descriptor describes the texture as the probability of, at a series of given angles and distances, two gray levels being together. Regarding the LBP texture descriptor, they represent the texture of the sample as a binary vector describing the positive or negative gradients between a given pixel and the surrounding ones. This texture descriptor is particularly interesting because it describes the texture in terms of gray-level differences instead of direct values, making it robust to brightness changes caused by the configuration of the device. Additionally, if desired, both descriptors can be made rotation invariant. This characteristic can be of interest depending on the quality and capture region of the extracted images but, as this methodology was tested with images that were all taken centered in the macula, this characteristic represents neither a significant change in the feature importance nor an impact on the results. Finally, as the problematic cystoid regions are usually mixed with normal retinal tissues, a fractal [61, 62] dimension-based texture descriptor was also added. This descriptor gives an idea of the lacunarity and coarseness level of a given region, which is greatly increased in the cystoid contour regions that present diffuse patterns. Posteriorly, these descriptors are classified with a model that was trained using a set of representative samples from both classes. In this way, and thanks to the previous complete image sampling, different regions of the image that belong to a cystoid fluid leakage scenario can be pinpointed.

28

Diabetes and Retinopathy

4.4 Binary map creation At this point, the methodology was common for two types of output maps. The binary maps have the objective of creating a binary mask over the cystoid regions. This binary mask, in its original idea, would represent the pixels that are surrounded by or belong to a cystoid fluid region. To do so, the optimal case would consist of a complete sampling of the image, where there would be a window for each ROI pixel. As an optimization, instead of a full sampling of the image, a partial sampling with an interpolation between windows is used to complement the information loss. Here, a nearest-neighbor (NN) interpolation is enough to create a defined cystoid region with the quality to replace the previously mentioned full pixel coverage. As shown in Fig. 7, this process consists in assigning to each pixel the same category as the closest window center.

4.5 Color map creation To create the color maps indicating the confidence level of each pixel, a more complex process is used instead. This type of map not only takes into account the closest classification to assign a value to the pixels, but also considers the value of each window that contained that same pixel when sampling the image. In this way, every pixel in the image represents the confidence that its neighborhood place in it for belonging to the cystoid class. The steps for the creation of these color maps are as follows: 1. Voting. To evaluate the confidence, the methodology considers each overlapping window as a ballot. Every single window will increase the confidence of the region they are covering by one vote, which will generate a map as the one displayed in Fig. 8. In this way, the confidence of a given pixel increases the more it appears in windows that were classified as cystoid regions.

FIG. 7 Window centers classified as cystoid region and the resulting interpolated binary map using the nearestneighbor designed method.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

29

FIG. 8 Windows containing a cystoid region (red) and the result of the accumulative voting procedure.

2. Normalization. In the previous step, not all pixels in the image are overlapped by the same number of windows. Two regions of the image present a lesser number of votes than the rest of the ROI: the limiting regions of the ROI and the image centers that became closer or farther during the rounding of the results. The first case occurs as no samples are taken outside the ROI. Because of this, every sample that would have overlapped the ROI and was centered outside is not counted toward the final voting. This results in a lesser number of votes in these ROI border regions compared to the internal ones. The second case depends on the chosen overlap and the image size, as when calculating the sample centers some of them may fall in noninteger positions, being rounded to its closest position. This rounding step is shown in the prenormalization voting result in Fig. 9 as a lattice pattern, product of the limiting areas of the windows that have been slightly displaced during this rounding process. This issue is also clearly visible in the green centers of each constituent window in Fig. 8, where it can be seen that the image does not create a perfect tabular pattern, but an slightly uneven one in some regions.

FIG. 9 Original voting result with the lattice patterns and the resulting normalized gray scale confidence map.

30

Diabetes and Retinopathy

FIG. 10 Color map generated with an overlap of 52px between samples.

Nonetheless, this issue is easily fixed by applying a normalization to the results. As shown in Fig. 9, by transforming the votes into the percentage of positive ballots over the total number of windows that overlapped a given pixel, a homogeneous detection that lacks the lattice pattern is obtained. Additionally, possible cystoid regions that are present near the limits of the retina that would be shown with lesser intensity, now will present the value they truly should have. 3. Color mapping. Finally, an intuitive and representative color scale is mapped into the resulting normalized confidence map. This map allows to visualize the different levels of confidence represented in the matrix in a way a human expert could easily analyze. The final result is shown in Fig. 10, where an scale of the confidence represented by each color in the map has also been added.

5 Discussion and conclusions Currently, one of the most relevant internal parts of the human organism is the retina as it not only allows the analysis of pathologies from other organic systems like the neural or the vascular, but these analysis can also be done in vivo and in a noninvasive way thanks to the optical coherence tomography medical imaging modality and the ease of access of the retina itself. Thanks to these characteristics, OCT has become one of the main

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

31

ophthalmological resources for the CAD systems, aiding the diagnosis of pathologies like the age-related macular degeneration or the diabetic retinopathy, both among the main causes of blindness. Both diseases also have in common the leakage of fluids between the retinal layers, rendering the patients progressively blind if not detected and treated in time. Currently, two paradigms are being used to detect these fluid leakages in the OCT imaging modality: a classical segmentation approach, where the contour of the fluids is detected and a regional analysis composed of two different complementary maps. The regional approach, composed of a binary map and a color map both herein explained, allow to analyze these pathological fluid patterns even on complex regions that the segmentation approach, simply, could not analyze. The binary regional map pretends to create a direct representation of the regional classification, where every single detection is presented to the expert clinician, the boundaries being clearly delimited. The confidence color map, on the other hand, compensates the lax behavior toward false detections of the binary regional map by representing the confidence of the neighboring regions on each detection. As shown in Fig. 11, while some FPs are detected and shown in the binary map, the

FIG. 11 Window centers, binary map, and color map generated with a sample overlap of 56px.

32

Diabetes and Retinopathy

color map only represents with intense red tones (the high confidence marker) the regions that truly belong to a cystoid region. On the other hand, the binary maps are less sensible to the changes in the sample overlap. The color maps entirely rely on the number of windows that overlap a pixel to represent a progressive confidence gradient, while the binary maps interpolate these values. If we compare the behavior of the two maps with an extremely low overlapping (Fig. 12) and a high one (Fig. 11), the binary map obtains similar results with both configurations. On the other hand, the confidence map shows stepper transitions, and even the false detection that in the original map was completely absent is visible at this level. In Figs. 13–16, a set of representative images belonging to a complete heterogeneous set of complex pathological cases is depicted. These images were taken with two representative devices of the domain, a CIRRUS HD-OCT Carl Zeiss Meditec confocal scanning laser ophtalmoscope and a Spectralis OCT confocal scanning laser ophtalmoscope from Heidelberg Engineering. The models that were used to generate the maps were trained with samples coming from the same device (Figs. 13 and 14) and with samples taken from images of both devices (Figs. 15 and 16). All these cases illustrate the variety of situations that a system of

FIG. 12 Window centers, binary map, and color map generated with a sample overlap of 32px.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

33

FIG. 13 Cirrus images and corresponding maps that were generated using a classifier trained with samples coming from that same capture device and an overlap between samples of 52px.

FIG. 14 Spectralis images and corresponding maps that were generated using a classifier trained with samples coming from the same capture device and an overlap between samples of 52px.

34

Diabetes and Retinopathy

FIG. 15 Cirrus images and corresponding maps that were generated using a classifier trained with samples coming from both capture devices and an overlap between samples of 52px.

FIG. 16 Spectralis images and corresponding maps that were generated using a classifier trained with samples coming from both capture devices and an overlap between samples of 52px.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

35

these characteristics could encounter in the medical field. Moreover, to establish a baseline, the first row of each figure contains a perfectly healthy retinal OCT B-Scan. The main issue the Cirrus device presents to a methodology based on texture descriptors like the one presented in this chapter is the similarity between the lower layers and the cystoid fluid in some images. The second row of Figs. 13 and 15 presents this situation clearly, being one of the cases where texture-based systems should have issues. The binary maps of this case illustrated in Fig. 13 show how the system actually presents detections in these problematic areas, but thanks to the color map representation based on the voting system only the area with true cystoid fluid regions is marked with an intense red color. In the third row of the same figure, a more complex case is presented compared to the previous one. In this image, not only the fluid accumulations are mixed with the darker retinal tissues, but also a small set of microcysts (also called pseudocysts as they usually do not present a defined cellular barrier like the bigger cystoid fluid bodies) is present in the left region. As shown in the binary map, the system perfectly detects this fluid accumulation and also satisfactorily assigns a moderate level of confidence in the color maps. The fourth row shows a similar case, where both perfectly defined cysts and mixed ones are in the original OCT B-Scan and are equally correctly detected. The fifth row of this figure shows an image with both a group of the aforementioned pseudocystoid bodies and a darkened layer with several shadows projected onto it uniformly, giving a fluid-like appearance and texture. As in the previous case, the pseudocyst groups are correctly identified and marked by the system, but some of these detections also fall in these darkened layers (as seen in the binary map). Nonetheless, the color map correctly assigns a dim confidence value, as these spurious detections shown in the binary map represent a minimal part of the votes that the color map takes into account when assigning its value. This image represents one of the advantages of a system based on neighboring information over one based on only the sample information, but both maps offer relevant points of view for this analysis. Finally, the last row presents a case where both the layer intensity and the borders of the cystoid bodies make the detections difficult. The rightmost fluid accumulation is completely mixed with the internal fluid and does not have a clear boundary to segment. Nonetheless, thanks to the regional approach analyzed in this chapter, both binary and color map representations create a successful and robust detection on all these fluid areas. This is a good example of how the diffuse limits of these cystoid bodies can be better represented by a regional approach than a precise segmentation strategy. Regarding the maps that were generated with the Spectralis device (Fig. 14), we can see beforehand the two main differences between the two capture device configurations. Spectralis images present a higher level of contrast between the retinal tissues and the fluid regions. Additionally, Cirrus images present a Gaussian-like noise pattern, while the Spectralis capture device presents a rain-like noise pattern with a similar gray level between noisy spots. In the same way as with the Cirrus images, the first row shows a healthy retinal OCT B-Scan, as baseline for the rest of the detections.

36

Diabetes and Retinopathy

In second and third rows, two different cases of diffuse alterations of the fluid patterns are shown. Both cases present denser sediments that, as in the first case, make the correct border detection difficult or, as in the second case, completely eliminate the possibility of it. In the first case, the reader can see how the methodology presented in this chapter is able to correctly distinguish the overall region shape and correctly present it in both map modalities. On the other hand, in the second case, the color maps offer a dim detection on the correct regions, but it is in the binary maps where an expert clinician could assess more easily these detections. It is in these cases where the binary maps offer the most relevant results, as in situations with such level of difficulty, a direct representation of the results is the one that is most descriptive and explanatory. The third row presents an analogous case to the situation presented in the third row of Fig. 13: a set of fluid accumulations mixed with small size cysts and diffuse borders as well as shadow-like patterns make difficult the detection difficult. Nonetheless, as the contrast levels in the images from this device are more stepped, the system offers a clear solution in both cases even in the diffuse and pseudocystoid zones. The fourth row shows a case where the cysts are greatly mixed with normal retinal tissue and the line between fluid region and normal retinal tissue is completely blurred. Moreover, as this system includes an important component of gradient orientation analysis, the dominant layer gradients make it harder for the system to distinguish the underlying pathological regions. Despite these issues, it can be seen how in the binary maps all the pathological regions (even the small microcystic ones near a central retinal vessel) are clearly marked. The color maps, on the other hand, all present a high level of confidence in clear fluid regions, but these detections diminish significantly in the fluid accumulations the more we get closer to the shape of a normal retinal layer. Finally, the last row presents the case of an image that is mixed with multiple pathological hyperreflective bodies, diffuse retinal tissues mixed with fluid accumulations and shadows cast by dense bodies. In the color maps, the reader can see how all the fluid accumulations are correctly detected by the system, but the circular rightmost body with a pattern similar to the retinal vessel shown in the previous row is marked with a lower level of confidence. Nonetheless, all the pathological bodies are marked despite the absence of defined borders and homogeneous texture. Also, the darker patterns that usually confuse similar methods are correctly ignored. This, in combination with the case presented in the third row, is one of the most complex cases that a system of this kind could encounter and, as shown in these examples, is still able to offer a satisfactory representation of the detections. Lastly, Figs. 15 and 16 show how the models can be improved if trained with samples coming from multiple devices. These maps show how training with samples with such different characteristics and conditions force the methodology to learn the main patterns that describe the issue at hand (the intraretinal fluid characteristics). This is particularly interesting in a methodology based on a regional analysis, as the system is trained with samples that can contain multiple types of patterns. For instance, a sample from a cystoid region can also contain part of the normal tissues that surround the same fluid accumulation. The results are sensibly better already in the baseline maps of Fig. 15 (the Cirrus images). If we compare these maps with the original results of Fig. 13, we can see how

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

37

the dim detections that are product of the FPs completely disappeared, and even the binary maps present a more clear idea of the nonpathological nature of the image. The same example is also shown in the fourth row, where the healthy regions that resulted in false detections also disappeared with the mixed model. These detections that were made with the model that was trained with mixed images result in the binary maps with a more adjusted region to the pathological zones, whereas color maps result in a more stepped transition between levels than the original ones. On the other hand, the Spectralis images benefited from the opposite effect. In Fig. 16, the reader can see how the original low confidence detections in the problematic regions now are represented with more intensity (specially in the sensible case presented in the third and fifth rows of Fig. 14). Moreover, the addition of the Cirrus patterns to the model (from the point of view of the Spectralis original model) helped to better shape both the diffuse region already captured in the original model in the last row as well as the internal fluid of the bigger central cyst that originally was not considered by the binary map. While the Cirrus maps focused more on texture features, the Spectralis maps based their detections mostly on gradient information (understandable in images with contrast levels higher than the Cirrus). This gradient information, when lacking in examples like the ones shown in the third and last two rows of Fig. 14, resulted in some misclassifications and low confidence detections. Thus, when training a model with patterns of both devices, the system is forced to learn and use these two different points of view to assess the type of region a given sample belongs to. This results in more coherent and robust maps that are able to identify the cystoid regions regardless of their complexity. Conclusively, these two representations of fluid presence allow the clinician to easily assess the severity of the cystoid fluid presence in the retinal layers including regions that could not be approached by a classical segmentation proposal. The binary map offers a clear and direct view of the regional analysis, while the confidence color map returns a representation of the fluid regions that is robust to the sensibilities of the used classification system.

Acknowledgments This study was supported by the Instituto de Salud Carlos III, Government of Spain and FEDER funds of the European Union through the DTS15/00153 research projects and by the Ministerio de Economı´a y Competitividad, Government of Spain through the DPI2015-69948-R research project. Also, this study has received financial support from the European Union (European Regional Development Fund [ERDF]) and the Xunta de Galicia, Centro singular de investigacio´n de Galicia accreditation 2016–19, Ref. ED431G/ 01; and Grupos de Referencia Competitiva, Ref. ED431C 2016-047.

References [1] J. Novo, J. Rouco, N. Barreira, M. Ortega, M.G. Penedo, A. Campilho, Wivern: a web-based system enabling computer-aided diagnosis and interdisciplinary expert collaboration for vascular research, J. Med. Biol. Eng. 37 (6) (2017) 920–935, https://doi.org/10.1007/s40846-017-0256-y. [2] J. Novo, A. Hermida, M. Ortega, N. Barreira, M.G. Penedo, J.E. Lo´pez, C. Calvo, Hydra: a web-based system for cardiovascular analysis, diagnosis and treatment, Comput. Methods Programs Biomed. 139 (2017) 61–81.

38

Diabetes and Retinopathy

[3] A. Ferna´ndez, M. Ortega, J de Moura, J. Novo, M.G. Penedo, Detection of reactions to sound via gaze and global eye motion analysis using camera streaming, Mach. Vis. Appl. 29 (7) (2018) 1069–1082. [4] J. Ahn, J.-Y. Lee, T.W. Kim, E.J. Yoon, S. Oh, Y.K. Kim, J.-M. Kim, S.J. Woo, K.W. Kim, B. Jeon, Retinal thinning associates with nigral dopaminergic loss in de novo Parkinson disease, Neurology 91 (11) (2018) 1003–1012. [5] M.M. Moschos, I.P. Chatziralli, Evaluation of choroidal and retinal thickness changes in Parkinson’s disease using spectral domain optical coherence tomography, Semin. Ophthalmol. 33 (4) (2018) 494–497. [6] M.M. Moschos, I. Markopoulos, I. Chatziralli, A. Rouvas, S.G. Papageorgiou, I. Ladas, D. Vassilopoulos, Structural and functional impairment of the retina and optic nerve in Alzheimer’s disease, Curr. Alzheimer Res. 9 (7) (2012) 782–788. [7] C.Y. Lui Cheung, Y.T. Ong, M.K. Ikram, S.Y. Ong, X. Li, S. Hilal, J.-A.S. Catindig, N. Venketasubramanian, P. Yap, D. Seow, C.P. Chen, T.Y. Wong, Microvascular network alterations in the retina of patients with Alzheimer’s disease, Alzheimer’s Dementia 10 (2) (2014) 135–142. -Ferna´ndez, B. Bejarano, [8] J. Toledo, J. Sepulcre, A. Salinas-Alaman, A. Garcı´a-Layana, M. Murie P. Villoslada, Retinal nerve fiber layer atrophy is associated with physical and cognitive disability in multiple sclerosis, Mult. Scler. J. 14 (7) (2008) 906–912. [9] R. Alonso, D. Gonza´lez-Moro´n, O. Garcea, Optical coherence tomography as a biomarker of neurodegeneration in multiple sclerosis: a review, Mult. Scler. Relat. Disord. 22 (2018) 77–82. [10] A.V. Stanton, B. Wasan, A. Cerutti, S. Ford, R. Marsh, P.P. Sever, S.A. Thom, A.D. Hughes, Vascular network changes in the retina with age and hypertension, J. Hypertens. 13 (12 Pt 2) (1995) 1724–1728. [11] A.D. Hughes, E. Martinez-Perez, A.-S. Jabbar, A. Hassan, N.W. Witt, P.D. Mistry, N. Chapman, A.V.Stanton, G. Beevers, R. Pedrinelli, K.H. Parker, S.A.M. Thom, Quantification of topological changes in retinal vascular architecture in essential and malignant hypertension, J. Hypertens. 24 (5) (2006) 889–894. [12] K.W. Bronson-Castain, M.A. Bearse Jr, J. Neuville, S. Jonasdottir, B. King-Hooper, S. Barez, M.E. Schneck, A.J. Adams, Early neural and vascular changes in the adolescent type 1 and type 2 diabetic retina, Retina 32 (1) (2012) 92–102, https://doi.org/10.1097/IAE.0b013e318219deac. [13] R.F. Mullins, S.R. Russel, D.H. Anderson, G.S. Hageman, Drusen associated with aging and age-related macular degeneration contain proteins common to extracellular deposits associated with atherosclerosis, elastosis, amyloidosis, and dense deposit disease, FASEB J. 14 (7) (2000) 835–846. [14] W.R. Morris, The eyes give the clue, Postgrad. Med. 91 (1) (1992) 195–202. [15] D. Huang, E.A. Swanson, C.P. Lin, J.S. Schuman, W.G. Stinson, W. Chang, M.R. Hee, T. Flotte, K. Gregory, C.A. Puliafito, Optical coherence tomography, Science 254 (5035) (1991) 1178–1181. [16] Z. Zhang, R. Srivastava, H. Liu, X. Chen, L. Duan, D.W. Kee Wong, C.K. Kwoh, T.Y. Wong, J. Liu, A survey on computer aided diagnosis for ocular diseases, BMC Med. Inf. Decis. Mak. 14 (1) (2014) 80. [17] J.M. Schmitt, Optical coherence tomography (OCT): a review, IEEE J. Sel. Top. Quantum Electron. 5 (4) (1999) 1205–1215, https://doi.org/10.1109/2944.796348. [18] G. Samagaio, J. de Moura, J. Novo, M. Ortega, Optical coherence tomography denoising by means of a Fourier Butterworth filter-based approach, in: S. Battiato, G. Gallo, R. Schettini, F. Stanco (Eds.), Image Analysis and Processing—ICIAP 2017, Springer International Publishing, 2017, pp. 422–432. [19] J. Novo, M.G. Penedo, J. Santos, Optic disc segmentation by means of GA-optimized topological active nets, in: A. Campilho, M. Kamel (Eds.), Image Analysis and Recognition, Springer, Berlin, Heidelberg, 2008, pp. 807–816. [20] G.B. Melo, R.D. Libera, A.S. Barbosa, L.M.G. Pereira, L.M. Doi, L.A.S. Melo, Comparison of optic disk and retinal nerve fiber layer thickness in nonglaucomatous and glaucomatous patients with high myopia, Am. J. Ophthalmol. 142 (5) (2006) 858–860.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

39

[21] J. de Moura, J. Novo, M. Ortega, P. Charlo´n, 3D retinal vessel tree segmentation and reconstruction with OCT images, in: A. Campilho, F. Karray (Eds.), Image Analysis and Recognition, Springer International Publishing, Cham, 2016, pp. 716–726. [22] J. de Moura, J. Novo, M. Ortega, N. Barreira, M.G. Penedo, Vessel tree extraction and depth estimation with OCT images, in: O. Luaces, J.A. Ga´mez, E. Barrenechea, A. Troncoso, M. Galar, H. Quintia´n, E. Corchado (Eds.), Advances in Artificial Intelligence, Springer International Publishing, Cham, 2016, pp. 23–33. [23] L. Zhang, K. Lee, M. Niemeijer, R.F. Mullins, M. Sonka, M.D. Abra`moff, Automated segmentation of the choroid from clinical SD-OCT, Invest. Ophthalmol. Vis. Sci. 53 (12) (2012) 7510. [24] H. Laviers, H. Zambarakji, Enhanced depth imaging-OCT of the choroid: a review of the current literature, Graefe’s Arch. Clin. Exp. Ophthalmol. 252 (12) (2014) 1871–1883. [25] S. Baamonde, J. de Moura, J. Novo, M. Ortega, Automatic detection of epiretinal membrane in OCT images by means of local luminosity patterns, in: I. Rojas, G. Joya, A. Catala (Eds.), Advances in Computational Intelligence, Springer International Publishing, Cham, 2017, pp. 222–235. [26] S. Baamonde, J. de Moura, J. Novo, J. Rouco, M. Ortega, Feature definition and selection for epiretinal membrane characterization in optical coherence tomography images, International Conference on Image Analysis and Processing, Springer, 2017, pp. 456–466. ´ lvarez, M. Ortega, Retinal vascular tortuosity assess[27] L. Ramos, J. Novo, J. Rouco, S. Romeo, M.D. A ment: inter-intra expert analysis and correlation with computational measurements, BMC Med. Res. Methodol. 18 (1) (2018) 144. [28] L. Sa´nchez, J. Novo, A. Ferna´ndez, J.M. Barja, Unified methodology for evaluating vessel tree tortuosity metrics in eye fundus images, Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems, June, 2013, pp. 528–529. https://doi.org/10.1109/ CBMS.2013.6627862. [29] M. Ortega, J. Rouco, J. Novo, M.G. Penedo, Vascular landmark detection in retinal images, in: R. Moreno-Dı´az, F. Pichler, A. Quesada-Arencibia (Eds.), Computer Aided Systems Theory— EUROCAST 2009, Springer, Berlin, Heidelberg, 2009, pp. 211–217. vez, J. de Moura, J. Novo, M. Ortega, M.I. Ferna´ndez, Automatic identification [30] G. Samagaio, A. Este of macular edema in optical coherence tomography images, Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications—Vol. 4, VISAPP, INSTICC, SciTePress, 2018, pp. 533–540. https://doi.org/ 10.5220/0006544105330540. [31] G. Samagaio, J de Moura, J. Novo, M. Ortega, Automatic segmentation of diffuse retinal thickening edemas using optical coherence tomography images, Procedia Comput. Sci. 126 (2018) 472–481. vez, J de Moura, J. Novo, M.I. Ferna´ndez, M. Ortega, Automatic macular edema [32] G. Samagaio, A. Este identification and characterization using OCT images, Comput. Methods Programs Biomed. 163 (2018) 47–63. [33] G.R. Wilkins, O.M. Houghton, A.L. Oldenburg, Automated segmentation of intraretinal cystoid fluid in optical coherence tomography, IEEE Trans. Biomed. Eng. 59 (4) (2012) 1109–1114. [34] S. Roychowdhury, D.D. Koozekanani, S. Radwan, K.K. Parhi, Automated localization of cysts in diabetic macular edema using optical coherence tomography images, in: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2013, pp. 1426–1429. [35] A. Gonza´lez, B. Remeseiro, M. Ortega, M.G. Penedo, P. Charlo´n, Automatic cyst detection in OCT retinal images combining region flooding and texture analysis, IEEE International Symposium on Computer-Based Medical Systems, 2013, pp. 397–400. [36] M. Pilch, K. Stieger, Y. Wenner, M.N. Preising, C. Friedburg, E. Meyer zu Bexten, B. Lorenz, Automated segmentation of pathological cavities in optical coherence tomography scans, Invest. Ophthalmol. Vis. Sci. 54 (6) (2013) 4385.

40

Diabetes and Retinopathy

[37] S.J. Chiu, M.J. Allingham, P.S. Mettu, S.W. Cousins, J.A. Izatt, S. Farsiu, Kernel regression based segmentation of optical coherence tomography images with diabetic macular edema, Biomed. Opt. Express 6 (4) (2015) 1172–1194. [38] A. Rashno, B. Nazari, D. Koozekanani, P. Drayna, S. Sadri, H. Rabbani, K. Parhi, Fully-automated segmentation of fluid regions in exudative age-related macular degeneration subjects: kernel graph cut in neutrosophic domain, PLoS ONE 12 (10) (2017) 1–26. [39] A. Rashno, D. Koozekanani, P.M. Drayna, B. Nazari, S. Sadri, H. Rabbani, K. Parhi, Fully-automated segmentation of fluid/cyst regions in optical coherence tomography images with diabetic macular edema using neutrosophic sets and graph algorithms, IEEE Trans. Biomed. Eng. 65 (5) (2017) 989–1001. [40] Y. Guo, H.D. Cheng, New neutrosophic approach to image segmentation, Pattern Recogn. 42 (5) (2009) 587–595. [41] T. Wang, Z. Ji, Q. Sun, Q. Chen, S. Yu, W. Fan, S. Yuan, Q. Liu, Label propagation and higher-order constraint-based segmentation of fluid-associated regions in retinal SD-OCT images, Inf. Sci. 358 (C) (2016) 92–111. [42] J. Wang, M. Zhang, A.D. Pechauer, L. Liu, T.S. Hwang, D.J. Wilson, D. Li, Y. Jia, Automated volumetric segmentation of retinal fluid on optical coherence tomography, Biomed. Opt. Express 7 (4) (2016) 1577–1589. [43] X. Chen, M. Niemeijer, L. Zhang, K. Lee, M.D. Abra`moff, M. Sonka, Three-dimensional segmentation of fluid-associated abnormalities in retinal OCT: probability constrained graph-search-graph-cut, IEEE Trans. Med. Imaging 31 (8) (2012) 1521–1531. [44] X. Xu, K. Lee, L. Zhang, M. Sonka, M.D. Abra`moff, Stratified sampling voxel classification for segmentation of intraretinal and subretinal fluid in longitudinal clinical OCT data, IEEE Trans. Med. Imaging 34 (7) (2015) 1616–1623. [45] A. Montuoro, S.M. Waldstein, B.S. Gerendas, U. Schmidt-Erfurth, H. Bogunovi c, Joint retinal layer and fluid segmentation in OCT scans of eyes with severe macular edema using unsupervised representation and auto-context, Biomed. Opt. Express 8 (3) (2017) 1874–1888. [46] G.N. Girish, B. Thakur, S.R. Chowdhury, A.R. Kothari, J. Rajan, Segmentation of intra-retinal cysts from optical coherence tomography images using a fully convolutional neural network model, IEEE J. Biomed. Health Inf. 23 (1) (2018) 296–304. [47] C.S. Lee, A.J. Tyring, N.P. Deruyter, Y. Wu, A. Rokem, A.Y. Lee, Deep-learning based, automated segmentation of macular edema in optical coherence tomography, Biomed. Opt. Express 8 (7) (2017) 3440–3448. [48] F.G. Venhuizen, B. van Ginneken, B. Liefers, F. van Asten, V. Schreur, S. Fauser, C. Hoyng, T. Theelen, C. I. Sa´nchez, Deep learning approach for the detection and quantification of intraretinal cystoid fluid in multivendor optical coherence tomography, Biomed. Opt. Express 9 (4) (2018) 1545–1569. [49] A.G. Roy, S. Conjeti, S.P.K. Karri, D. Sheet, A. Katouzian, C. Wachinger, N. Navab, ReLayNet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional network abs/1704.02161, http://arxiv.org/abs/1704.02161, 2017. [50] O. Ronneberger, P. Fischer, T. Brox, U-Net: convolutional networks for biomedical image segmentation, in: N. Navab, J. Hornegger, W.M. Wells, A.F. Frangi (Eds.), Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, Springer International Publishing, Cham, 2015, pp. 234–241. [51] J. Moura, P.L. Vidal, J. Novo, J. Rouco, M. Ortega, Feature definition, analysis and selection for cystoid region characterization in optical coherence tomography, in: Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 21st International Conference KES-2017, Marseille, France, 6–8 September 2017, 2017, , pp. 1369–1377. [52] J. Moura, J. Novo, J. Rouco, M.G. Penedo, M. Ortega, Automatic identification of intraretinal cystoid regions in optical coherence tomography, 2017, pp. 305–315.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

41

[53] P.L. Vidal, J de Moura, J. Novo, M.G. Penedo, M. Ortega, Intraretinal fluid identification via enhanced maps using optical coherence tomography images, Biomed. Opt. Express 9 (10) (2018) 4730–4754. [54] E.W. Dijkstra, A note on two problems in connexion with graphs, Numer. Math. 1 (1) (1959) 269–271. [55] S.J. Chiu, X.T. Li, P. Nicholas, C.A. Toth, J.A. Izatt, S. Farsiu, Automatic segmentation of seven retinal layers in SD-OCT images congruent with expert manual segmentation, Opt. Express 10 (10) (2010) 19413–19428. [56] N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in: Computer Vision and Pattern Recognition, CVPR’05, 2005, , pp. 886–893. [57] D. Gabor, Theory of communication, J. Inst. Electr. Eng. 93 (1946) 429–457. [58] M. Haghighata, S. Zonouzb, M. Abdel-Mottaleba, CloudID: trustworthy cloud-based and crossenterprise biometric identification, Expert Syst. Appl. 42 (21) (2015) 7905–7916. [59] R.M. Haralick, K. Shanmugam, I.H. Dinstein, Textural features for image classification, IEEE Trans. Syst. Man Cybern. SMC-3 (6) (1973) 610–621. [60] T. Ojala, M. Pietikainen, T. Maenpaa, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, IEEE Trans. Pattern Anal. Mach. Intell. 24 (7) (2002) 971–987. [61] S. Buczkowski, S. Kyriacos, F. Nekka, L. Cartilier, The modified box-counting method: analysis of some characteristic parameters, Pattern Recogn. 31 (1998) 411–418. [62] O.S. Al-Kadi, D. Watson, Texture analysis of aggressive and nonaggressive lung tumor CE CT images, IEEE Trans. Biomed. Eng. 55 (7) (2008) 1822–1830.

42

Diabetes and Retinopathy

Pla´cido L. Vidal received his degree in Computer ˜ a, Engineering in 2017 from the University of A Corun Spain. He is currently pursuing his M.Sc. degree in Bioinformatics and Ph.D. degree in Computer Science. His research interests are focused on medical image analysis, computer vision, and pattern recognition.

Joaquim de Moura received his degree in Computer ˜a Engineering in 2014 from the University of A Corun (Spain). In 2016, he received his M.Sc. degree in Computer Engineering from the same university. He is currently pursuing his Ph.D. degree in Computer Science in a collaborative project between ophthalmology centers in ˜ a. His research Galicia and the University of A Corun interests include computer vision, machine learning algorithms and analysis, and medical imaging processing of various kinds.

Jorge Novo received his M.Sc. and Ph.D. degrees (cum Laude) in Computer Science from the University of ˜ a in 2007 and 2012, respectively. He has also A Corun worked, as a visiting researcher, with CMR images in the detection of landmark points at Imperial College London and as a postdoctoral research fellow at the INEB and INESC-TEC research institutes in the development of CAD systems for lung cancer diagnosis with chest CT images. His main research interests lie in the fields of computer vision, pattern recognition, and biomedical image processing.

Chapter 2 • Intraretinal fluid map generation in optical coherence tomography

43

Manuel G. Penedo received the B.S. and Ph.D. degrees in Physics from the University of Santiago de Compostela, Spain, in 1990 and 1997, respectively. He is currently a Professor at the Department of Computer Science in the ˜ a, Spain, and the coordinator of the University of A Corun Center for Research in Information and Communication Technologies (CITIC). His main research interests include computer vision, biomedical image processing, and video processing.

Marcos Ortega received his M.Sc. degree in Computer Science in 2004 and his Ph.D. degree cum Laude in 2009. He also worked on face biometrics studying the face evolution due to aging effects as a visiting researcher in the University of Sassari and methods for age estimation under different facial expression conditions as a visiting postdoctoral fellow in the University of Amsterdam. He is currently an Associate Professor in the Department of ˜ a. His Computer Science at the University of A Corun research areas of interest are medical image analysis, computer vision, biometrics, and human behavior analysis.

3

Fully automated identification and clinical classification of macular edema using optical coherence tomography images Joaquim de Mouraa,c, Gabriela Samagaioa,c, Jorge Novoa,c, Marı´a Isabel Ferna´ndezb, Francisco Go´mez-Ullab, Marcos Ortegaa,c a CIT IC -R ESE AR CH CENT ER OF I N FOR MAT ION AND C OMMUNICAT ION TE CHNOLOGIES , ´ MEZ- ULLA ˜ A, SP AIN b OPHTHALMOLOGICAL I NSTITUTE GO UNIVERSITY OF A C ORUN˜ A, A CORUN AND DEPARTMENT OF OP HT HALMOLOGY, UNI VERSITY HOSPITAL OF SANTIAGO DE C OM P OST E L A , SA N T I AG O D E C OM P OST E L A , SP A I N c DEPART MENT OF COMP UTER SCI ENCE , UNIVER SIT Y O F A CORUN˜ A, A C ORUN˜ A, SPAIN

1 Background and significance Recent advances in artificial intelligence (AI) produced a significant impact on the field of the automatic analysis of medical images [1]. These advances allowed the design of new technological systems that assist clinicians in the analysis and monitoring of many diseases such as breast cancer [2, 3], lung cancer [4, 5], brain cancer [6, 7], or colon cancer [8, 9], among others of significative relevance. These systems facilitate the early pathological identification, improving the quality of diagnosis and management of the patients. In that sense, nowadays, computer-aided diagnosis (CAD) systems have become a relevant research topic in medicine such as medical imaging [10], audiometry [11, 12], or diagnostic radiology [13], among others. These computational tools are used to help clinicians in the complex task of analysis of many clinical scenarios through the interpretation of different types of medical imaging modalities [14–16]. In this way, CAD systems have become a part of the routine clinical practice, facilitating and simplifying the work of the clinical specialists [17–19]. In ophthalmology, CAD systems have widely spread over the years, proving to be a very interesting field of research [20–22]. These systems facilitate the development of personalized treatments of different eye conditions associated with systemic diseases or vision disorders [23–25]. Optical coherence tomography (OCT) is a noninvasive medical imaging modality that is widely used for clinicians to capture images of different types of ocular

Diabetes and Retinopathy. https://doi.org/10.1016/B978-0-12-817438-8.00003-1 © 2020 Elsevier Inc. All rights reserved.

45

46

Diabetes and Retinopathy

FIG. 1 Representative example of the common terminologies used for OCT images: C-scan, B-scan, and A-scan.

tissues [26–28]. This medical image modality uses the principle of interferometry to generate, in real time, a high-resolution cross-sectional scans of the main retinal tissues [29, 30]. The OCT device produces a set of images, B-scans, through the recovery of the longitudinal and lateral A-scan reflections. A complete set of B-scans produces a single three-dimensional (3D) image from the eye fundus, known as C-scan. Fig. 1 presents an illustrative representation of the common terminologies of OCT images. Normally, OCT images facilitate an easy inspection of the main retinal tissues [31] and vascular structures [32], which enables the identification of abnormal structures that are present in these retinal scans [33, 34]. In this context, the clinical specialists use the information provided by these OCT scans to diagnose different diseases that can affect the human eyes such as the diabetic macular edema (DME) or age-related macular degeneration (AMD) diseases, which are among the leading causes of reversible and preventable blindness in the industrialized countries [35]. In particular, DME is a microvascular complication of diabetes that constitutes a concerning global health problem [36]. Despite that this macular disease is derived from the same fundamental cause, the leakage and the accumulation of intraretinal fluid, also known as macular edema (ME), it presents significant heterogeneity and variability among each other. In this way, these different characteristics make the automatic identification of this ocular disorder a particularly complex task. Fig. 2 shows a set of OCT images of patients with different degrees of DME disease. In particular, we

Chapter 3 • Fully automated identification and clinical classification of macular edema 47

FIG. 2 Examples of OCT images. First row, OCT images of healthy patients. Second row, OCT images of patients with DME disease.

can see the remarkable deterioration of the morphological structures within the retinal tissues. Based on the OCT image modality, Otani et al. [37] presented a classification of ME into three different clinical categories: cystoid macular edema (CME), serous retinal detachment (SRD), and diffuse retinal thickening (DRT). For that the authors used different

FIG. 3 Example OCT scan with the simultaneous presence of the three types of ME disease.

48

Diabetes and Retinopathy

clinical characteristics that are present in the OCT images as reflectivity or thickness of the retina. Posteriorly, Panozzo et al. [38] complemented this study classifying also the ME in these categories but using five parameters to characterize them. The parameters used in this case were: morphology, volume, reflectivity, the thickness of the retina, and the presence of the epiretinal membrane. Nowadays, this clinical classification is widely used worldwide by clinical specialists for the diagnosis of the DME disease. Fig. 3 shows an OCT scan with the simultaneous presence of the three types of ME disease, where we can observe the considerable variability that presents this macular disorder. In the recent years, some computational proposals were presented focusing their studies in the automatic identification of intraretinal fluid using OCT images. As reference, Hassan et al. [39] employed a support vector machine (SVM) classifier using a subset of five features that are obtained from the OCT images. These features are: three based on the retinal thickness and two based on the regional characteristics of fluid regions. Schlegl et al. [40] presented a fully automatic system that uses a deep learning approach for the quantification of CME regions using OCT images. Lu et al. [41] presented a system that uses the convolutional neural network (CNN) model for the segmentation of CME and SRD edemas. Similarly, Girish et al. [42] presented a system for the analysis and segmentation of the CMEs using a CNN model. Rashno et al. [43] presented a system for the identification of CME regions in OCT images. To do so, the authors used the neutrosophic domain information combined with graph algorithms. In this chapter, we present the first fully automatic system for the identification of three different pathological types of ME (CME, SRD, and DRT) using OCT images. To achieve this, two retinal regions are delimited: one corresponding to the inner limiting membrane (ILM)/outer plexiform layer (OPL) region and other to the OPL/retinal pigment epithelium (RPE) region. Then, the system localizes the presence of all the types of ME inside of these retinal regions. For that the system combines clinical knowledge with image processing methods (for the identification of CME and SRD edemas) and machine learning strategies (for the identification of DRT edemas). Finally, the system presents an intuitive representation of the presence of all types of MEs identified and characterized by our methodology. The chapter is organized in this manner. Section 2 includes all phases of the methodology. Section 3 presents the results and discussions of the proposed experiments. Section 4 includes the conclusions of this work and suggestions for the possible future of this line of research.

2 Computational identification and characterization of the MEs As input, the presented methodology receives an OCT image. Each OCT image is centered on the macula, which facilitates the identification of the abnormal macular structures within the retinal tissue, as is the case of the MEs. Using as reference clinical

Chapter 3 • Fully automated identification and clinical classification of macular edema 49

FIG. 4 Main stages of the presented pipeline for the identification and classification of MEs: CME, SRD, and DRT.

criterions, the presented system first establishes the retinal region, which contains as limits the RPE and the ILM retinal layers. Posteriorly, two retinal regions are defined: the outer and the inner retina. The lower region, the outer retina is limited by the OPL/RPE retinal layers, whereas the superior region, the outer retina, is limited by the ILM/OPL retinal layers, being the OPL. For the identification of the fluid region occupied by each ME type, the method uses three different strategies. Regarding the presence of CME and SRD pathological regions, the system locates the presence of these two types by combining clinical restrictions with image processing strategies. Regarding the presence of DRT pathological regions, to identify the “sponge-like” pattern, a machine learning strategy was implemented only in the outer region of the retina. And lastly, the system displays a tagged OCT scan with the precise identification of each ME type. As described in Fig. 4, the presented methodology consists of two main stages, namely as: region of interest delimitation and identification of the different types of ME. Each of these stages will be discussed here.

2.1 Region of interest delimitation In the human eye, the retina is an organ that is composed of different layers. OCT scans permit an easy visualization of these retinal layers with histological quality [44, 45].

FIG. 5 OCT scan with the retinal layer boundaries and the identified retinal regions (inner and outer retina).

50

Diabetes and Retinopathy

In particular, almost all the OCT capture devices allow the visualization of 10 retinal layers, as we can see in Fig. 5. These retinal layers are: RPE, nerve fiber layer, ILM, outer nuclear layer, OPL, inner nuclear layer, inner plexiform layer, ganglion cell layer, external limiting membrane, and junctions between the inner and outer segments (ISOS). To facilitate the localization of the area occupied by each ME type, the presented methodology segments four retinal boundaries: ILM, RPE, ISOS, and OPL. These retinal layers facilitate the delimitation of the retina and the subsequent subdivision into inner and outer retinal regions [46]. 1. Retinal layers segmentation: In this stage, four main retinal layer boundaries are identified: ILM, RPE, ISOS, and OPL. The retinal layers mentioned are the most significant, since they delimit the regions of the retina that are most used by clinical specialists in the detection of different ocular disorders. In particular, to segment the ILM, RPE, and ISOS layers, we follow the work of Chiu et al. [47], for its simplicity and for being a consolidated and robust strategy. The presented system employees graph theory [48] to represent each OCT scan as a node graph, where each node corresponds to a pixel. The optimal pathways from each side of the OCT scans are determined by dynamic programming [49]. The minimum weighted are calculated by the Dijkstra’s shortest path first algorithm [50], identifying the aimed retinal layers. Fig. 6A shows a representative scheme of the ILM, RPE, and ISOS layer identification using a graph theory approach. The patients with ME disease usually present an advanced state of degradation in the innermost layers of the retina, which difficults an accurate segmentation of the OPL [51]. Therefore, the previous method did not present acceptable results under these aforementioned conditions. To solve this issue, a new strategy based on a region growing technique [52] was designed to segment this retinal layer, as we can see in Fig. 6B. The ISOS layer, which was initially detected, is utilized as a reference to determine the area over it. In this way, first, N seeds are generated within this region, where N represents 10% of the width of the input OCT image. The use of a representative number of seeds in the region of interest ensures the extraction of the OPLs even at significant stages of DME disease. Then, the surrounding pixels are grouped with other pixels with similar intensity of values. The upper boundary of the

FIG. 6 An illustrative scheme of the retinal layers segmentation stage. (A) Scheme of the ILM, ISOS, and RPE layers segmentation using a graph theory approach. (B) Scheme of the OPL layer segmentation using region growing algorithm.

Chapter 3 • Fully automated identification and clinical classification of macular edema 51

FIG. 7 OCT scan with the precise segmentation of the aimed four retinal layer boundaries.

FIG. 8 A representative scheme of the region of interest delimitation stage. (A) Input OCT scan. (B) Extraction of the region of the retina between the RPE and the ILM layer boundaries. (C) Extraction of the inner retina between the OPL and the ILM layer boundaries. (D) Extraction of the outer retina between the OPL and the ILM layer boundaries.

resultant region describes the OPL. In Fig. 7, we can observe an OCT image, where the ILM, OPL, ISOS, and RPE retinal layers were extracted. 2. Division of the retina in two subregions: Outer and inner retina. Next, two retinal regions are delimited: one corresponding to the inner retina and other for the outer retina. Each retinal region was delimited using the previously identified retinal layer boundaries as baseline. Thus, the inner retina is between the OPL and ILM layer boundaries, while the outer retina is between the OPL and RPE layer boundaries [53],

52

Diabetes and Retinopathy

FIG. 9 An illustrative example of the region of interest delimitation step. (A) Input OCT scan. (B) OCT scan after the extraction of the retinal regions, where the red region represents the inner retina, whereas the blue region represents the outer retina.

as we can see in Fig. 8. This strategy significantly reduces the area of search for each type of ME, which increases the efficiency of the presented system in the identification process. In Fig. 9, we can see a representative scheme of the region of interest delimitation stage, where the red region indicates the inner retina and the blue region indicates the outer retina.

2.2 Identification of the different types of macular edema In medical imaging, the task of automatic identification of pathological regions is a common practice in many computer vision and graphic systems [54–56]. Most recently, significant advances have been made to facilitate this clinical task, including the identification of ME disease in this chapter [57]. As indicated, the ME is defined as an intraretinal fluid accumulation within the main tissues of the retina [58]. In particular, these MEs present significant heterogeneity and variability between each other [59]. In the presented methodology, three different strategies were designed for the characterization of the pathological regions that are characterized by each type of ME. These ME types are: SRD, DRT, and CME. Each one of these strategies is going to be discussed next. 1. Serous retinal detachment: In the OCT images, the SRD edemas are recognized by the specialists as single hyporeflective fluid accumulations centered in the macular region [60]. This ME type typically appears in the outer retina (OPL/RPE). As consequence, the fluid accumulation produces a dome-like elevation as a result of the detached retinal tissues, typically, with a significant contrast when compared with the surrounding tissues [61, 62]. Fig. 10 illustrates the abnormal accumulation of fluid within the region

FIG. 10 OCT scan with SRD disease, marked as +.

Chapter 3 • Fully automated identification and clinical classification of macular edema 53

FIG. 11 OCT scans with the automatic identification of the presence of SRD edemas, marked as +.

of the photoreceptors (ISOS/RPE layer boundaries), which leads to a breakdown of the normal anatomical arrangement of the retina and its supporting tissues. Using the clinical knowledge as reference, an automatic method was implemented to detect the presence of this ME type using the OCT images. For that, the search space is restricted to the outer retina. Then, we apply a multilevel thresholding algorithm [63] inside this retinal region. This algorithm uses the pixel-level information to segment the regions with similar intensity profiles inside the outer retina. The optimal threshold value was determined empirically [46]. Finally, a list of different clinical criteria was used to reduce the set of possible false SRD candidates [64, 65]. The used clinical criteria are: the relative position, minimum area, morphology, thickness of the region comprehended between the RPE and the ISOS layers, and intensity profile of the retinal tissues. In Fig. 11A and B, we can see two representative OCT scans with the automatic identification of this ME type. 2. Cystoid macular edema: CMEs are usually described as fluid regions with a significant low-intensity pattern when compared with the surrounding retinal tissue [66]. These edemas present a considerable variability in their morphological structures that could appear since cystoid to a petaloid-like appearance [67]. In the early stages of DME disease, the CME edemas emerge in the inner retinal region. However, in more severe clinical stages of this macular disease, they could spread over the outer retinal region,

FIG. 12 OCT scan with CME disease, marked as *.

54

Diabetes and Retinopathy

FIG. 13 OCT scans with the automatic identification of the presence CME edemas, marked as *.

where the contrast with these retinal layers is significantly lower, thus enormously complicating their identification [68]. Moreover, the presence of these MEs can lead to a significant alteration of the thickness of the retina, which is perceptible mainly in the macular zone [69]. Fig. 12 illustrates an OCT scan with CME disease. For the identification of the CME edemas, a similar strategy was adopted to the one proposed to identify the presence of the SRD edemas. In particular, this strategy locates the presence of fluid accumulation by combining the clinical knowledge with an adaptive multilevel thresholding algorithm. For that, first, the system searches for CME edemas in both retinal regions. In this sense, the method uses a threshold value that was determined empirically [46]. Subsequently, both results are combined to obtain a unique representation of these pathological structures. For that, a Watershed algorithm was applied based on the flooding simulation process [70]. Finally, the system combines different clinical knowledge to eliminate the presence of false CME candidates [71–73]. Fig. 13A and B illustrates two OCT scans with the automatic identification of different CME edemas in both retinal regions. 3. Diffuse retinal thickening: DRT edemas are characterized by a retinal swelling in the outer retina that presents a reduced reflectivity. These edemas are generally recognized by their undefined morphological shape and a “sponge-like” appearance [74]. This criterion is commonly used by the clinicians to determine the presence of these ME type in the OCT scans [75]. Besides, DRT edemas do not have a well-defined limiting

FIG. 14 OCT scan with DRT edema, marked in yellow.

Chapter 3 • Fully automated identification and clinical classification of macular edema 55

membrane, which leads to the propagation of the retinal fluid within the retinal layers [76]. Consequently, the outer retina suffers a significant alteration in its morphological architecture, as illustrated in Fig. 14. For the identification of the DRT edemas, a learning strategy was implemented only in the outer retinal region. This specific strategy was developed due to the great variability of patterns that present the OCT scans, characterizing this ME type. For that, a complete and heterogeneous set of 18 features was obtained from this retinal region [77]. To determine the optimal subset of features that better characterize the DRT edemas, three different feature selectors were applied: sequence forward selector (SFS), SVM-forward selector (SVM-FS), and robust feature selection (RFS). Then, the naive Bayes (NB) and SVM classifiers were trained to test the potential of the implemented method. Finally, a postprocessing step was applied to address the possible misclassifications. Fig. 15 illustrates the designed pipeline for the

FIG. 15 An illustrative scheme of the DRT edema detection stage. First, a set of features is extracted and chosen by different feature selection approaches. Then, a machine learning strategy is employed to validate the potential of the designed system. Finally, a postprocessing step with an aggregation factor is used to rectify the possible misclassifications.

56

Diabetes and Retinopathy

identification of the presence of DRT edemas that is composed of three main steps: feature extraction, classification, and postprocessing. Each one of these steps is going to be discussed next. • Feature extraction: In this step, a representative set of features was obtained in windows within the search space, the outer retina. In particular, the method uses a window width of a predefined size. This value was empirically determined [46]. On the other hand, the height of the windows is variable and depends on the thickness of the outer retina. Then, a set of 18 features was extracted to detect the DRT edemas within the defined windows, including intensity image analysis, gray-level intensity histogram (GLIH), mask thickness analysis, and texture analysis (GLCM) [78]. And finally, to obtain the optimal subset of features that describes the DRT edemas, three different feature selectors were applied: SFS [79], SVM-FS [80], and RFS [81]. These selectors were used successfully in similar applications of medical imaging and related areas. In particular, the SFS algorithm, as a forward-oriented selection method, initiates with an empty set of features and inserts one feature to the final set each step until a predetermined number of features is achieved or the algorithm does not provide further improvement. The RFS algorithm selects the optimal set of features utilizing an emphasizing joint l2, 1-norm regularization, those with joint sparsity. Finally, SVM-FS algorithm employs a l1-norm regularization for predicting the set of features that most accurately distinguish the presence of DRT in the OCT scans. • Classification: In this step, two representative classifiers were trained from the subset of features that were previously chosen. In particular, we used the NB [82] and SVM [83] classifiers. The NB technique is a probabilistic classifier with an assumption of independence among predictors, whereas the SVM technique creates a model as a nonprobabilistic linear classifier that finds the optimal hyperplane that separates two classes. These learning algorithms have been used to solve similar problems in medical imaging. • Postprocessing stage: The presence of different types of artifacts (very common in OCT images) and retinal structures may alter the precise identification of the DRT

FIG. 16 OCT scan with the automatic identification of the presence of DRT edemas using the postprocessing stage. Yellow regions, direct result of the machine learning stage. Green regions, result after the postprocessing stage with the aggregation factor, where consecutive DRT regions were joined.

Chapter 3 • Fully automated identification and clinical classification of macular edema 57

edemas [84]. To do that, we employed a postprocessing stage that uses an aggregation factor (d) to rectify the misclassifications that are originated by these retinal artifacts and structures, improving the performance of the presented system. This aggregation factor merges two contiguous DRT regions, using as criterion the interval (d) between these identified pathological regions. In this manner, we obtained merged pathological regions that are closer to the clinical scenario [37, 38]. Fig. 16 illustrates an OCT scan with the automatic identification of the presence of DRT edemas using the postprocessing stage.

3 Results and discussion The presented system was validated employing 170 OCT scans from different individuals with different degrees of DME disease. All scans were taken using a Spectralis OCT capture device from Heidelberg Engineering using a preset seven line raster B-scan and with a gap of 240 μm between each B-scan. These scans are centered in the macula, being taken from the right and left eyes. In addition, this capture device represents one of the most widely used by the ophthalmological services. The local medical ethics committee approved the study and the tenets of the Declaration of Helsinki were observed. Regarding the experiments, the initial dataset was manually labeled by an expert clinician, who identified all the types of ME in the OCT scans. The methodology was evaluated through the four significant metrics: accuracy, recall, precision, and F-measure. These metrics are expressed as described in Eqs. (1)–(4), where (TN), (TP), (FN), and (FP) represent true negative, true positive, false negative, and false positive, respectively. Accuracy ¼

TP + TN TP + TN + FP + FN

Recall ¼

TP TP + FN

Precision ¼ Fmeasure ¼ 2*

TP TP + FP

Precision*Recall Precision + Recall

(1) (2) (3) (4)

With respect to SRD edemas, they are not as frequent as other types of MEs. In fact, this ME type only affects a reduced group of patients. Due to this, the employed dataset presents only 10 cases with SRD edemas. These 10 SRD edemas were accurately localized by the presented method. With respect to CME edemas, we divided the evaluation process into two subretinal regions, one corresponding to the outer retina and other to the inner retina. We validated the throughput of the presented system using 379 CME edemas that were manually labeled by an expert clinician. As result, the presented method reaches F-measure values of 81.65% and 96.08% in the outer and inner retinal regions, respectively. The difference between the obtained results is related to the fact that the inner retina presents a greater

58

Diabetes and Retinopathy

contrast and clearly defined limits, facilitating the precise identification of this ME type in this retinal region. Complementary, we report the overall behavior of the presented system throughout the retinal region, reaching a global 91.99% of F-measure. With respect to DRT detections, a machine learning approach was implemented for the localization of this ME type. We validated the efficiency of the presented method using 68,976 samples of non-DRTedemas and 51,346 samples of DRTedemas. These samples were labeled by a clinical expert. The proposed experiments were repeated using different configurations: RFS, SVM-FS, and SFS feature selectors; and NB and SVM classifiers. Both learning strategies were trained and validated using a 10-fold cross-validation with 10 repetitions. Regarding the selected features, we observed that most of the features were taken from the mask of the retinal thickness, GLCM texture-based, and intensity features. In particular, these features offer a high discriminant power of this ME type in OCT images. The best results were achieved with the NB classifier combined with the RFS feature selector and the SVM classifier combined with the SFS feature selector, where accuracy achieved values of 87.49% and 86.14%, respectively. With respect to the machine learning strategy, two representative classifiers were trained using the selected subset of features. As result, the proposed tool reaches F-measure values of 84.45% and 82.79% using the best configurations of NB and SVM classifiers, respectively. As we can see, the presented method provided satisfactory results, being the NB classifier slightly better than the SVM classifier.

FIG. 17 OCT scan with the automatic identification of the three pathological types of ME: CME as *, SRD as +, and DRT as yellow.

Chapter 3 • Fully automated identification and clinical classification of macular edema 59

Despite the satisfactory results that were obtained using the machine learning strategy, we implemented a postprocessing approach using an aggregation factor (d) to rectify the misclassifications that were originated by the presence of artifacts or pathological structures. As result, the presented method achieves an F-measure of 87.54% with an aggregation factor of d ¼ 72 for the NB classifier and an F-measure of 85.22% with an aggregation factor of d ¼ 21 for the SVM classifier. In this experiment, we can observe a significant improvement in the results provided by both classifiers using the aggregation factor. In Fig. 17, we can observe an OCT scan with the automatic identification of the three types of ME, simultaneously. In this way, the presented system facilitates the identification of each ME type, providing a set of useful information that can be used in early diagnosis and treatment of different ocular disorders, improving the life quality of the patients.

4 Conclusions DME is a leading cause of reversible blindness in people with diabetes that constitutes a concerning global health problem. This relevant eye disease is defined by the abnormal presence of fluid regions, also known as MEs, within of the retinal tissues that significantly decrease the visual acuity of the patient. OCT capture devices provide a set of images that can permit an easy visualization and identification of different retinal structures and pathological conditions, including ME disease. Based on this medical imaging modality, Otani et al. [37] proposed a classification of MEs into three different pathological categories: CME, SRD, and DRT. For that, the authors used different characteristics and clinical properties that are visible in OCT scans. In this chapter, we expose and detail the fundamentals of an automatic system for the identification and clinical characterization of MEs using OCT scans. For this purpose, the search space of these MEs is first restricted. In particular, four main retinal layer boundaries are identified: ILM, RPE, ISOS, and OPL. The segmentation of these retinal layer boundaries allows a precise delimitation of the retina and its subsequent subdivision in the inner and outer regions. In particular, the SRD and CME edemas present similar properties, such as intensity and contrast with respect to the surrounding tissues. In this way, we used two similar strategies for the localization of these ME types. Both strategies were based on the adaptive thresholding method and were complemented using different clinical criterions to remove false candidate identifications. Regarding the presence of DRT edemas, a machine learning strategy was used for the localization of this ME type. This strategy was adopted because this ME type does not present well-defined limiting membranes or significant contrast with the surrounding tissues. In addition, the significant variability in terms of patterns (form and texture) makes the localization of this ME a more complex scenario. For the identification process of this ME type, a complete set of 18 features were obtained, including intensity image analysis, GLIH, and mask thickness analysis and texture analysis (GLCM) features. To obtain the optimal subset of features that describes the DRT edemas, three different feature selectors were applied: SFS, SVM-FS,

60

Diabetes and Retinopathy

and RFS. Then, using the most relevant subset of features, the NB and SVM classifiers were trained using the optimized parameters, testing the potential of the presented system. And finally, we employed a postprocessing step that uses an aggregation factor (d) to rectify the possible misclassifications that are originated by the presence of artifacts and retinal structures. The validation of the presented method was performed using 170 OCT scans of different patients, labeled by a clinical expert. Regarding the presence of SRD edemas, the tool accurately identified all pathological cases that were included in the analyzed dataset. The system also provided positive results with an F-measure of 91.99% for the CME identifications. Regarding the DRT edema, the best configuration was achieved using the NB classifier combined with the RFS feature selector, reaching an 87.54% of F-measure. Consequently, the presented system has demonstrated its effectiveness for the precise identification and localization of all the types of ME, even when they are combined in the same region of the retina. Thus, it provides a fully automated tool to help clinical experts in the diagnosis and monitoring of this relevant ocular disease. Summarizing, this chapter presents the first proposed computational system capable of accurately identifying and distinguishing the presence of all ME types in OCT scans, bridging a gap in the medical diagnostic and treatment field and helping the clinical specialists in this complicated and tedious task of identification of the ME cases as well as improve the life quality of the individuals once this disease is earlier diagnosed.

Acknowledgments This work is supported by the Instituto de Salud Carlos III, Government of Spain and FEDER funds of the European Union through the DTS15/00153 and DTS18/00136 research projects and by the Ministerio de Ciencia, Innovacio´n y Universidades, Government of Spain through the DPI2015-69948-R and RTI2018095894-B-I00 research projects. Also, this work has received financial support from the European Union (European Regional Development Fund [ERDF]) and the Xunta de Galicia, Centro singular de investigacio´n de Galicia accreditation 2016–19, Ref. ED431G/01; and Grupos de Referencia Competitiva, Ref. ED431C 2016-047.

References [1] G. Dougherty, Image analysis in medical imaging: recent advances in selected examples, Biomed. Imaging Interv. J. 6 (3) (2010) e32. [2] B. Zheng, Y.-H. Chang, D. Gur, Computerized detection of masses from digitized mammograms: comparison of single-image segmentation and bilateral-image subtraction, Acad. Radiol. 2 (12) (1995) 1056–1061. [3] M.L. Giger, C.J. Vyborny, R.A. Schmidt, Computerized characterization of mammographic masses: analysis of spiculation, Cancer Lett. 77 (2–3) (1994) 201–211. [4] Y. Abe, K. Hanai, M. Nakano, Y. Ohkubo, T. Hasizume, T. Kakizaki, M. Nakamura, N. Niki, K. Eguchi, T. Fujino, et al., A computer-aided diagnosis (CAD) system in lung cancer screening with computed tomography, Anticancer Res. 25 (1B) (2005) 483–488.

Chapter 3 • Fully automated identification and clinical classification of macular edema 61

[5] J. Novo, J. Rouco, A. Mendonc¸a, A. Campilho, Reliable lung segmentation methodology by including juxtapleural nodules, in: International Conference Image Analysis and Recognition, Springer, 2014, pp. 227–235. [6] K. Selvanayaki, M. Karnan, CAD system for automatic detection of brain tumor through magnetic resonance image—a review, Int. J. Eng. Sci. Technol. 2 (10) (2010) 5890–5901. [7] E. Dandıl, M. C ¸ akırog˘lu, Z. Eks¸ i, Computer-aided diagnosis of malign and benign brain tumors on MR images, in: ICT Innovations 2014, Springer, 2015, pp. 157–166. [8] M.G. Linguraru, S. Zhao, R.L. Van Uitert, J. Liu, J.G. Fletcher, A. Manduca, R.M. Summers, CAD of colon cancer on CT colonography cases without cathartic bowel preparation, in: Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE, IEEE, 2008, pp. 2996–2999. [9] Y. Mori, S.-E. Kudo, K. Wakamura, M. Misawa, Y. Ogawa, M. Kutsukawa, T. Kudo, T. Hayashi, H. Miyachi, F. Ishida, Novel computer-aided diagnostic system for colorectal lesions by using endocytoscopy (with videos), Gastrointest. Endosc. 81 (3) (2015) 621–629. [10] K. Doi, Computer-aided diagnosis in medical imaging: historical review, current status and future potential, Comput. Med. Imaging Graph. 31 (4–5) (2007) 198–211. [11] A. Ferna´ndez, M. Ortega, J. de Moura, J. Novo, M.G. Penedo, Detection of reactions to sound via gaze and global eye motion analysis using camera streaming, Mach. Vis. Appl. 29 (7) (2018) 1069–1082. [12] A. Ferna´ndez, J. de Moura, M. Ortega, M.G. Penedo, Detection and characterization of the scleraevaluation of eye gestural reactions to auditory stimuli, in: VISAPP (2), 2015, pp. 313–320. [13] B.M. Ter Haar Romeny, K.J. Zuiderveld, P.F.G.M. van Waes, T. van Walsum, R. van der Weijden, J. Weickert, R. Stokking, O. Wink, S. Kalitzin, T. Maintz, et al., Advances in three-dimensional diagnostic radiology, J. Anat. 193 (3) (1998) 363–371. [14] K. Doi, Diagnostic imaging over the last 50 years: research and development in medical imaging science and technology, Phys. Med. Biol. 51 (13) (2006) R5. [15] M. Dı´az, J. Novo, M.G. Penedo, M. Ortega, Automatic extraction of vascularity measurements using OCT-A images, Procedia Comput. Sci. 126 (2018) 273–281. [16] J. de Moura, J. Novo, M. Ortega, N. Barreira, P. Charlo´n, Artery/vein classification of blood vessel tree in retinal imaging, in: VISIGRAPP (4: VISAPP), 2017, pp. 371–377. [17] S. Baamonde, J. de Moura, J. Novo, J. Rouco, M. Ortega, Feature definition and selection for epiretinal membrane characterization in optical coherence tomography images, in: International Conference on Image Analysis and Processing, Springer, 2017, pp. 456–466. [18] J. Jiang, P. Trundle, J. Ren, Medical image analysis with artificial neural networks, Comput. Med. Imaging Graph. 34 (8) (2010) 617–631. [19] J. de Moura, J. Novo, J. Rouco, M.G. Penedo, M. Ortega, Automatic detection of blood vessels in retinal OCT images, in: International Work-Conference on the Interplay Between Natural and Artificial Computation, Springer, 2017, pp. 3–10. [20] Z. Zhang, R. Srivastava, H. Liu, X. Chen, L. Duan, D.W.K. Wong, C.K. Kwoh, T.Y. Wong, J. Liu, A survey on computer aided diagnosis for ocular diseases, BMC Med. Inform. Decis. Mak. 14 (1) (2014) 80. [21] J. Novo, M.G. Penedo, J. Santos, Optic disc segmentation by means of GA-optimized topological active nets, in: International Conference Image Analysis and Recognition, Springer, 2008, pp. 807–816. [22] L.F. Porter, G.C.M. Black, Personalized ophthalmology, Clin. Genet. 86 (1) (2014) 1–11. [23] J. Novo, J. Rouco, N. Barreira, M. Ortega, M.G. Penedo, A. Campilho, Wivern: a web-based system enabling computer-aided diagnosis and interdisciplinary expert collaboration for vascular research, J. Med. Biol. Eng. 37 (6) (2017) 920–935.

62

Diabetes and Retinopathy

[24] J. Novo, A. Hermida, M. Ortega, N. Barreira, M.G. Penedo, J.E. Lo´pez, C. Calvo, Hydra: a web-based system for cardiovascular analysis, diagnosis and treatment, Comput. Methods Programs Biomed. 139 (2017) 61–81. [25] M. Pena-Seijo, M.G. Penedo, M.O. Hortas, S. Gonza´lez, F.G.-U. Irazazabal, A. Pose-Reino, Sirius, computerized tool for automatic analysis of retinal microcirculation:PP.7.183, J. Hypertens. 29 (2011) e202. [26] J. de Moura, J. Novo, M. Ortega, N. Barreira, M.G. Penedo, Vessel tree extraction and depth estimation with OCT images, in: Conference of the Spanish Association for Artificial Intelligence, Springer, 2016, pp. 23–33. [27] M. Adhi, J.S. Duker, Optical coherence tomography-current and future applications, Curr. Opin. Ophthalmol. 24 (3) (2013) 213. [28] A. Gonza´lez, M.G. Penedo, S. Va´zquez, J. Novo, P. Charlo´n, Cost function selection for a graph-based segmentation in OCT retinal images, in: International Conference on Computer Aided Systems Theory, Springer, 2013, pp. 125–132. [29] S.A. Boppart, M.E. Brezinski, J.G. Fujimoto, Optical coherence tomography imaging in developmental biology, in: Developmental Biology Protocols, Springer, 2000, pp. 217–233. [30] J.G. Fujimoto, C. Pitris, S.A. Boppart, M.E. Brezinski, Optical coherence tomography: an emerging technology for biomedical imaging and optical biopsy, Neoplasia 2 (1–2) (2000) 9–25. [31] S. Baamonde, J. de Moura, J. Novo, M. Ortega, Automatic detection of epiretinal membrane in OCT images by means of local luminosity patterns, in: International Work-Conference on Artificial Neural Networks, Springer, 2017, pp. 222–235. [32] J de Moura, J. Novo, P. Charlo´n, N. Barreira, M. Ortega, Enhanced visualization of the retinal vasculature using depth information in OCT, Med. Biol. Eng. Comput. 55 (12) (2017) 2209–2225. [33] P.L. Vidal, J de Moura, J. Novo, M.G. Penedo, M. Ortega, Intraretinal fluid identification via enhanced maps using optical coherence tomography images, Biomed. Opt. Express 9 (10) (2018) 4730–4754. [34] G. Samagaio, J. de Moura, J. Novo, M. Ortega, Optical coherence tomography denoising by means of a Fourier Butterworth filter-based approach, in: International Conference on Image Analysis and Processing, Springer, 2017, pp. 422–432. [35] R.R.A. Bourne, S.R. Flaxman, T. Braithwaite, M.V. Cicinelli, A. Das, J.B. Jonas, J. Keeffe, J.H. Kempen, J. Leasher, H. Limburg, et al., Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis, Lancet Glob. Health 5 (9) (2017) e888–e897. [36] P. Romero-Aroca, Managing diabetic macular edema: the leading cause of diabetes blindness, World J. Diabetes 2 (6) (2011) 98. [37] T. Otani, S. Kishi, Y. Maruyama, Patterns of diabetic macular edema with optical coherence tomography, Am. J. Ophthalmol. 127 (6) (1999) 688–693. [38] G. Panozzo, B. Parolini, E. Gusson, A. Mercanti, S. Pinackatt, G. Bertoldo, S. Pignatto, Diabetic macular edema: an OCT-based classification, Semin. Ophthalmol. 19 (1–2) (2004) 13–20. [39] B. Hassan, G. Raja, T. Hassan, M.U. Akram, Structure tensor based automated detection of macular edema and central serous retinopathy using optical coherence tomography images, J. Opt. Soc. Am. A 33 (4) (2016) 455–463. [40] T. Schlegl, S.M. Waldstein, H. Bogunovic, F. Endstraßer, A. Sadeghipour, A.-M. Philip, D. Podkowinski, B.S. Gerendas, G. Langs, U. Schmidt-Erfurth, Fully automated detection and quantification of macular fluid in OCT using deep learning, Ophthalmology 125 (4) (2018) 549–558. [41] D. Lu, M. Heisler, S. Lee, G. Ding, M.V. Sarunic, M.F. Beg, Retinal fluid segmentation and detection in optical coherence tomography images using fully convolutional neural network, ArXiv preprint arXiv:1710.04778 (2017).

Chapter 3 • Fully automated identification and clinical classification of macular edema 63

[42] G.N. Girish, B. Thakur, S.R. Chowdhury, A.R. Kothari, J. Rajan, Segmentation of intra-retinal cysts from optical coherence tomography images using a fully convolutional neural network model, IEEE J. Biomed. Health Inform. 23 (1) (2018) 296–304. [43] A. Rashno, D.D. Koozekanani, P.M. Drayna, B. Nazari, S. Sadri, H. Rabbani, K.K. Parhi, Fully automated segmentation of fluid/cyst regions in optical coherence tomography images with diabetic macular edema using neutrosophic sets and graph algorithms, IEEE Trans. Biomed. Eng. 65 (5) (2018) 989–1001. [44] J. de Moura, J. Novo, M. Ortega, P. Charlo´n, 3D retinal vessel tree segmentation and reconstruction with OCT images, in: International Conference Image Analysis and Recognition, Springer, 2016, pp. 716–726.
, D.E. Olson, T.Q. Duong, Structural [45] H. Cheng, G. Nair, T.A. Walker, M.K. Kim, M.T. Pardue, P.M. Thule and functional MRI reveals multiple retinal layers, Proc. Natl. Acad. Sci. 103 (46) (2006) 17525–17530.
vez, J. de Moura, J. Novo, M.I. Fernandez, M. Ortega, Automatic macular edema [46] G. Samagaio, A. Este identification and characterization using OCT images, Comput. Methods Programs Biomed. 163 (2018) 47–63. [47] S.J. Chiu, X.T. Li, P. Nicholas, C.A. Toth, J.A. Izatt, S. Farsiu, Automatic segmentation of seven retinal layers in SDOCT images congruent with expert manual segmentation, Opt. Express 18 (18) (2010) 19413–19428. [48] D.B. West, et al., Introduction to Graph Theory, vol. 2, Prentice hall, Upper Saddle River, NJ, 2001. [49] R. Howard, Dynamic Programming and Markov Processes, John Wiley, New York, NY, 1964. [50] E.W. Dijkstra, A note on two problems in connexion with graphs, Numer. Math. 1 (1) (1959) 269–271. [51] M.R. Hee, C.A. Puliafito, C. Wong, J.S. Duker, E. Reichel, B. Rutledge, J.S. Schuman, E.A. Swanson, J. G. Fujimoto, Quantitative assessment of macular edema with optical coherence tomography, Arch. Ophthalmol. 113 (8) (1995) 1019–1029. [52] A. Mehnert, P. Jackway, An improved seeded region growing algorithm, Pattern Recogn. Lett. 18 (10) (1997) 1065–1071. [53] M. Shahidi, Z. Wang, R. Zelkha, Quantitative thickness measurement of retinal layers imaged by optical coherence tomography, Am. J. Ophthalmol. 139 (6) (2005) 1056–1061. [54] J. Moura, J. Novo, J. Rouco, M. Penedo, M. Ortega, Automatic identification of intraretinal cystoid regions in optical coherence tomography, in: Conference on Artificial Intelligence in Medicine in Europe, Springer, 2017, pp. 305–315. [55] J de Moura, J. Novo, J. Rouco, M. Penedo, M. Ortega, Automatic vessel detection by means of brightness profile characterization in OCT images, Procedia Comput. Sci. 112 (C) (2017) 980–988. [56] J. de Moura, J. Novo, M. Ortega, N. Barreira, M.G. Penedo, Interactive three-dimensional visualization system of the vascular structure in OCT retinal images, in: International Conference on Computer Aided Systems Theory, Springer, 2017, pp. 306–313.
vez, J. de Moura, J. Novo, M. Ortega, M.I. Ferna´ndez, Automatic identification of [57] G. Samagaio, A. Este macular edema in optical coherence tomography images, in: VISIGRAPP (4: VISAPP), 2018, pp. 533–540. [58] J de Moura, P.L. Vidal, J. Novo, J. Rouco, M. Ortega, Feature definition, analysis and selection for cystoid region characterization in optical coherence tomography, Procedia Comput. Sci. 112 (2017) 1369–1377. € nte, J. Flammer, Choroidal capillary and venous congestion in central serous chorioretinopa[59] C. Pru thy, Am. J. Ophthalmol. 121 (1) (1996) 26–34. [60] T. Iida, N. Hagimura, T. Sato, S. Kishi, Evaluation of central serous chorioretinopathy with optical coherence tomography, Am. J. Ophthalmol. 129 (1) (2000) 16–20.

64

Diabetes and Retinopathy

[61] D. Gaucher, C. Sebah, A. Erginay, B. Haouchine, R. Tadayoni, A. Gaudric, P. Massin, Optical coherence tomography features during the evolution of serous retinal detachment in patients with diabetic macular edema, Am. J. Ophthalmol. 145 (2) (2008) 289–296. [62] J. de Moura, J. Novo, S. Penas, M. Ortega, J. Silva, A.M. Mendonc¸a, Automatic characterization of the serous retinal detachment associated with the subretinal fluid presence in optical coherence tomography images, Procedia Comput. Sci. 126 (2018) 244–253. [63] F. Yan, H. Zhang, C.R. Kube, A multistage adaptive thresholding method, Pattern Recogn. Lett. 26 (8) (2005) 1183–1191. [64] S. Ooto, A. Tsujikawa, S. Mori, H. Tamura, K. Yamashiro, N. Yoshimura, Thickness of photoreceptor layers in polypoidal choroidal vasculopathy and central serous chorioretinopathy, Graefes Arch. Clin. Exp. Ophthalmol. 248 (8) (2010) 1077–1086. [65] J.M. Gelfand, R. Nolan, D.M. Schwartz, J. Graves, A.J. Green, Microcystic macular oedema in multiple sclerosis is associated with disease severity, Brain 135 (6) (2012) 1786–1793. [66] B.S. Fine, A.J. Brucker, Macular edema and cystoid macular edema, Am. J. Ophthalmol. 92 (4) (1981) 466–481. [67] R.B. Nussenblatt, S.C. Kaufman, A.G. Palestine, M.D. Davis, F.L. Ferris, Macular thickening and visual acuity: measurement in patients with cystoid macular edema, Ophthalmology 94 (9) (1987) 1134–1139. [68] M. Yanoff, B.S. Fine, A.J. Brucker, R.C. Eagle Jr, Pathology of human cystoid macular edema, Surv. Ophthalmol. 28 (1984) 505–511. [69] M. Ota, A. Tsujikawa, T. Murakami, N. Yamaike, A. Sakamoto, Y. Kotera, K. Miyamoto, M. Kita, N. Yoshimura, Foveal photoreceptor layer in eyes with persistent cystoid macular edema associated with branch retinal vein occlusion, Am. J. Ophthalmol. 145 (2) (2008) 273–280. [70] H.K. Hahn, H.-O. Peitgen, IWT-interactive watershed transform: a hierarchical method for efficient interactive and automated segmentation of multidimensional gray-scale images, in: Medical Imaging 2003: Image Processing, vol. 5032, International Society for Optics and Photonics, 2003, pp. 643–654. [71] J.D.M. Gass, E.W.D. Norton, Cystoid macular edema and papilledema following cataract extraction: a fluorescein fundoscopic and angiographic study, Arch. Ophthalmol. 76 (5) (1966) 646–661. [72] B. Wolff, G. Azar, V. Vasseur, J.A. Sahel, C. Vignal, M. Mauget-Fay¨sse, Microcystic changes in the retinal internal nuclear layer associated with optic atrophy: a prospective study, J. Ophthalmol. 2014 (2014) 1–5. [73] Y.M. Helmy, H.R.A. Allah, Optical coherence tomography classification of diabetic cystoid macular edema, Clin. Ophthalmol. (Auckland, NZ) 7 (2013) 1731. [74] W. Goebel, T. Kretzchmar-Gross, Retinal thickness in diabetic retinopathy: a study using optical coherence tomography (OCT), Retina 22 (6) (2002) 759–767. [75] M. Shahidi, Y. Ogura, N.P. Blair, M.M. Rusin, R. Zeimer, Retinal thickness analysis for quantitative assessment of diabetic macular edema, Arch. Ophthalmol. 109 (8) (1991) 1115–1119. [76] B.Y. Kim, S.D. Smith, P.K. Kaiser, Optical coherence tomographic patterns of diabetic macular edema, Am. J. Ophthalmol. 142 (3) (2006) 405–412. [77] G. Samagaio, J de Moura, J. Novo, M. Ortega, Automatic segmentation of diffuse retinal thickening edemas using optical coherence tomography images, Procedia Comput. Sci. 126 (2018) 472–481. [78] R.M. Haralick, K. Shanmugam, et al., Textural features for image classification, IEEE Trans. Syst. Man Cybern. (6) (1973) 610–621. [79] W. Siedlecki, J. Sklansky, On automatic feature selection, in: Handbook of Pattern Recognition and Computer Vision, World Scientific, Singapore, 1993, pp. 63–87.

Chapter 3 • Fully automated identification and clinical classification of macular edema 65

[80] J. Bi, K. Bennett, M. Embrechts, C. Breneman, M. Song, Dimensionality reduction via sparse support vector machines, J. Mach. Learn. Res. 3 (2003) 1229–1243. [81] F. Nie, H. Huang, X. Cai, C.H. Ding, Efficient and robust feature selection via joint l2, 1-norms minimization, in: Advances in Neural Information Processing Systems, 2010, pp. 1813–1821. [82] N. Friedman, D. Geiger, M. Goldszmidt, Bayesian network classifiers, Mach. Learn. 29 (2–3) (1997) 131–163. [83] C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn. 20 (3) (1995) 273–297. € ller, G.J. Jaffe, U. Schmidt-Erfurth, Differ[84] M.R. Munk, L.M. Jampol, C. Simader, W. Huf, T.J. Mittermu entiation of diabetic macular edema from pseudophakic cystoid macular edema by spectral-domain optical coherence tomography, Invest. Ophthalmol. Vis. Sci. 56 (11) (2015) 6724–6733.

Joaquim de Moura received his degree in Computer ˜a Engineering in 2014 from the University of A Corun (Spain). In 2016, he received his MSc degree in Computer Engineering from the same university. He is currently pursuing his PhD in Computer Science in a collaborative project between ophthalmology centers in Galicia and the ˜ a. His research areas of interest are University of A Corun computer vision, pattern recognition, machine learning algorithms, and biomedical imaging processing of various kinds.

Gabriela Samagaio obtained the MSc degree in Bioengineering at Faculty of Engineering at the University of Porto (FEUP) in 2018. In 2017 and 2018, she enrolled a curricular research Erasmus internship at the Varpa Research Group, which is affiliated to the Department of Computer Science, Faculty of Informatics, University of ˜ a. Her research interest includes computer vision, A Corun machine learning algorithms, and biomedical imaging processing of various kinds.

66

Diabetes and Retinopathy

Jorge Novo received his MSc and PhD degrees (cum Laude) ˜ a in in Computer Science from the University of A Corun 2007 and 2012, respectively. He has also worked, as a visiting researcher, with CMR images in the detection of landmark points at the Imperial College London and as a postdoctoral research fellow at the INEB and INESC-TEC research institutes in the development of CAD systems for lung cancer diagnosis with chest CT images. His main research interests lie in the fields of computer vision, pattern recognition, and biomedical image processing.

Marı´a Isabel Ferna´ndez received her MD degree in Medicine and Surgery in 1986 and her PhD degree cum Laude in 1990. She is currently an ophthalmologist consultant in the University Hospital Complex of Santiago de Compostela and an associate professor in the Health Sciences Department in the area of ophthalmology at the University of Santiago de Compostela. Her research areas of interest lie in the Medical Retina, specializing in applying diagnostic imaging to the knowledge and treatment of the retinal diseases, combining it as an associate professor and as a researcher. Publications: 39 articles, in national and international magazines. Author/coauthor of 24 chapters of scientific books, taking part in the creation of the SERV guides about retinal vein occlusions, central serous chorioretinopathy, and uveal melanoma. More than 100 communications or oral presentations in congresses, instructional courses, round table discussions, seminars, and other national and international scientific meetings, being first researcher or collaborator in more than 50 clinical trials and research projects.

Chapter 3 • Fully automated identification and clinical classification of macular edema 67

Dr. Francisco Go´mez-Ulla is an eminence in Ophthalmology. Founder and Medical Director of the Instituto Oftalmolo´gico Go´mez-Ulla, Professor of Ophthalmology at the University of Santiago de Compostela (USC) and Head of the Ophthalmology Service of the CHUS, has more than 40 years of experience and extensive training in different fields of the specialty. In this sense, it stands out in the treatment of retinal detachment, DMAE, diabetic retinopathy, diabetic macular pathology, retinal vascular pathology, floating bodies and flashes, high myopia, cataract surgery, and presbyopia surgery. Dr. Go´mez-Ulla has received the most prestigious awards in his specialty, such as the awards Arruga, Castroviejo, Professor Barea, is Best Doctor Spanish 2011, Bausch & Lomb award to the best Spanish retinologist 2012, Galician prize of the year 2015, and award to the “professional prestige” 2018 by the magazine Ejecutivos.

Marcos Ortega received his MSc degree in Computer Science in 2004 and his PhD degree cum Laude in 2009. He also worked on face biometrics studying the face evolution due to aging effects as a visiting researcher in the University of Sassari and methods for age estimation under different facial expression conditions as a visiting postdoctoral fellow in the University of Amsterdam. He is currently an Associate Professor in the Department of ˜ a. His Computer Science at the University of A Corun research areas of interest are medical image analysis, computer vision, biometrics, and human behavior analysis.

4

Optimal surface segmentation with subvoxel accuracy in spectral domain optical coherence tomography images Abhay Shaha, Michael D. Abra`moffa,b,c, Xiaodong Wua,d D EP A RTM E NT O F ELE CT RICA L AND CO M PUT ER E N GI N EE RIN G , UNI VE RS IT Y OF IO WA, IO WA CI TY , I A, UNIT ED S TATE S b DEPAR TMENT OF BIO MEDI CAL E NGINE ERING, UNIVERSITY O F IO WA, IO WA CI TY , I A , U N IT ED ST A TES c DEPARTMENT OF OP HTHALMOLOGY AND V ISUAL SCIENCES, C AR VER COLLEGE OF MEDICINE, UNIVERSITY O F I OWA, IOWA CITY, I A, UNITED S TA TE S d DEPARTMENT OF RADI ATION O NCOL OGY, UNIVERSITY O F IOWA, IOWA CITY, I A, UNI TE D STAT ES a

1 Introduction Accurate segmentation of surfaces and object boundaries in medical images is quintessential for quantitative analysis, diagnosis, and management of diseases. Popular graph-based segmentation approaches: graph search [1] and graph cut [2, 3] have been extensively used for various such image segmentation applications. In each of these methods, the segmentation task is converted to an energy minimization problem and the respective constraints with respect to the target segmentation is modeled into the energy terms. These state-ofthe-art segmentation techniques employ a flexible and robust framework, which allows for extension of these methods to encode various user-defined constraints based on the segmentation task while ensuring global optimality of the solution. The advantages of using such a formulation are: (1) flexible modeling ability to incorporate various terms like likelihoods, neighbor relationships, prior information, context information, and surface/object interaction; (2) graph-based methods provide a framework for computing efficient solutions for a large variety of image segmentation problems; and (3) readily extensible to segment surfaces in an N(N
2)-dimensional space. The pioneer work, optimal net surface segmentation method [4], first introduced a graph-based framework to segment multiple surfaces in the N-D space (N
2) in polynomial time. The method was built upon a key observation based on the voxel column structure in digital volumetric images. The method constructed multicolumn graphs from images, wherein each voxel corresponded to a node in the graph space. The algorithm Diabetes and Retinopathy. https://doi.org/10.1016/B978-0-12-817438-8.00004-3 © 2020 Elsevier Inc. All rights reserved.

69

70

Diabetes and Retinopathy

constructs edges between the graph nodes to encode and model the target surface and prior information, while the resulting resultant target surfaces are obtained by computing the minimum st-cut [5] over the constructed graph. Global optimality with respect to the objective function employed is guaranteed by the method. The popular optimal surface segmentation method (graph search) [1] demonstrated the applicability and effectiveness of the optimal net surface segmentation method. In this work, the terms graph search and the optimal surface segmentation method shall be used interchangeably, but both refer to the same method [1]. The state-of-the-art method has been extensively used for various medical image segmentation tasks, such as knee bone and cartilage [6, 7], heart [8, 9], airways and vessels trees [10, 11], lungs [12], liver [13], prostate and bladder [14], and retinal surfaces [15–17]. The method has also been shown to segment complex surface topologies by additionally including various prior information of the target object [18, 19]. Spectral domain optical coherence tomography (SD-OCT) [20, 21] is widely used in imaging of the retina because of its capability to extract cross-sectional information of the retina. The segmentation of retinal layers in OCT plays a key role in assessing layer thicknesses, which is beneficial in assessing various disease states. However, manual tracings of the retinal layers are time consuming and result in substantial intra/interobserver variability. Automated methods for layer segmentation in OCT volumes overcome such issues. A plethora of methods has been studies and developed to automatically segment retinal surfaces in OCT volumes. Intensity-based surface segmentation methods have been developed, utilizing correlation between adjacent A-scans [22], Canny edge detectorbased segmentation [23, 24], and exploitation of prior information from retinal vessels by using an iterative thresholding technique [25]. These methods have major limitations—sensitivity to noise because of minimal use of textural information, blood vessel shadows, and motion artifacts. The next group of common techniques used for retinal surface segmentation is active contour-based methods [26–28], shape model-based methods [29, 30], and machinelearning-based methods [31–33]. These methods suffer from the following limitations— requirement of prior information to design various energy function terms, hand-crafted feature extraction, limited flexibility for transferability to new data and different applications since the terms are designed for a specific target application, and is not fully automated. The next group of methods is graph based. Herein, each voxel in the graph is transformed into a graph node and then traditional graph algorithms like Dijakstra’s method and dynamic programming are used to find the shortest path based on the node costs encoded in the graph nodes to segment the target surfaces. Refs. [34–37] show the application of such methods in segmentation of retinal surfaces in B-scans of OCT volumes. The limitations for these methods include—handcrafted careful design of various transformations, no utilization of three-dimensional (3D) information, and user-determined initialization of the starting points for the shortest path computation.

Chapter 4 • Optimal surface segmentation with subvoxel accuracy in SD-OCT

71

State-of-the-art graph search (optimal surface segmentation) [1, 4] has been applied in a variety of ways for retinal surface segmentation in OCT volumes. The method was applied to segmentation of multiple retinal surfaces in macular OCTs [15] but suffered from computation time limitations due to exponential growth in size of the graph with respect to image size and number of target surfaces. To further increase computational efficiency, multiresolution graph search technique [16] to accelerate the processing time. The method was applied to segment of multiple retinal surfaces in optic nerve head (ONH) OCT scans. The method was further developed to incorporate prior target surface information in the graph construction to better model the constraints and the cost images used in the graph construction through machine learning [18, 19, 38, 39]. The method was shown to have superior accuracy in segmentation of surfaces in OCT images of humans, mice, and canines. The advantage of incorporating the prior information within the graph construction constraints allows for both the global and local features inclusion, thus resulting in higher accuracy and robustness. By adding the prior information, the graph-theoretic algorithm has added the ability to entirely consider both global and local priors and achieves better robustness. In digital image processing, an orthogonal matrix of pixel (voxel) intensities is generally used to represent image data. Therefore, the accuracy of a surface segmentation is limited to a single unit voxel precision, which is equivalent to the distance between two adjoining nodes in the graph space. However, partial volume effects in the image volumes can be exploited to attain subvoxel accuracy (accuracy higher than a single unit voxel) [40, 41]. One of the ways to utilize the partial volume effect is by computing a gradient vector flow (GVF), resulting in an irregularly sampled space and nonequidistant spacing between adjoining graph nodes. Subvoxel accurate centers for each voxel can be determined by computing a displacement field directly from the image data [40] by utilizing the previously ignored partial volume information. The resultant irregularly sampled space requires a generalized version of the graph construction to accommodate nonequidistant spacing between orthogonal adjoining nodes. The subvoxel accurate graph search method [40] is extension of the graph search method to segment multiple surfaces in images by utilizing additional information from the partial volume effects at the voxel level by computing a displacement field from the image data. The method uses the conventional optimal surface segmentation method [1] to create the base graph, followed by computation of a displacement field from the input image to locate the subvoxel accurate voxel centers and adjusts the edges to model the surface constraints in the deformed graph space. Such a deformation results in nonequidistant spacing between the adjoining nodes. This is equivalent to nonuniform sampling across the axial direction in a volume cube for 3D surface segmentation purposes. The method was shown to achieve subvoxel accuracy for surface segmentation. An example is shown in Fig. 1. However, the method was designed to only employ hard surface constraints (fixed number of voxels tolerance for surface smoothness and surface separation constraints), which limits the flexibility in constraining surfaces with various other types of constraints. Specifically, the previous approach was not capable of

72

Diabetes and Retinopathy

(A)

(B)

FIG. 1 A 33 voxel grid example showing subvoxel accuracy. The red node represents a voxel in the graph space. (A) Equidistant graph nodes spacing. Subvoxel accurate segmentation truth is shown in green and result from traditional graph search method is shown in yellow. (B) Brown arrows indicate the displacement of the nodes using a derived displacement field. Segmentation using the subvoxel accurate graph search is shown in blue.

incorporating a convex surface smoothness constraint in the graph with nonequidistant spacing between adjoining nodes. We extend the subvoxel accurate graph search method [40] to allow for incorporation of convex surface constraints to segment multiple surfaces with subvoxel accuracy and demonstrate the improved performance on OCT segmentations. The presented method generalizes the optimal surface segmentation method with convex priors [18] to segment surfaces in irregularly sampled space. The use of convex priors allows for incorporation of many different prior information in the graph framework as discussed previously while attaining subvoxel accuracy. The proposed method does not require a two-step approach of graph construction followed by edge adjustments based on the displacement field deformations as used in Ref. [40], but instead provides an elegant single shot method to construct the graph and the corresponding edges in an irregularly sampled space while incorporation convex priors. We demonstrate the performance of the proposed method on SD-OCT retinal volumes for subvoxel and super-resolution segmentation accuracy compared to traditional graph search method with convex priors [18].

2 Methods 2.1 Problem formulation and energy function 2.1.1 Original formulation in regularly sampled space Denote a cube volume I(x, y, z) of size X  Y  Z. A surface is modeled as a function S(x, y), where xx ¼ f0, 1, …,X  1g, yy ¼ f0, 1, …,Y  1g, and Sðx, yÞz ¼ f0,1, …, Z  1g. A column of voxels fðIðx, y,zÞjz ¼ 0, 1, …,Z  1g is represented by col(x, y) parallel to axial direction, that is, the z-axis; corresponding to each (x, y)-pair. Two neighboring (x, y)-pairs

Chapter 4 • Optimal surface segmentation with subvoxel accuracy in SD-OCT

73

in the image domain x  y is denoted by p and q. Ns is the neighborhood setting in the image domain. Therefore, S( p) is a labeling for col( p) with label set z (S( p) z). For each column p, col( p) is intersected by S( p) at a single-voxel location. The goal of the method is to find λ (λ
1) surfaces Si(x, y), i ¼ 1,2, …,λ in I with global optimality with respect to prior information and surface constraints. The following cost functions are employed for the formulation: •

•

•

The data cost, that is, the total cost of all voxels on the target segmented surface Si is P modeled as pxyDi(Si( p)). The data cost function directly corresponds to the inverse probability of a voxel belonging to the target surface. The surface smoothness which controls the feasibility of the surface positions with P respect to two neighboring columns, modeled as ðp,qÞNs Vpq ðSi ðpÞ,Si ðqÞÞ (Eq. 1). The surface smoothness term herein is modeled as a hard constraint and specifies the maximum possible difference between surface positions of two neighboring columns. The surface separation constraint which governs the feasible minimum and maximum separation between adjacent surfaces, modeled as Hp(Si( p), Si+1( p)) (Eq. 2). ( Vpq ðSi ðpÞ,Si ðqÞÞ ¼

∞, if jSi ðpÞ  Si ðqÞj > Δpq , 0 , otherwise

(1)

where Δpq is the surface smoothness hard constraint. 8 > < ∞, if ðSi + 1 ðpÞ  Si ðpÞÞ < δ min , Hp ðSi ðpÞ,Si + 1 ðpÞÞ ¼ ∞, if ðSi + 1 ðpÞ  Si ðpÞÞ > δ max , > : 0 , otherwise

(2)

where δ min and δ max are the minimum and maximum surface separation constraint between surfaces Si and Si+1. Example of such hard constraints is shown in Fig. 2.

|Si(p)– Si(q)| ≤ Δpq z = Z–1 . . . . Si(p) . . . z=0 Col (p) p = (x,y)

Δpq

Col (q) q = (x+1,y)

δmin≤ Si+1(p)–Si(p) ≤ δmax z = Z–1 . δmin . Si+1(p) . . δmax . . . z=0 Col (p) p = (x,y)

Col (p) p = (x,y)

FIG. 2 (Left) Neighboring columns p and q, with surface smoothness constraint for surface Si, red arcs show the feasible st-cut. (Right) Corresponding columns with surface separation constraint for surfaces Si, Si+1, red arcs show the feasible st-cut.

74

Diabetes and Retinopathy

Therefore, the following energy function shown in Eq. (3) is optimized for the solution: 0 λ X X @ EðSÞ ¼ Di ðSi ðpÞÞ + i¼1

pxy

X ðp, qÞNs

1 Vpq ðSi ðpÞ, Si ðqÞÞA +

λ1 X X i¼1 pxy

Hp ðSi + 1 ðpÞ, Si ðpÞÞ

(3)

For each surface Si, a subgraph Gi is constructed wherein each node in the subgraph corresponds to a voxel in the image. For λ surfaces, λ subgraphs G1 ,…,Gi ,Gi + 1 ,…, Gλ are constructed. The graph G is the union of the λ subgraphs Gi’s with additional intersurface edges added between the corresponding columns. The target segmentations are solved for by computing a single minimum st-cut on G. The method also allows for flexibility in the constraints. For example, a convex penalty can be imposed on the surface smoothness term as shown in Eq. (4). Herein, the coefficient w is used to control the extent of surface regularization. An example of convex penalty-based surface smoothness constraint is shown in Fig. 3. The surface smoothness term can similarly be modified to incorporate prior information [18, 19]. Vpq ðSi ðpÞ,Si ðqÞÞ ¼ w  ψðSi ðpÞ  Si ðqÞÞ

(4)

where w > 0 is the weight coefficient and ψ(.) is a convex function.

2.1.2 Formulation in irregularly sampled space to achieve subvoxel accuracy

  Denote a volume ~Iðx, y, z Þ, where xx ¼ f0, 1, …, X  1g, yy ¼ f0, 1, …,Y  1g, and z .   A column col(x, y), column fð~Iðx, y, z Þj z  corresponds to each (x, y)-pair. Assume each  col(x, y) has exactly Z elements obtained by sampling strictly in the increasing order along   the z -direction, which are indexed by f0, 1, …, Z 1g along col(x, y). This results in an

FIG. 3 Surface smoothness constraint for two neighboring columns p and q for a surface Si. The arc in red shows a st-cut for a given surface position Si( p) for column p. The smoothness cost for the given cut is w  ψ(Si( p)  Si(q)).

Chapter 4 • Optimal surface segmentation with subvoxel accuracy in SD-OCT

75



image I(x, y, z) of size X  Y  Z , where xx ¼ f0, 1, …, X  1g, yy ¼ f0, 1, …,Y  1g, and    z  z ¼ f0, 1, …, Z 1g. Thus, allowing for nonequidistant spacing between two adjacent  elements in the column. We assume Z ¼ Z for the remainder of this chapter for ease of understanding. A mapping function is defined for each column p as Lp : f0, 1, …, Z  1g !  for mapping   sampled point indexes in I(p, z) to ~Iðp, z Þ. For example, Lp(i) denotes the z -coordinate of the i + 1th sample along column p, and Lp(i + 1) > Lp(i) (strict increasing order sampling along column p). An example is shown in Fig. 4. A surface is modeled as S( p), where  SðpÞz ¼ f0, 1, …,Z  1g. The function Lp(S( p)) defines the “physical” location (the z -coordinate) of surface S at column p. For simultaneously segmenting λ (λ
2) surfaces, the goal of the problem is to seek the surface labeling Si( p) on all columns in I for each surface Si, where i ¼ 1, 2, …,λ, with minimum separation dj, j+1 where j ¼ 1, 2, …,λ  1 between adjacent pair of surfaces. Note that the surfaces are ordered, that is, Lp(Si+1( p))
Lp(Si( p)). The corresponding energy function for this formulation is shown in the following equation: EðSÞ ¼

λ X

X

i¼1

pxy

+

X

Di ðLp ðSi ðpÞÞÞ

Vpq ðLp ðSi ðpÞÞ, Lq ðSi ðqÞÞÞ

(5)

ðp, qÞNs

+

λ1 X X i¼1 pxy

! Hp ðLp ðSi + 1 ðpÞÞ, Lp ðSi ðpÞÞÞ

z = Z–1

.

Irregular sampling

Column transforma on

ǁ

.

z=1

z=0

column(p) in ~

column(p) in Nodes in graph space Mapping of column(p) in to ~

FIG. 4 Column structure example for irregularly sampled space with mapping function.

76

Diabetes and Retinopathy

Herein, the surface smoothness term is modeled as a convex function as shown in the following equation: Vpq ðLp ðSi ðpÞÞ,Lq ðSi ðqÞÞÞ ¼ ψðLp ðSi ðpÞÞ  Lq ðSi ðqÞÞÞ

(6)

where ψ(.) is a convex function, and ψ(0) ¼ 0 [4]. For simplicity, the surface separation term is modeled as a hard constraint and shown in the following equation: ( Hp ðLp ðSi + 1 ðpÞÞ,Lp ðSi ðpÞÞÞ ¼

∞,

if Lp ðSi + 1 ðpÞÞ  Lp ðSi ðpÞÞ < di, i + 1 ,

0,

otherwise

(7)

where di, i+1 is the minimum separation distance between two adjacent surfaces.

2.2 Graph construction A subgraph Gi is constructed for each surface Si. Data cost term is encoded for a given cost volume Di by addition of intracolumn edges. Surface smoothness term Vab(.) is encoded by addition of intercolumn edges between a pair of neighboring columns a and b. Surface separation term is encoded by addition of intersurface edges between corresponding columns of subgraphs Gi and Gi+1. For segmenting λ surfaces, graph G is constructed form the union of subgraphs Gi’s. A minimum st-cut is then computed on G to solve for the target segmentations. The final surface positions for the resultant segmentations are obtained by applying the mapping function La(.). Denote a node ni(a, z) (z z) in Gi to represent each voxel in cost volume Di. Following edges are added in the graph construction:

2.2.1 Intracolumn edges Monotonicity of target surfaces and data term encoding is done by adding intracolumn edges to each subgraph Gi (described in Ref. [1]).

2.2.2 Intercolumn edges A convex function-based surface smoothness term is encoded in the graph by adding intercolumn arcs between neighboring columns p and q to each subgraph Gi. Denote a function operator f(r1, r2) as shown in the following equation: (

f ðr1 ,r2 Þ ¼

0,

if r1 < r2 ,

ψðr1  r2 Þ,

otherwise

(8)

where ψ(.) is a convex function. A function g(.) is used for weight setting of the intercolumn edges. The following intercolumn edges are added: A directed edge with weight g(k1, k2) (Eq. 9) is added from node ni(p, k1) to node ni(q, k2), for all k1  [0, Z  1] and k2  [1, Z  1]. A directed edge is also added from node ni(p, k1) to terminal node t with weight setting g(k1, Z).

Chapter 4 • Optimal surface segmentation with subvoxel accuracy in SD-OCT

77

gðk1 ,k2 Þ ¼f ðLp ðk1 Þ, Lq ðk2  1ÞÞ  f ðLp ðk1  1Þ, Lq ðk2  1ÞÞ  f ðLp ðk1 Þ, Lq ðk2 ÞÞ

(9)

+ f ðLp ðk1  1Þ, Lq ðk2 ÞÞ

where if k1 ¼ 0 (i.e., k1  162z), then f(Lp(k1  1), Lq(k2  1)) ¼ f(Lp(k1  1), Lq(k2)) ¼ 0 and if k2 ¼ Z (i.e., k262z), then f(Lp(k1), Lq(k2)) ¼ f(Lp(k1  1), Lq(k2)) ¼ 0. Similarly, edges are added for all nodes from column q to column p as shown in the following equation: gðk1 ,k2 Þ ¼ f ðLq ðk1 Þ, Lp ðk2  1ÞÞ  f ðLq ðk1  1Þ, Lp ðk2  1ÞÞ  f ðLq ðk1 Þ, Lp ðk2 ÞÞ

(10)

+ f ðLq ðk1  1Þ, Lp ðk2 ÞÞ

In any finite s-t cut C, the total weight of the edges between any two adjacent columns p and q (denoted by Cp, q) equals to the surface smoothness cost of the resulting surface Si with Si( p) ¼ k1 and Si(q) ¼ k2, which is ψ(Lp(k1)  Lq(k2)), where ψ(.) is a convex function. Example of graph construction with convex smoothness constraints is shown in Fig. 5. For the ith surface, we denote an edge from ni(p, k1) to node ni(q, k2) as Ei ðpk1 , qk2 Þ. For clarity, we denote the following types of edges: • • •

Type I: Ei ðpk1 ,qk2 Þ with k2 > k1 Type II: Ei ðpk1 , qk2 Þ with k2 ¼ k1 Type III: Ei ðpk1 ,qk2 Þ with k2 < k1

The edge weights in the example graph are summarized in Table 1 and a quadratic convex function ψ(k1  k2) ¼ (k1k2)2 is used in this example. The following can be verified from the example shown in Fig. 5: • •

•

•

The correct cost of cut C1 ¼ (2112)2 ¼ 81. Intercolumn edges for computing cost of C1 are Type I edges: E(p2, q3), E(p1, q3). Using Table 1, cost of cut C1 ¼ 65 + 16 ¼ 81. The correct cost of cut C2 ¼ (2537)2 ¼ 144. Intercolumn edges for computing cost of C2 are Type I edges: E(q4, p5), E(q3, p4), Type II edges: E(q4, p4). Using Table 1, cost of cut C2 ¼ 9 + 9 + 126 ¼ 144. The correct cost of cut C3 ¼ (253)2 ¼ 484. Intercolumn edges for computing cost of C3 are Type I edges: E(p0, q2), E(p1, q2), E(p1, q3), E(p2, q3), Type II edges: E(p3, q3), E(p2, q2), Type III edges: E(p3, q2). Using Table 1, cost of cut C3 ¼ 1 + 152 + 16 + 65 + 88 + 90 + 72 ¼ 484. The correct cost of cut C4 ¼ (251)2 ¼ 576. Intercolumn edges for computing cost of C4 are Type I edges: E(p0, q1), E(p0, q2), E(p1, q2), E(p1, q3), E(p2, q3), Type II edges: E(p3, q3), E(p2, q2), E(p1, q1), Type III edges: E(p3, q2), E(p3, q1), E(p2, q1). Using Table 1, cost of cut C4 ¼ 8 + 1 + 152 + 16 + 65 + 88 + 90 + 48 + 72 + 16 + 20 ¼ 576.

2.2.3 Intersurface edges Intersurface edges are added between corresponding columns p in subgraph Gi and Gi+1 to enforce the surface separation term Hp() in a similar manner as Ref. [40]. Specifically, along

78

Diabetes and Retinopathy

t z=5

52

50

z=5 C2

z=4

z=3

34

37

25

28

z=4

z=3 C1

z=2

21

12

Lp(z)

Node of column a with corresponding label Lp(z).

Lq(z)

Node of column b with corresponding label Lq(z).

z=2 C3

z=1

16

3

z=1 C4

+∞ Intracolumn edge Intercolumn edge type I

z=0

4

1

Column p

Column q

z=0

Intercolumn edge type II Intercolumn edge type III

s

FIG. 5 Example graph construction with surface smoothness constraint modeled as a convex function in irregularly sampled space.

every column p in Gi, each node ni(p, z) has a directed edge with +∞ weight to the node ni+1(p, z0 ) (z0 z, Lp(z0 )  Lp(z)
di, i+1, Lp(z0  1)  Lp(z) < di, i+1). Another edge with +∞ weight is added from node ni(p, z) to the terminal node t if Lp(Z  1)  Lp(z) < di, i+1. Example graph construction showing the intersurface edges is shown in Fig. 6. Verification can be made that no finite s-t cut is possible when Lp(z0 )  Lp(z) < di, i+1, since the cost of the cut would be infinite due to inclusion of intersurface edge with +∞ weight.

Chapter 4 • Optimal surface segmentation with subvoxel accuracy in SD-OCT

Table 1

79

Intercolumn edge weights of graph construction in Fig. 5.

Edge

Type

Weight

Edge

Type

Weight

E(p0, q1) E(p0, q2) E(p1, q1) E(p1, q2) E(p1, q3) E(p2, q1) E(p2, q2) E(p2, q3) E(p3, q1) E(p3, q2) E(p3, q3) E(p4, q1) E(p4, q2) E(p4, q3) E(p4, q4) E(p5, q1) E(p5, q2) E(p5, q3) E(p5, q4) E(p5, q5) E(p5, q6)

I I II I I III II I III III II III III III II III III III III II I

8 1 48 152 16 20 90 65 16 72 88 36 162 279 36 72 324 576 315 221 4

E(q2, E(q3, E(q3, E(q3, E(q3, E(q4, E(q4, E(q4, E(q4, E(q4, E(q5, E(q5, E(q5, E(q5, E(q5,

III III III II I III III III II I III III III III II

64 368 95 40 9 216 90 72 126 9 312 130 104 234 247

p1) p1) p2) p3) p4) p1) p2) p3) p4) p5) p1) p2) p3) p4) p5)

Note: ψ(k1  k2) ¼ (k1k2)2.

FIG. 6 Example graph to demonstrate surface separation constraint edges. Only the intersurface edges are shown for clarity. di, i+1 ¼ 2. C1 is a feasible cut and C2 is an infeasible cut (Lp(z0 ¼ 1)  Lp(z ¼ 1) < di, i+1).

80

Diabetes and Retinopathy

2.3 Surface recovery from minimum s-t cut The minimum s-t cut on the graph produces λ surfaces Si, where i ¼ 1, 2, …,λ. The final surface positions for each column p are recovered by applying the mapping function Lp : f0, 1, …, Z  1g ! , where p x y, resulting in surface positions for each column   Lp ðSi ðpÞÞ z , where z .

3 Experimental methods The following experiments were carried out to demonstrate the performance of the method. The first experiment is designed toward showing subvoxel segmentation accuracy and the second experiment is used to show super-resolution segmentation accuracy (achievement of adequate segmentation accuracy on downsampled data compared to segmentation accuracy in original resolution with convex surface smoothness constraints). The proposed method is compared with optimal surface segmentation with convex constraints in regularly sampled space [18] (OSCS). To demonstrate the utility of our method, three surfaces were simultaneously segmented in this study. The surfaces are S1—internal limiting membrane (ILM), S2—inner aspect of retinal pigment epithelium drusen complex (IRPEDC), and S3—outer aspect of Bruch membrane (OBM) as shown in Fig. 7.

3.1 Data 30 SD-OCT volumes of normal eyes and their respective expert manual tracings were used in this experiment from Ref. [42]. The size of the OCT volumes were (1000  100  512 voxels with voxel size 6.54  67  3.23μm3). The 30 volumes were selected randomly from

S1 S2 S3

(A)

(B)

FIG. 7 (A) A single B-scan from an SD-OCT volume of a normal eye, and (B) three identified target surfaces S1, S2, and S3.

Chapter 4 • Optimal surface segmentation with subvoxel accuracy in SD-OCT

81

the data source. Five SD-OCT volumes were used for cost function design parameters tuning, while 25 SD-OCT volumes were used for final validation and evaluation of the methods. Since, the obtained expert manual tracing had been marked with traditional equidistant voxel centers; therefore, for valid comparison, “input volume data” was created by downsampling the volumes and corresponding expert manual tracings were mapped to the downsampled version to create “the expert subvoxel accurate manual tracings.”

3.2 Workflow The following steps are involved in preprocessing the data for the experiments: • • • •

application of a 10  10  10 median filter; application of a 10  10  10 Gaussian filter with a standard deviation of 7; downsampling the original volumes by a factor of 4 in the x-direction and factor of η in the z-direction (results in size of 250  100  512 η voxels); and cost image volumes Di, η target surfaces Si(i ¼ 1, 2, 3) at scale η is created from input volume data.

3.2.1 Experiment for subvoxel accuracy The experiment is carried out with η ¼ 4 for the downsampled image volumes. First, the cost images are segmented using the OSCS method to obtain segmentations for comparisons. Next, the shift in voxel centers is computed by GVF [43] (Section 3.2.4) on the input data. The computed deformation field is then applied to cost image volumes to obtain D0i,η¼4 . Details regarding the deformation field application can be found in Ref. [40]. Finally, the proposed method is applied to the deformed volumes D0i, η¼4 (i ¼ 1, 2, 3) with nonequidistant voxel center spacing. The generated input volume data were used to evaluate segmentation accuracy of the two methods with respect to the expert subvoxel accurate manual tracings. For fair and robust analysis, the deformation obtained from GVF was applied to the automated segmentations obtained from the OSCS method, resulting in deformed OSCS (DOSCS) segmentations. The workflow for this experiment is shown in Fig. 8.

3.2.2 Experiment for super-resolution accuracy The data in this experiment are downsampled at η ¼ 2, 4, 6, 8. OSCS method is used to segment the cost image at the original scale (η ¼ 1) to create segmentations for comparisons with the proposed method. The generated input volumes at different scales are segmented using the proposed method. The voxel center shift was computed using GVF at each scale η. The computed GVF is then applied to the cost image volumes. Finally, the deformed cost image volumes D0i, η (i ¼ 1, 2, 3 and η ¼ 2, 4, 6, 8) at each scale η (nonequidistant spacing between voxel centers) were segmented with the proposed method. The workflow for the experiment is shown in Fig. 9.

82

Diabetes and Retinopathy

Segmenta on using OSCS method Original SD-OCT volume (1000 x 100 x 512 voxels)

Preprocessing

Down-sample (250 x 100 x 128 voxels)

DOSCS segmenta ons

Performance evalua on

GVF Segmenta on using proposed method Expert manual tracings

Map to low resolu on (subvoxel accurate expert manual tracings)

FIG. 8 Experiment design for segmentation of SD-OCT volumes of normal eye with subvoxel accuracy.

Segmenta on using OSCS [2] method Original SD-OCT volume (1000 x 100 x 512 voxels)

Down-sample (250 x 100 x 512/n voxels)

GVF

Segmenta on using proposed method

Performance evalua on

Expert manual tracings

FIG. 9 Experiment design for segmentation of SD-OCT volumes of normal eye for super-resolution accuracy.

3.2.3 Cost function design For surfaces S1 and S3, a 3D Sobel filter of size 5  5  5 voxels was applied to generate cost volumes D1 and D3. To detect surface S2, cost volume D2 is generated using machine learning. A 11  11 window centered at each voxel I(x, y, z) with the corresponding voxel intensities was used to generate a feature vector of 121 features and a 10-tree random forest classifier [44] was trained on the parameter tuning set to learn the probability maps with respect to the target surfaces using the expert manual tracings. The trained classifier is then used to produce probability maps D02(x, y, z) and cost volume D2 is created as D2(x, y, z) ¼ (1  D02(x, y, z))  255.

3.2.4 Gradient vector flow A GVF [43] is a feature preserving diffusion of the gradient in a given image volume. In this study, GVF is used as a deformation field F(x, y, z) to shift (Eq. 11) the evenly distributed voxels to the deformed space.

Chapter 4 • Optimal surface segmentation with subvoxel accuracy in SD-OCT

0

0

0

ðx ,y ,z Þ ¼ ðx, y, zÞ + γFðx, y, zÞ

83

(11)

where γ is a normalization factor. The displacement of each voxel center is confined within the same voxel and F(x, y, z) is normalized (Eq. 12) to a maximum deformation of half the voxel size δ. λ¼

δ 2  maxðx, y , zÞðX , Y , ZÞ jjFðx, y, zÞjj

(12)

3.2.5 Parameter setting Same parameters were used for segmentation by both the OSCS and proposed method. A linear (convex) function, ψ(k1  k2) ¼ jk1  k2j and hard constraint are used to model surface smoothness term and surface separation term Hp(.). For the surface separation term, only the minimum surface separation is enforced. The minimum separation parameters used were d1, 2 ¼ 15 and d2, 3 ¼ 1 for η ¼ 4 in the experiment for subvoxel accuracy. The minimum separation parameters used in the experiment for super-resolution accuracy at different scales were: d1, 2 ¼ 60 and d2, 3 ¼ 4 for η ¼ 1, d1, 2 ¼ 30 and d2, 3 ¼ 2 for η ¼ 2, d1, 2 ¼ 15 and d2, 3 ¼ 1 for η ¼ 4, d1, 2 ¼ 10 and d2, 3 ¼ 0.8 for η ¼ 6, d1, 2 ¼ 7 and d2, 3 ¼ 0.5 for η ¼ 8.

4 Results Unsigned mean surface positioning error (UMSP) and unsigned average symmetric surface distance error (UASSD) are used to evaluate segmentation accuracy. The UMSP error was computed by averaging the vertical difference between the subvoxel accurate manual tracings and the automated segmentations for all columns in the input volume data. The UASSD error was calculated by averaging the closest distance between all surface points of the automated segmentation and those of the expert manual tracings in the physical space. A two-tailed paired t-test with P .05 was used to assess the statistical significance of the observed differences.

4.1 Results for subvoxel accuracy The USMP and ASSD errors are summarized in Tables 2 and 3. The proposed method produced significantly lower UMSP and UASSD errors for all three surfaces, S1 (P < .01), S2 (P < .01), and S3 (P < .001) compared to the OSCS method and the DOSCS segmentations. Qualitative illustration is shown in Figs. 10 and 11 and shows higher accuracy segmentations from the proposed method compared to the OSCS and DOSCS methods for all the three surfaces. The DOSSCS segmentations failed to achieve the globally optimum solution as shown in the last row in Fig. 10, while the proposed method results in subvoxel accurate segmentations. This also demonstrates that solution computed in the irregularly sampled graph space encodes potential surface locations more accurately.

84

Diabetes and Retinopathy

Table 2

UMSP (mean  standard deviation) in voxels.

Surface

OSCS vs. Obsv

DOSCS vs. Obsv

Our method vs. Obsv

S1 S2 S3

0.38  0.05 (P < .01) 0.58  0.37 (P < .01) 0.93  0.47 (P < .001)

0.34  0.05 (P < .01) 0.57  0.36 (P < .01) 0.74  0.45 (P < .001)

0.23  0.04 0.50  0.32 0.47  0.43

Overall

0.63  0.30 (P < .01)

0.55  0.29 (P < .01)

0.40  0.26

Obsv—subvoxel accurate expert manual tracings.

Table 3

UASSD (mean  standard deviation) in μm.

Surface

OSCS vs. Obsv

DOSCS vs. Obsv

Our method vs. Obsv

S1 S2 S3

4.91  0.63 (P < .01) 7.35  3.91 (P < .01) 12.06  5.03 (P < .001)

4.58  0.73 (P < .01) 7.12  3.76 (P < .01) 9.10  4.97 (P < .001)

3.05  0.55 6.51  3.61 6.37  4.77

Overall

8.11  3.19 (P < .01)

6.93  3.15 (P < .01)

5.31  2.98

4.2 Results for super-resolution accuracy The UASSD errors are summarized in Table 4. There was no significant difference between the proposed method at η ¼ 2 and the OSCS method at original scale (η ¼ 1) for S1 (P >.05), S2 (P >.05) and S3 (P >.05). For η ¼ 4, there was a significant difference for S2 (P