Using explainable deep learning in da Vinci Xi robot for tumor detection

Azad, Rohan Ibn; Mukhopadhyay, Subhas; Asadnia, Mohsen

doi:10.21307/ijssis-2021-017

Abstract

Deep learning has proved successful in computer-aided detection in interpreting ultrasound images, COVID infections, identifying tumors from computed tomography (CT) scans for humans and animals. This paper proposes applications of deep learning in detecting cancerous cells inside patients via laparoscopic camera on da Vinci Xi surgical robots. The paper presents method for detecting tumor via object detection and classification/localizing using GRAD-CAM. Localization means heat map is drawn on the image highlighting the classified class. Analyzing images collected from publicly available partial robotic nephrectomy videos, for object detection, the final mAP was 0.974 and for classification the accuracy was 0.84.

References

Abaza, R. 2020a. Avoiding Positive Margins During Robotic Partial Nephrectomy presented by Ronney Abaza, Seattle Science Foundation, Seattle, Washington, Available at: https://www.youtube.com/watch?v=C3VTbb_1GAM&ab_channel=SeattleScienceFoundation (Accessed March 20, 2021).
Search in Google Scholar Back to article
Abaza, R. 2020b. Robotic Partial Nephrectomy for Complex Tumors presented by Ronney Abaza, Seattle Science Foundation, Seattle, Washington, Available at: https://www.youtube.com/watch?v=vvf16vBrgxQ&t=662s&ab_channel=SeattleScienceFoundation (Accessed March 20, 2021).
Search in Google Scholar Back to article
American Institute of Minimally Invasive Surgery 2019. DA VINCI XI, American Medical Center, Available at: https://www.aimisrobotics.com/da-vinci-xi/ (Accessed June 26, 2021).
Search in Google Scholar Back to article
Alexey, A. B. n.d. darknet, Available at: https://github.com/AlexeyAB/darknet.
Search in Google Scholar Back to article
Aly, G. H., Marey, M., El-Sayed, S. A. and Tolba, M. F. 2021. YOLO based breast masses detection and classification in full-field digital mammograms. Computer Methods and Programs in Biomedicine 200: 105823. Available at: https://doi.org/10.1016/j.cmpb.2020.105823.
Search in Google Scholar Back to article
Asadnia, M., Kottapalli, A. G. P., Miao, J., Benson, R. A., Sabbagh, A., Kropelnicki, P. and Tsai, J. 2013. High temperature characterization of PZT (0.52/0.48) thin-film pressure sensors. Journal of Micromechanics and Microengineering 24(1): 015017.
Search in Google Scholar Back to article
Asadnia, M., Chua, L. H., Qin, X. and Talei, A. 2014. Improved particle swarm optimization–based artificial neural network for rainfall-runoff modeling. Journal of Hydrologic Engineering 19(7): 1320–1329.
Search in Google Scholar Back to article
Asadnia, M., Yazdi, M. S. and Khorasani, A. 2010. An improved particle swarm optimization based on neural network for surface roughness optimization in face milling of 6061-T6 Aluminum. International Journal of Applied Engineering Research 5(19): 3191–3201.
Search in Google Scholar Back to article
Asadnia, M., Khorasani, A. M. and Warkiani, M. E. 2017. An accurate PSO-GA based neural network to model growth of carbon nanotubes. Journal of Nanomaterials 2017.
Search in Google Scholar Back to article
Aubreville, M., Bertram, C. A., Donovan, T. A., Marzahl, C., Maier, A. and Klopfleisch, R. 2020. A completely annotated whole slide image dataset of canine breast cancer to aid human breast cancer research. Scientific Data 7(1): 417, doi: 10.1038/s41597-020-00756-z.
Open DOI Search in Google Scholar Back to article
Bazaz, S. R., Mehrizi, A. A., Ghorbani, S., Vasilescu, S., Asadnia, M. and Warkiani, M. E. 2018. A hybrid micromixer with planar mixing units. RSC Advances 8(58): 33103–33120.
Search in Google Scholar Back to article
Bennet, M., Thamilvalluvan, B., Alphonse, P. P., Thendralarasi, D. R., Sujithra, K. J. I. J. O. S. S. and Systems, I. 2017. Performance and analysis of automatic license plate localization and recognition from video sequences. International Journal on Smart Sensing and Intelligent Systems 10: 330–343.
Search in Google Scholar Back to article
Bochkovskiy, A., Wang, C. and Liao, H. 2020. YOLOv4: optimal speed and accuracy of object detection. Computer Vision and Pattern Recognition 1.
Search in Google Scholar Back to article
Brunese, L., Mercaldo, F., Reginelli, A., Santone, A. J. C. M. and Biomedicine, P. I. 2020. Explainable deep learning for pulmonary disease and Coronavirus COVID-19 detection from x-rays vol. 196: 105608–105608.
Search in Google Scholar Back to article
Charibaldi, N., Harjoko, A., Azhari, Hisyam, B. J. I. J. O. S. S. and Systems, I. 2018. A new HGA-FLVQ model for Mycobacterium Tuberculosis detection,”. International Journal on Smart Sensing and Intelligent Systems 11: 1–13.
Search in Google Scholar Back to article
Chen, Z., Zhang, T. and Ouyang, C. 2018. End-to-end airplane detection using transfer learning in remote sensing images. Remote Sensing 10(1): 139, doi: 10.3390/rs10010139.
Open DOI Search in Google Scholar Back to article
Chollet, F. 2017. Deep Learning with Python. Manning Publications, Shelter Island, NY.
Search in Google Scholar Back to article
Chung, A. G., Khalvati, F., Shafiee, M. J., Haider, M. A. and Wong, A. 2015. Prostate cancer detection via a quantitative radiomics-driven conditional random field framework. IEEE Access 3: 2531–2541, doi: 10.1109/ACCESS.2015.2502220.
Open DOI Search in Google Scholar Back to article
David, B. and Samadi, M. D. n.d. History and The Future of Robotic Surgery, Robotic Oncology, Available at: https://www.roboticoncology.com/history-of-robotic-surgery/.
Search in Google Scholar Back to article
Depeursinge, A., Vargas, A., Platon, A., Geissbuhler, A., Poletti, P. -A. and Müller, H. 2012. Building a reference multimedia database for interstitial lung diseases. Computerized Medical Imaging and Graphics 36(3): 227–238, Available at: https://doi.org/10.1016/j.compmedimag.2011.07.003.
Search in Google Scholar Back to article
Engel, D., Jason, D. and Engel, M. D. 2016. Robotic Partial Nephrectomy, Oroogic Surgeons of Washington, Available at: https://www.youtube.com/watch?v=UXWjNqTwb_4&ab_channel=JasonD.Engel%2CM.D (Accessed March 20, 2021).
Search in Google Scholar Back to article
Ge, L., Dan, D. and Hui, L. 2020. An accurate and robust monitoring method of full-bridge traffic load distribution based on YOLO-v3 machine vision. Structural Control and Heath Monitoring 27.
Search in Google Scholar Back to article
Geron, A. 2019. Hands-on Machine Learning with Scikit-learn, Keras & TensorFlow. o’Reiley Media, Inc, Sebatopol, CA.
Search in Google Scholar Back to article
GlobalCastMD. n.d. 02 Robotic partial nephrectomycourse tips for retroperitoneral partial nephrectomy James Porter HD, Available at: https://www.youtube.com/watch?v=S80t7cnFLus&ab_channel=GlobalCastMD.
Search in Google Scholar Back to article
Hadjiyski, N. 2020. Kidney cancer staging: deep learning neural network based approach. 2020 International Conference on e-Health and Bioengineering (EHB), October 29–30, pp. 1–4, doi: 10.1109/EHB50910.2020.9280188.
Open DOI Search in Google Scholar Back to article
Hagihghi, R., Razmjou, A., Orooji, Y., Warkiani, M. E. and Asadnia, M. 2020. A miniaturized piezoresistive flow sensor for real-time monitoring of intravenous infusion. Journal of Biomedical Materials Research Part B: Applied Biomaterials 108(2): 568–576.
Search in Google Scholar Back to article
Hammal, S., Bourahla, N. and Laouami, N. 2020. Neural-network based prediction of inelastic response spectra. Civil Engineering Journal 6(6): 1124–1135.
Search in Google Scholar Back to article
Hampton, L. 2015. da Vinci Xi Right Robotic Partial Nephrectomy-Unedited, VCUrobotics, Richmond, VI, Available at: https://www.youtube.com/watch?v=6eyZzoScc54&ab_channel=VCUrobotics (Accessed March 20, 2021).
Search in Google Scholar Back to article
He, K., Zhang, X., Ren, S. and Sun, J. 2016. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27–30, pp. 770–778, doi: 10.1109/CVPR.2016.90.
Open DOI Search in Google Scholar Back to article
Inc, A. and Grove, B. 2021. Indocyanine Green Side Effects, Available at: https://www.drugs.com/sfx/indocyanine-green-side-effects.html.
Search in Google Scholar Back to article
Junnumtuam, S., Niwitpong, S. -A. and Niwitpong, S. 2021. The Bayesian confidence interval for coefficient of variation of zero-inflated poisson distribution with application to daily COVID-19 deaths in Thailand. Emerging Science Journal 5: 62–76.
Search in Google Scholar Back to article
Kharate, G., Ghotkar, A. J. I. J. O. S. S. and Systems, I. 2016. Vision based multi-feature hand gesture recognition for Indian sign language manual signs. International Journal on Smart Sensing and Intelligent Systems 9: 124–147.
Search in Google Scholar Back to article
Khorasani, A. M., Gibson, I., Asadnia, M. and O’Neill, W. 2018. Mass transfer and flow in additive manufacturing of a spherical component. International Journal of Advanced Manufacturing Technology 96: 3711–3718.
Search in Google Scholar Back to article
Kibel, A. 2018. Robotic Assisted Laparoscopic Partial Nephrectomy, Brigham and Women’s Hospital, Boston, MA, Available at: https://www.youtube.com/watch?v=GQm90mWVMJM&ab_channel=BrighamAndWomen%27sHospital (Accessed March 20, 2021).
Search in Google Scholar Back to article
Kottapalli, A. G. P., Asadnia, M., Miao, J. and Triantafyllou, M. 2015. Soft polymer membrane micro-sensor arrays inspired by the mechanosensory lateral line on the blind cavefish. Journal of Intelligent Material Systems and Structures 26(1): 38–46.
Search in Google Scholar Back to article
Li, M., Zhang, Z., Lei, L., Wang, X. and Guo, X. 2020. Agricultural greenhouses detection in high-resolution satellite images based on convolutional neural networks: comparison of faster R-CNN, YOLO v3 and SSD. Sensors 20(17), 10.3390/s20174938.
Open DOI Search in Google Scholar Back to article
Lin, T., Goyal, P., Girshick, R., He, K. and Dollár, P. 2020. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2): 318–327, doi: 10.1109/TPAMI.2018.2858826.
Open DOI Search in Google Scholar Back to article
Long, J., Shelhamer, E. and Darrell, T. 2015. Fully convolutional networks for semantic segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 7–12, pp. 3431–3440, doi: 10.1109/CVPR.2015.7298965.
Open DOI Search in Google Scholar Back to article
Mahmud, M. A. P., Azadi, F. E. S., Myers, M., Pejcic, B., Abbassi, R., Razmjou, A. and Asadnia, A. 2020. Recent progress in sensing nitrate, nitrite, phosphate, and ammonium in aquatic environment. Chemosphere 259: 127492.
Search in Google Scholar Back to article
Moshizi, S. A., Azadi, S., Belford, A., Ramjou, A., Qu, S., Han, Z. J. and Asadnia, M. 2020. Development of an ultra-sensitive and flexible piezoresistive flow sensor using vertical graphene nanosheets. Nano-micro Letters 12.
Search in Google Scholar Back to article
Nakhaeinia, D., Payeur, P., Aragon, A. C., Cretu, A-. M., Laganiere, R. and Macknojia, R. 2016. Surface following with an rgb-d vision-guided robotic system for automated and rapid vehicle inspection. International Journal on Smart Sensing and Intelligent Systems 9: 419–447.
Search in Google Scholar Back to article
National Kidney Foundation. n.d. Nephrectomy, Available at: https://www.kidney.org/atoz/content/nephrectomy.
Search in Google Scholar Back to article
N. Cancer. n.d. Kidney cancer: stages Available at: https://www.cancer.net/cancer-types/kidney-cancer/stages.
Search in Google Scholar Back to article
Ohira, N. 2018. Memory-efficient 3D connected component labeling with parallel computing. Signal, Image and Video Processing 12(3): 429–436, doi: 10.1007/s11760-017-1175-7.
Open DOI Search in Google Scholar Back to article
Pantanowitz, L., Garza, G., Bien, L., Heled, R., Laifenfeld, D., Linhart, C., Sandbank, J., Shach, A. and Shalev, V. 2020. An artificial intelligence algorithm for prostate cancer diagnosis in whole slide images of core needle biopsies: a blinded clinical validation and deployment study. The Lancet Digital Health 2(8): e407–e416, doi: 10.1016/S2589-7500(20)30159-X.
Open DOI Search in Google Scholar Back to article
Poggiali, E., Dacrema, A. and Bastoni, D. 2020. Can Lung US Help Critical Care Clinicians in the early diagnosis of Novel Coronavirus (COVID-19) pneumonia? Radiology 295.
Search in Google Scholar Back to article
Porter, J. 2015. LIVE SURGERY: Retroperitoneal Robotic Partial Nephrectomy, Seattle Science Foundation, Seattle, Washington, Available at: https://www.youtube.com/watch?v=nwrbKNbLCv8&t=5045s&ab_channel=SeattleScienceFoundation (Accessed March 2021).
Search in Google Scholar Back to article
P. N. U. Specialist. Robotic partial nephrectomy comparisons https://www.youtube.com/watch?v=epvKkH3ekRo&ab_channel=PacificNorthwestUrologySpecialists%2CPLLC, Available at: https://www.youtube.com/watch?v=epvKkH3ekRo&ab_channel=PacificNorthwestUrologySpecialists%2CPLLC.
Search in Google Scholar Back to article
Razfar, M., Asadnia, M., Haghshenas, M. and Farahnakian, M. 2010. Optimum surface roughness prediction in face milling X20Cr13 using particle swarm optimization algorithm. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture 224(11): 1645–1653.
Search in Google Scholar Back to article
Razmjou, A., Asadnia, M., Ghaebi, O., Yang, H-. C., Warkiani, M. E., Hou, J. and Chen, V. 2017. Preparation of iridescent 2D photonic crystals by using a mussel-inspired spatial patterning of ZIF-8 with potential applications in optical switch and chemical sensor. ACS Applied Materials & Interfaces 9(43): 38076–38080.
Search in Google Scholar Back to article
Redmon, J. and Farhadi, A. 2018. YOLOv3: an incremental improvement. Computer Vision and Pattern Recognition 1.
Search in Google Scholar Back to article
Rogers, C. 2015. Dr. Craig Rogers: da Vinci Partial Nephrectomy, Vattikuti Foundation, Bangalore, Available at: https://www.youtube.com/watch?v=gdg7EhsKki8&ab_channel=VattikutiFoundation (Accessed March 20, 2021).
Search in Google Scholar Back to article
Roth, H. R., Le, L., Liu, J., Yao, J., Seff, A., Cherry, K., Kim, L. and Summers, R. M. 2016. Improving computer-aided detection using convolutional neural networks and random view aggregation. IEEE Transactions on Medical Imaging 35(5): 1170–1181, doi: 10.1109/TMI.2015.2482920.
Open DOI Search in Google Scholar Back to article
Roy, S., Menapace, W., Oei, S., Luijten, B., Fini, E., Saltori, C., Huijben, I. A. M., Chennakeshava, N., Mento, F., Sentelli, A., Peschiera, E., Trevisan, R., Maschietto, G., Torri, E., Inchingolo, R., Smargiassi, A., Soldatti, G., Rota, P., Passerini, A., Sloun, R. J. G. V., Ricci, E. and Demi, L. 2020. Deep learning for classification and localization of COVID-19 markers in point-of-care lung ultrasound. IEEE Transactions on Medical Imaging 39(8): 2676–2687, doi: 10.1109/TMI.2020.2994459.
Open DOI Search in Google Scholar Back to article
Seff, A., Cherry, K. M., Roth, H., Liu, J., Wang, S., Hoffman, J., Turkbey, E. B. and Summers, R. M. 2014. 2D view aggregation for lymph node detection using a shallow hierarchy of linear classifiers. Medical Image Computing and Computer-Assisted Interval 17(Pt 1): 544–552, doi: 10.1007/978-3-319-10404-1_68.
Open DOI Search in Google Scholar Back to article
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D. and Batra, D. 2017. Grad-CAM: visual explanations from deep networks via gradient-based localization. 2017 IEEE International Conference on Computer Vision (ICCV), October 22–29, pp. 618–626, doi: 10.1109/ICCV.2017.74.
Open DOI Search in Google Scholar Back to article
Shin, H-. C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D. and Summers, R. M. 2016. Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging 35(5): 1285–1298, doi: 10.1109/TMI.2016.2528162.
Open DOI Search in Google Scholar Back to article
Simonyan, K. and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition.
Search in Google Scholar Back to article
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhouchke, V. and Rabinovich, A. 2015. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 7–12, pp. 1–9, doi: 10.1109/CVPR.2015.7298594.
Open DOI Search in Google Scholar Back to article
tzutalin. 2017. LabelImg, Available at: https://github.com/tzutalin/labelImg.
Search in Google Scholar Back to article
Ünver, H. M. and Ayan, E. 2019. Skin lesion segmentation in dermoscopic images with combination of YOLO and GrabCut Algorithm. Diagnostics (Basel, Switzerland) 9(3), doi: 10.3390/diagnostics9030072.
Open DOI Search in Google Scholar Back to article
Wang, Y., Zheng, B., Gao, D. and Wang, J. 2020. A weakly-supervised framework for COVID-19 classification and lesion localization from chest CT. IEEE Transactions on Medical Imaging 39(8): 2615–2625, doi: 10.1109/TMI.2020.2995965.
Open DOI Search in Google Scholar Back to article
Wang, Y., Zheng, B., Gao, D. and Wang, J. 2018. Fully convolutional neural networks for prostate cancer detection using multi-parametric magnetic resonance images: an initial investigation. 2018 24th International Conference on Pattern Recognition (ICPR), August 20–24, pp. 3814–3819, doi: 10.1109/ICPR.2018.8545754.
Open DOI Search in Google Scholar Back to article
Wu, Y. -H., Gao, S. -H., Mei, J., Xu, J., Fan, D. -P., Zhang, R. -G. and Cheng, M. -M. 2021. JCS: an explainable COVID-19 diagnosis system by joint classification and segmentation 30: 3113–3126.
Search in Google Scholar Back to article
Zeiler, M. D., Fergus, R. 2014. “Visualizing and understanding convolutional networks”, In Fleet, D., Pajdla, T., Schiele, B. and Tuytelaars, T. (Eds), Computer Vision – ECCV 2014 Cham: Springer International Publishing, pp. 818–833.
Search in Google Scholar Back to article
Zhang, H., Cisse, M., Dauphin, Y. N. and Lopezpaz, D. 2018. Mixup: Beyond empirical risk minimization. presented at the in Proc. Int. Conf. Learn. Represent.
Search in Google Scholar Back to article

Using explainable deep learning in da Vinci Xi robot for tumor detection

Abstract

Paradigm

My account