Design of a Multifunctional Operating Station Based on Augmented Reality (MOSAR)

Ivanova, Veronika; Vasilev, Plamen; Stoianov, Ivilin; Andreev, Rumen; Boneva, Ani

doi:10.2478/cait-2021-0009

Abstract

Design principles of a novel Multifunctional Operation Station (MOS) using Augmented Reality (AR) technology (MOSAR) are proposed in this paper. AR-based design allows more ergonomic remote instrument control in real time in contrast to classical instrument-centered interfaces. Another advantage is its hierarchical software structure including multiple programming interpreters. The MOSAR approach is illustrated with a remote surgical operating station that controls intelligent surgical instruments. The implementation of the Operation Station (MOS) is based on the multiplatform open-source library Tcl/Tk, and an AR extension has been developed on a Unity platform, using Vuforia SDK.

References

1. Ivanova, V., A. Boneva, Y. Doshev, S. Ivanov, P. Vasilev. Multifunctional Operating Station Based on Tcl/Tk and its Applications. – In: Proc. of 6th IEEE International Conference “Big Data, Knowledge and Control Systems Engineering” (BdKCSE’19), 27 February 2020, pp. 1-7, E ISBN: 978-1-7281-6481-6, Print on Demand (PoD) ISBN: 978-1-7281-6482-3, DOI: 10.1109/BdKCSE48644.2019.9010662.10.1109/BdKCSE48644.2019.9010662
Search in Google Scholar Back to article
2. Vasilev, P., E. Janev, G. Elenkov. Communication Interface Module for WEB – Based Control – In: Proc. of VIII International Congress “Machines, Technolоgies, Materials”, Year XIX, Vol. 3, 2011, pp. 79-82. Print ISSN: 1310-3946 (in Bulgarian).
Search in Google Scholar Back to article
3. Mountney, P., J. Fallert, S. Nicolau, L. Soler, W. P. Mewes. An Augmented Reality Framework for Soft Tissue Surgery – Medical Image Computing and Computer-Assisted Intervention – MICCAI 2014. – Lecture Notes in Computer Science, 2014, pp. 423-431. ISBN: 978-3-319-10403-4, 8673, DOI:10.1007/978-3-319-10404-1_53.10.1007/978-3-319-10404-1_5325333146
Search in Google Scholar Back to article
4. Ha, H.-G., J. Hong. Augmented Reality in Medicine – J. Hanyang Med Rev, Vol. 36, 2016, No 4, pp. 242-247. pISSN 1738-429X, eISSN 2234-4446. https://doi.org/10.7599/hmr.2016.36.4.24210.7599/hmr.2016.36.4.242
Search in Google Scholar Back to article
5. Sheng, B., S. Masood, Y. Jung. Chapter Twenty, Virtual and Augmented Reality in Medicine – Biomedical Information Technology, Elsevier Inc, 2020, pp. 673-686. ISBN 978-0-12-816034-3. https://doi.org/10.1016/B978-0-12-816034-3.00020-110.1016/B978-0-12-816034-3.00020-1
Search in Google Scholar Back to article
6. Ilchev, S., Z. Ilcheva. Internet-of-Things Communication Protocol for Low-Cost Devices in Heterogeneous Wireless Networks. – In: Proc. of 18th International Conference on Computer Systems and Technologies (CompSysTech’7), Ruse, Bulgaria, 23-24 June, 2017, ACM ICPS Vol. 1369, pp. 272-279. ISBN: 978-1-4503-5234-5, DOI: 10.1145/3134302.3134329.10.1145/3134302.3134329
Search in Google Scholar Back to article
7. Ilchev, S., Z. Ilcheva. Thermoelectric Cooling Driver for Laser Projection Systems – In: Proc. of Big Data, Knowledge and Control Systems Engineering Conference (BdKCSE’19), 21-22 November 2019, Sofia, Bulgaria, pp. 1-9, IEEE. Electronic ISBN: 978-1-7281-6481-6, Print on Demand(PoD) ISBN: 978-1-7281-6482-3. DOI: 10.1109/BdKCSE48644.2019.9010606.10.1109/BdKCSE48644.2019.9010606
Search in Google Scholar Back to article
8. Borissova, D., I. Mustakerov. Methodology for Design of Web-Based Laparoscopy e-Training System. – European Journal of Open, Distance and E-Learning, ERIC Number: EJ954909, 2011, No 2, pp. 1-9. ISSN-1027-5207. https://eric.ed.gov/?id=EJ954909
Search in Google Scholar Back to article
9. Rokhsaritalemi, S., A. Sadeghi-Niaraki, S.-M. Choi. A Review on Mixed Reality: Current Trends, Challenges and Prospects – J. MDPI, Appl. Sci, Vol. 10, 2020, No 636, pp. 1-26. https://doi.org/10.3390/app1002063610.3390/app10020636
Search in Google Scholar Back to article
10. Jagadeesan, J., B. Xavier, F. King, A. Hosny, D. Black, S. Pieper, A. Tavakkoli. A Novel Mixed Reality Navigation System for Laparoscopy Surgery – Med Image Comput Comput Assist Interv, Vol. 11073, September 2018, pp. 72-80. DOI: 10.1007/978-3-030-00937-3_9.10.1007/978-3-030-00937-3_9651286731098598
Search in Google Scholar Back to article
11. Ajey, L. Virtual Reality and Its Military Utility. – Journal of Ambient Intelligence and Humanized Computing, Vol. 4, 2013, No 1, Springer, pp. 17-26. DOI: 10.1007/s12652-011-0052-4.10.1007/s12652-011-0052-4
Search in Google Scholar Back to article
12. Liu, X., J. Zhang, G. Hou, Z. Wang. Virtual Reality and Its Application in Military. – In: Proc. of 2nd International Symposium on Resource Exploration and Environmental Science, IOP Conf. Series: Earth and Environmental Science,Vol. 170, 032155, IOP Publishing, 2018, pp. 1-8. DOI: 10.1088/1755-1315/170/3/032155.10.1088/1755-1315/170/3/032155
Search in Google Scholar Back to article
13. Adrian, I., D. Trandabăț. Enhancing the Attractiveness of Learning through Augmented Reality. – In: Proc. of International Conference on Knowledge Based and Intelligent Information and Engineering Systems, KES2018, 3-5 September 2018, Belgrade, Serbia, Procedia Computer Science, Vol. 126, Elsevier, 2018, pp. 166-175. https://www.sciencedirect.com/science/article/pii/S187705091831194310.1016/j.procs.2018.07.220
Search in Google Scholar Back to article
14. Popchev, I. P., D. A. Orozova. Towards Big Data Analytics in the e-Learning Space. – Cybernetics and Information Technologies, Vol. 19, 2019, No 3, pp. 16-24. DOI: 10.2478/cait-2019-0023. http://www.cit.iit.bas.bg/CIT_2019/v-19-3/10341-Vol-19_Issue_3-02_paper.pdf.10.2478/cait-2019-0023
Search in Google Scholar Back to article
15. Ma, Z., W. Liu. Data Receiving Method Based on Multimedia Timing in RealTime System – Cybernetics and Information Technologies, Vol. 17, 2017, No 1, pp. 126-134. DOI: 10.1515/cait-2017-0010. http://www.cit.iit.bas.bg/CIT_2017/v-17-1/10_paper.pdf10.1515/cait-2017-0010
Search in Google Scholar Back to article
16. Yi, Y., X. Hu. Robot Simultaneous Localization and Mapping Based on Self-Detected Waypoint – Cybernetics and Information Technologies, Vol. 16, 2016, No 2, pp. 212-221. DOI: 10.1515/cait-2016-0031. http://www.cit.iit.bas.bg/CIT_2016/v-16-2/10341-Vol-16_Issue_2-16_paper.pdf10.1515/cait-2016-0031
Search in Google Scholar Back to article
17. Mojtaba, N., A. Heydarian, V. Balali, K. Han. Trend Analysis on Adoption of Virtual and Augmented Reality in the Architecture, Engineering, and Construction Industry. – MDPI, Vol. 5, 2020, Issue 1, 26, pp. 1-18. https://doi.org/10.3390/data501002610.3390/data5010026
Search in Google Scholar Back to article
18. https://www.uml.org/what-is-uml.htm
Search in Google Scholar Back to article
19. Welch, B. B., K. Jones, J. Hobbs. Practical Programming in Tcl and Tk, Prentice Hall Professional. – Computers, 2003, pp. 1-882.
Search in Google Scholar Back to article
20. http://www.tcl.tk/
Search in Google Scholar Back to article
21. Ivanova, V., D. Bachvarov, A. Boneva, R. Andreev, N. Dobrinkova. System for Analysis and Control of Mechanical Properties of Biological Tissues, Utility Model. Registration No 3323/31.10.2019. p 6293 (in Bulgarian). www.bpo.bg/images/stories/buletini/binder-2019-11-15.pdf
Search in Google Scholar Back to article
22. Ivanova, V., D. Bachvarov, A. Boneva. An Advanced Robot System for Diagnostic and Therapeutics Tasks with Application in Laparoscopic Surgery. – Journal of Computer Engineering and Information Technology, Vol. 7, 2018, Issue 1, pp. 1-9. SciTechnol, ISSN: 2412-8856 (Online), DOI 10.4172 /2324-9307.1000202.
Search in Google Scholar Back to article
23. Ivanova, V.-A.-G. Laparoscopic Executive Tools for Robots. PhD Thesis, Scientific Field: Mechanical Engineering, Professional Area: Scientific Specialty Robots and Manipulators, Institute of Robotics, Bulgarian Academy of Sciences, Scientific Advisor: Professor Dr. Veselin Pavlov, Asoc. Professor Dr. Ivan Chavdarov, Sofia 2020, pp. 1-136 (in Bulgarian). http://ir.bas.bg/competitions/atanasova/avto_ata.pdf
Search in Google Scholar Back to article
24. Itamiya, T., T. Iwai, T. Kaneko. The Holographic Human for Surgical Navigation Using Microsoft HoloLens. – J. EPiC Series in Engineering, Vol. 1, 2018, pp. 26-30. https://doi.org/10.29007/wjjx10.29007/wjjx
Search in Google Scholar Back to article
25. Kim, H. S., N. Hong, M. Kim, S. G. Yoon, H. W. Yu, H.-J., Kong, S.-J. Kim, Y. J. Chai, H. J. Choi, J. Y. Choi, K. E. Lee, S. Kim, H. C. Kim. Application of a Perception Neuron® System in Simulation-Based Surgical Training. – Journal of Clinical Medicine, Vol. 8(1), 2019, No 124, pp. 1-11. ISSN: 2077-0383. https://doi.org/10.3390/jcm801012410.3390/jcm8010124635218530669562
Search in Google Scholar Back to article
26. Lee, D., J. W. Yi, J. Hong, Y. J. Chai, H. C. Kim, H.-J. Kong. Augmented Reality to Localize Individual Organ in Surgical Procedure. – Health Information Research, Vol. 24, October 2018, No 4, pp. 394-401. DOI: 10.4258/hir.2018.24.4.394. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6230535/pdf/hir-24-394.pdf10.4258/hir.2018.24.4.394623053530443429
Search in Google Scholar Back to article
27. Faure, F., C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, S. Coti. SOFA: A Multi-Model Framework for Interactive Physical Simulation. – Part of the Studies in Mechanobiology, Tissue Engineering and Biomaterials Book Series, Vol. 11, Springer, Berlin, Heidelberg, 2012, pp 283-321. Online ISBN: 978-3-642-29014-5, DOI: https://doi.org/10.1007/8415_2012_125.10.1007/8415_2012_125
Search in Google Scholar Back to article
28. https://developers.google.com/ar/discover
Search in Google Scholar Back to article
29. Akçayır, M., G. Akçayır. Advantages and Challenges Associated with Augmented Reality for Education: A Systematic Review of the Literature. – J. Educational Research Review, Vol. 20, February 2017, pp. 1-11. DOI:10.1016/j.edurev.2016.11.002.10.1016/j.edurev.2016.11.002
Search in Google Scholar Back to article
30. Belhaj, S. K., E. Ghribi, Y. Labyed, M. N. Saidi, A. Tamtaoui, N. Kaabouch. Mixed-Reality Aided System for Glioblastoma Resection Surgery Using Microsoft HoloLens. – IEEE Xplore, IEEE, 2019, pp. 079-084. e-ISSN: 2154-0373, e-ISBN: 978-1-7281-0927-5, DOI: 10.1109/EIT.2019.8833738.10.1109/EIT.2019.8833738
Search in Google Scholar Back to article
31. Pfeiffer, M., C. Riediger, S. Leger, J.-P. K¨uhn, D. Seppelt, R.-T. Hoffmann, J. Weitz, S. Speidel. Non-Rigid Volume to Surface Registration Using a Data-Driven Biomechanical Model. – Computer Science, Engineering ArXiv, 2020, arXiv:2005.14695v1. https://arxiv.org/pdf/2005.14695.pdf
Search in Google Scholar Back to article
32. Mendizabal, A., E. Tagliabue, T. Hoellinger, J.-N. Brunet, S. Nikolaev, S. Cotin. Data-Driven Simulation for Augmented Surgery. Bilen Emek Abali, Ivan Giorgio. – Developments and Novel Approaches in Biomechanics and Metamaterials, Vol. 132, 2020, pp. 71-96. 978-3-030-50464-9. ff10.1007/978-3-030-50464-9ff. https://hal.inria.fr/hal-02538101/document10.1007/978-3-030-50464-9_5
Search in Google Scholar Back to article
33. Pellicer-Valero, O. J., M. J. Rupérez, S. Martínez-Sanchis, J. D. Martín-Guerrero. Real-Time Biomechanical Modeling of the Liver Using Machine Learning Models Trained on Finite Element Method Simulations – J. Expert Systems with Applications, Vol. 143, 2019, No 1, Elsevier, Ltd., pp. 1-12. ISSN 0957-4174. https://doi.org/10.1016/j.eswa.2019.11308310.1016/j.eswa.2019.113083
Search in Google Scholar Back to article
34. https://library.vuforia.com/getting-started/overview.html
Search in Google Scholar Back to article
35. Schmid, C., R. Mohr, C. Bauckhage. Evaluation of Interest Point Detectors. International. – Journal of Computer Vision, Springer Verlag, Vol. 37, 2000, No 2, pp. 151-172. ff10.1023/A:1008199403446ff. ffinria-00548302.10.1023/A:1008199403446
Search in Google Scholar Back to article
36. Fleet, D. J., Y. Weiss. Optical Flow Estimation. – Handbook of Mathematical Models in Computer Vision, Springer, 2006, pp. 237-257. ISBN: 978-0-387-26371-7.10.1007/0-387-28831-7_15
Search in Google Scholar Back to article
37. Jurado, S., Garrido, R. Muñoz-Salinas F. J., Madrid-Cuevas M. J., Marín-Jiménez. Automatic Generation and Detection of Highly Reliable Fiducial Markers under Occlusion. – J. Pattern Recognition, Vol. 47, June 2014, Issue 6, pp. 2280-2292.10.1016/j.patcog.2014.01.005
Search in Google Scholar Back to article
38. Talbot, H., N. Haouchine, I. Peterlik, J. Dequidt, C. Duriez, H. Delingette, S. Cotin. Surgery Training, Planning and Guidance Using the SOFA Framework. – Eurographics, HAL Id: hal-01160297, Zurich, Switzerland, May 2015, pp. 1-5. https://hal.inria.fr/hal-01160297/document
Search in Google Scholar Back to article
39. Boubaker, M., M. Haboussi, J.-F. Ganghoffer, P. Aletti. Finite Element Simulation of Interactions between Pelvic Organs: Predictive Model of the Prostate Motion in the Context of Radiotherapy. – Journal of Biomechanics, Vol. 42, Elsevier Ltd, 2009, pp. 1862-1868. ISSN: 0021-9290, DOI: 10.1016/j.jbiomech.2009.05.022.10.1016/j.jbiomech.2009.05.02219559437
Search in Google Scholar Back to article
40. Fouard, C., A. Deram, Y. Keraval, E. Promayon. CamiTK: A Modular Framework Integrating Visualization – Image Processing and Biomechanical Modeling, Studies in Mechanobiology. – Tissue Engineering and Biomaterials, Vol. 11, Springer, Berlin, Heidelberg, 2012, pp. 283-321. Online ISBN: 978-3-642-29014-5, DOI: 10.1007/8415_2012_118. https://link.springer.com/chapter/10.1007/8415_2012_125
Search in Google Scholar Back to article
41. Khor, W. S., B. Baker, K. Amin, A. Chan, K. Patel, J. Wong. Augmented and Virtual Reality in Surgery – the Digital Surgical Environment: Applications, Limitations and Legal Pitfalls. – J. Annals of Translational Medicine, Vol. 4, December 2016, No 23, pp. 1-10. DOI: 10.21037/atm.2016.12.23. https://pubmed.ncbi.nlm.nih.gov/28090510/10.21037/atm.2016.12.23522004428090510
Search in Google Scholar Back to article
42. Makhataeva, Z., H. A. Varol. Augmented Reality for Robotics: A Review. – J. Robotics, Vol. 9, 2020, Issue 2, pp. 1-28. https://doi.org/10.3390/robotics902002110.3390/robotics9020021
Search in Google Scholar Back to article
43. Bauer, M., B. Bruegge, G. Klinker, A. MacWilliams, T. Reicher, S. Riß, C. Sandor, M. Wagner. Design of a Component-Based Augmented Reality Framework. IEEE Xplore, IEEE, INSPEC Accession Number: 7121584, 2001, pp. 45-54. Print ISBN: 0-7695-1375-1, DOI: 10.1109/ISAR.2001.970514.10.1109/ISAR.2001.970514
Search in Google Scholar Back to article
44. https://nxrev.com/2020/05/vuforia-vs-arcore/
Search in Google Scholar Back to article
45. ARCore. (Retreived on 02.11.2020). https://developers.google.com/ar
Search in Google Scholar Back to article

Design of a Multifunctional Operating Station Based on Augmented Reality (MOSAR)

Abstract

Paradigm

My account