Communication with the Manipulator Using Gestures to Enhance the Manipulability of Persons with Reduced Mobility

Sebastian Koryl; Katarzyna Zadarnowska; Krzysztof Arent

doi:10.14313/jamris-2026-015

.blurhash-client-img { display: none !important; }

Communication with the Manipulator Using Gestures to Enhance the Manipulability of Persons with Reduced Mobility

Journal of Automation, Mobile Robotics and Intelligent Systems

Volume 20 (2026): Issue 2 (June 2026)

By: Sebastian Koryl, Katarzyna Zadarnowska and Krzysztof Arent

Open Access

|Jun 2026

Abstract

This paper presents a robotic system that assists people with reduced mobility in the activities of picking up and putting down objects out of their reach. Human-robot communication is non-verbal, using gestures that have been specifically selected for the robot’s use. Gestures are read out using an RGB-D camera while the commands they express are executed online by a small UR3 cobot. The evaluation of the system has shown that it is useful and safe in the sense of the SUS and GQS, respectively.

References

H. F. M. Van der Loos and D. J. Reinkensmeyer. Springer Handbook of Robotics, chapter “Rehabilitation and Health Care Robotics”, 1223–1251. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.
Search in Google Scholar Back to article
A. Nanavati, V. Ranganeni, and M. Cakmak, “Physically Assistive Robots: A Systematic Review of Mobile and Manipulator Robots that Physically Assist People with Disabilities”, Annual Review of Control, Robotics, and Autonomous Systems, vol. 7, no. Volume 7, 2024, 2024, 123–147, https://doi.org/10.1146/annurev-control-062823-024352
Search in Google Scholar Back to article
R. Soyama, S. Ishii, and A. Fukase, “8 Selectable Operating Interfaces of the Meal-Assistance Device “My Spoon”. In: Advances in Rehabilitation Robotics: Human-Friendly Technologies on Movement Assistance and Restoration for People with Disabilities, Springer Berlin Heidelberg, 2004, 155–163, 10.1007/10946978_8
Open DOI Search in Google Scholar Back to article
A. Campeau-Lecours, H. Lamontagne, S. Latour, P. Fauteux, V. Maheu, F. Boucher, C. Deguire, and L.-J. C. L’Ecuyer, “Kinova Modular Robot Arms for Service Robotics Applications”, International Journal of Robotics Applications and Technologies, vol. 5, no. 2, 2017, 10.4018/IJRAT.2017070104
Open DOI Search in Google Scholar Back to article
“iARM”. Assistive Innovations, 2026. https://ww w.assistive-innovations.com/robotarm
Search in Google Scholar Back to article
“BATEO”. ACCREA, 2026. https://accrea.com/products/bateo/
Search in Google Scholar Back to article
S. Jain, A. Farshchiansadegh, A. Broad, F. Abdollahi, F. Mussa-Ivaldi, and B. Argall, “Assistive Robotic Manipulation through Shared Autonomy and a Body-Machine Interface”. In: 2015 IEEE International Conference on Rehabilitation Robotics (ICORR), 2015, 526–531, 10.1109/ICORR.2015.7281253
Open DOI Search in Google Scholar Back to article
A. Vijay Joseph, A. Mathur, J. Verma, and A. Singh, “Gesture Based Wireless Control for a Robotic Manipulator”, International Journal of Engineering & Technology, vol. 7, no. 2.31, 2018, 231–234, 10.14419/ijet.v7i2.31.13449
Open DOI Search in Google Scholar Back to article
P. Sihombing, R. B. Muhammad, H. Herriyance, and E. Elviwani, “Robotic Arm Controlling Based on Fingers and Hand Gesture”. In: 2020 3rd International Conference on Mechanical, Electronics, Computer, and Industrial Technology (MECnIT), 2020, 40–45, 10.1109/MECnIT48290.2020.9166592
Open DOI Search in Google Scholar Back to article
X. Li, “Human-Robot Interaction Based on Gesture and Movement Recognition”, Signal Processing: Image Communication, vol. 81, 2020, 115686, https://doi.org/10.1016/j.image.2019.115686
Search in Google Scholar Back to article
W. Rhee, Y. G. Kim, J. H. Lee, J. W. Shim, B. S. Kim, D. Yoon, M. Cho, and S. Kim, “Unconstrained Lightweight Control Interface for Robot-Assisted Minimally Invasive Surgery Using MediaPipe Framework and Head-Mounted Display”, Virtual Reality, vol. 28, no. 114, 2024, https://doi.org/10.1007/s10055-024-00986-1
Search in Google Scholar Back to article
A. R. Khan, A. Khan, and A. Faisal, “Gesture-Controlled Robotic Arm for Real-Time Surgical Tool Assistance”. In: 2025 Fourth International Conference on Power, Control and Computing Technologies (ICPC2T), 2025, 886–891, 10.1109/ICPC2T63847.2025.10958621
Open DOI Search in Google Scholar Back to article
J. Xie, Z. Xu, J. Zeng, Y. Gao and K. Hashimoto, “Human-Robot Interaction Using Dynamic Hand Gesture for Teleoperation of Quadruped Robots with a Robotic Arm”, Electronics, vol. 14, no. 5, 2025, 10.3390/electronics14050860
Open DOI Search in Google Scholar Back to article
C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee, W.-T. Chang, W. Hua, M. Georg, and M. Grundmann. “MediaPipe: A Framework for Building Perception Pipelines”, 2019.
Search in Google Scholar Back to article
F.-J. Chu, R. Xu, Z. Zhang, P. A. Vela, and M. Ghovanloo, “Hands-Free Assistive Manipulator Using Augmented Reality and Tongue Drive System”. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018, 5463–5468, 10.1109/IR0S.2018.8594508
Open DOI Search in Google Scholar Back to article
P. D. Lillo, F. Arrichiello, D. D. Vito, and G. Antonelli, “BCI-Controlled Assistive Manipulator: Developed Architecture and Experimental results”, IEEE Transactions on Cognitive and Developmental Systems, vol. 13, no. 1, 2021, 91–104, 10.1109/TCDS.2020.2979375
Open DOI Search in Google Scholar Back to article
K. P. Hawkins, P. M. Grice, T. L. Chen, C.-H. King, and C. C. Kemp, “Assistive Mobile Manipulation for Self-Care Tasks Around the Head”. In: 2014 IEEE Symposium on Computational Intelligence in Robotic Rehabilitation and Assistive Technologies (CIR2AT), 2014, 16–25, 10.1109/CIRAT.2014.7009736
Open DOI Search in Google Scholar Back to article
B. Kulecki, “Multimodal Robot Programming Interface Based On RGB-D Perception and Neural Scene Understanding Modules”, Journal of Automation, Mobile Robotics and Intelligent Systems, vol. 17, no. 3, 2024, 29–37, 10.14313/JAMRIS/3-2023/20
Open DOI Search in Google Scholar Back to article
C.-S. Chung, K. Williams, S. M. Eckstein, I. Eckstein, R. Cooper, and R. A. Cooper, “Manipulation Performance of an Assistive Robotic Arm Control Interface for Individuals with Cerebral Palsy - A Case Study”. In: Proceedings of the RESNA Annual Conference, 2022.
Search in Google Scholar Back to article
K. Arent, J. Jakubiak, M. Drwięga, M. Cholewinski, G. Stollnberger, M. Giuliani, M. Tscheligi, D. Szczęśniak-Stańczyk, M. Janowski, W. Brzozowski, and A. Wysokinski, “Control of Mobile Robot for Remote Medical Examination: Design Concepts and Users’ Feedback from Experimental Studies”. In: 2016 9th International Conference on Human System Interactions (HSI), 2016, 76–82, 10.1109/HSI.2016.7529612
Open DOI Search in Google Scholar Back to article
A. Lebrasseur, J. Lettre, F. Routhier, P. S. Archambault, and A. Campeau-Lecours, “Assistive Robotic Arm: Evaluation of the Performance of Intelligent Algorithms”, Assistive Technology, vol. 33, no. 2, 2021, 95–104, 10.1080/10400435.2019.1601649, PMID: 31070524.
Open DOI Search in Google Scholar Back to article
S. Koryl. “Technology for Human-Manipulator Communication Using Gestures”. Master’s thesis, Wroclaw University of Science and Technology, 2024.
Search in Google Scholar Back to article
Google. MediaPipe Solutions Guide, 2024. https://ai.google.dev/edge/mediapipe/solutions/guide
Search in Google Scholar Back to article
G. Amprimo, G. Masi, G. Pettiti, G. Olmo, L. Priano, and C. Ferraris, “Hand Tracking for Clinical Applications: Validation of the Google MediaPipe Hand (GMH) and the Depth-Enhanced GMH-D Frameworks”, Biomedical Signal Processing and Control, vol. 96, 2024, 106508, https://doi.org/10.1016/j.bspc.2024.106508
Search in Google Scholar Back to article
Y. Alemu. “Robotic System Technology to Support the Manipulability of a Person with Motor Disabilities”. Master’s thesis, Wroclaw University of Science and Technology, 2024.
Search in Google Scholar Back to article
G. Niemeyer, C. Preusche, S. Stramigioli, and D. Lee, “Telerobotics”. In: Springer Handbook of Robotics, 2016.
Search in Google Scholar Back to article
R. A. Grier, A. Bangor, P. Kortum, and S. C. Peres, “The System Usability Scale: Beyond Standard Usability Testing”, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 57, no. 1, 2013, 187–191, 10.1177/1541931213571042
Open DOI Search in Google Scholar Back to article
R. Szczepanowski, T. Niemiec, E. Cichon, K. Arent, M. Florkowski, and J. Sobecki, “Factor Analysis of the Polish Version of Godspeed Questionnaire (Gqs)”, Journal of Automation, Mobile Robotics and Intelligent Systems, vol. 16, no. 2, 2023, 30–35, 10.14313/JAMRIS/2-2022/13
Open DOI Search in Google Scholar Back to article
G. Pochwatko, J.-C. Giger, M. Rozanska-Walczuk, J. Swidrak, K. Kukielka, J. Mozaryn, and N. Piçarra, “Polish Version of the Negative Attitude Toward Robots Scale (NARS-PL)”, Journal of Automation, Mobile Robotics and Intelligent Systems, vol. 9, no. 3, 2015, 65–72, 10.14313/JAMRIS_2-2015/25
Open DOI Search in Google Scholar Back to article
A. Bangor, P. Kortum, and J. Miller, “Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale”, Journal of Usability Studies, vol. 4, no. 3, 2009, 114–123.
Search in Google Scholar Back to article

Articles in this issue

DOI: https://doi.org/10.14313/jamris-2026-015 | Journal eISSN: 2080-2145 | Journal ISSN: 1897-8649

Journal RSS Feed

Language: English

Page range: 1 - 8

Submitted on: Jan 20, 2025

Accepted on: May 13, 2025

Published on: Jun 22, 2026

Published by: Łukasiewicz Research Network – Industrial Research Institute for Automation and Measurements PIAP

In partnership with: Paradigm Publishing Services

Publication frequency: 4 issues per year

Keywords:

human-robot interactions

Related subjects:

Computer sciences,

Artificial intelligence,

Engineering,

Electrical engineering,

Control engineering, metrology and testing,

Mechanical engineering,

Fundamentals of mechanical engineering

© 2026 Sebastian Koryl, Katarzyna Zadarnowska, Krzysztof Arent, published by Łukasiewicz Research Network – Industrial Research Institute for Automation and Measurements PIAP
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.

Volume 20 (2026): Issue 2 (June 2026)