Have a personal or library account? Click to login
Multimodal Robot Programming Interface Based on RGB-D Perception and Neural Scene Understanding Modules Cover

Multimodal Robot Programming Interface Based on RGB-D Perception and Neural Scene Understanding Modules

Open Access
|Mar 2024

References

  1. R. Adamini, N. Antonini, A. Borboni, S. Medici, C. Nuzzi, R. Pagani, A. Pezzaioli, and C. Tonola. “User-friendly human-robot interaction based on voice commands and visual systems,” 2021 24th International Conference on Mechatronics Technology (ICMT), 2021, pp. 1–5.
  2. A. Bochkovskiy, C. Wang, and H.M. Liao. “YOLOv4: Optimal speed and accuracy of object detection,” CoRR, vol. abs/2004.10934, 2020.
  3. M. Chen, C. Liu, and G. Du. “A human–robot interface for mobile manipulator,” Intelligent Service Robotics, vol. 11, no. 3, 2018, pp. 269–278; doi: 10.1007/s11370-018-0251-3.
  4. P. de la Puente, D. Fischinger, M. Bajones, D. Wolf, and M. Vincze. “Grasping objects from the floor in assistive robotics: Real world implications and lessons learned,” IEEE Access, vol. 7, 2019, pp. 123725–123735.
  5. W. Dudek and T. Winiarski. “Scheduling of a robot’s tasks with the tasker framework,” IEEE Access, vol. 8, 2020, pp. 161449–161471.
  6. V. Dutta, and T. Zielińska. “Predicting human actions taking into account object affordances,” Journal of Intelligent & Robotic Systems, vol. 93, 2019, pp. 745–761.
  7. V. Dutta, and T. Zielińska. “Prognosing human activity using actions forecast and structured database,” IEEE Access, vol. 8, 2020, pp. 6098–6116.
  8. Q. Gao, J. Liu, Z. Ju, and X. Zhang. “Dual-hand detection for human–robot interaction by a parallel network based on hand detection and body pose estimation,” IEEE Transactions on Industrial Electronics, vol. 66, no. 12, 2019, pp. 9663–9672.
  9. M. Gualtieri, J. Kuczynski, A.M. Shultz, A. Ten Pas, R. Platt, and H. Yanco. “Open world assistive grasping using laser selection,” 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 4052–4057.
  10. K. He, G. Gkioxari, P. Dollár, and R. Girshick. “Mask R-CNN,” 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980–2988.
  11. B. Kulecki. “Intuitive robot programming and interaction using RGB-D perception and CNN-based objects detection,” Automation 2022: New Solutions and Technologies for Automation, Robotics and Measurement Techniques, R. Szewczyk, C. Zieliński, and M. Kaliczyńska, eds., Cham, 2022, pp. 233–243.
  12. B. Kulecki, and D. Belter. “Intuicyjny interfejs programowania robota z neuronowymi modułami do interpretacji sceny,” Postępy robotyki. T.2, C. Zieliński and A. Mazur, eds., Warszawa, 2022, pp. 89–98.
  13. B. Kulecki, and D. Belter. “Robot programming interface with a neural scene understanding system,” Proceedings of the 3rd Polish Conference on Artifiicial Intelligence, April 25–27, 2022, Gdynia, Poland, 2022, pp. 130–133.
  14. B. Kulecki, K. Młodzikowski, R. Staszak, and D. Belter. “Practical aspects of detection and grasping objects by a mobile manipulating robot,” Industrial Robot, vol. 48, no. 5, 2021, pp. 688–699.
  15. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C.L. Zitnick. “Microsoft COCO: Common objects in context,” Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds., Cham, 2014, pp. 740–755.
  16. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A.C. Berg. “SSD: Single shot multibox detector,” Computer Vision – ECCV 2016, B. Leibe, J. Matas, N. Sebe, and M. Welling, eds., Cham, 2016, pp. 21–37.
  17. X. Lv, M. Zhang, and H. Li. “Robot control based on voice command,” 2008 IEEE International Conference on Automation and Logistics, 2008, pp. 2490–2494.
  18. I. Maurtua, I. Fernández, A. Tellaeche, J. Kildal, L. Susperregi, A. Ibarguren, and B. Sierra. “Natural multimodal communication for human–robot collaboration,” International Journal of Advanced Robotic Systems, vol. 14, no. 4, 2017; doi: 10.1177/1729881417716043.
  19. O. Mazhar, S. Ramdani, B. Navarro, R. Passama, and A. Cherubini. “Towards real-time physical human-robot interaction using skeleton information and hand gestures,” IEEE/RSJ Int. Conf. on Int. Robots and Systems (IROS), 2018, pp. 1–6.
  20. C. Nuzzi, S. Pasinetti, M. Lancini, F. Docchio, and G. Sansoni. “Deep learning-based hand gesture recognition for collaborative robots,” IEEE Instrumentation Measurement Magazine, vol. 22, no. 2, 2019, pp. 44–51.
  21. K.-B. Park, S. H. Choi, J. Y. Lee, Y. Ghasemi, M. Mohammed, and H. Jeong. “Hands-free human–robot interaction using multimodal gestures and deep learning in wearable mixed reality,” IEEE Access, vol. 9, 2021, pp. 55448–55464.
  22. P. Putthapipat, C. Woralert, and P. Sirinimnuankul. “Speech recognition gateway for home automation on open platform,” 2018 International Conference on Electronics, Information, and Communication (ICEIC), 2018, pp. 1–4.
  23. S. Sharan, T. Q. Nguyen, P. Nauth, and R. Araujo. “Implementation and testing of voice control in a mobile robot for navigation,” 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), 2019, pp. 145–150.
  24. M. Simão, P. Neto, and O. Gibaru. “Natural control of an industrial robot using hand gesture recognition with neural networks,” IECON 2016 – 42nd Annual Conference of the IEEE Industrial Electronics Society, 2016, pp. 5322–5327.
  25. R. Staszak, B. Kulecki, W. Sempruch, and D. Belter. “What’s on the other side? A single-view 3D scene reconstruction,” 2022 17th International Conference on Control, Automation, Robotics and Vision (ICARCV), 2022, pp. 173–180.
  26. F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann. “Mediapipe hands: On-device real-time hand tracking,” arXiv e-prints, Jun 2020; doi: 10.48550/arXiv.2006.10214.
DOI: https://doi.org/10.14313/jamris/3-2023/20 | Journal eISSN: 2080-2145 | Journal ISSN: 1897-8649
Language: English
Page range: 29 - 37
Submitted on: Jan 14, 2023
Accepted on: May 24, 2023
Published on: Mar 4, 2024
Published by: Łukasiewicz Research Network – Industrial Research Institute for Automation and Measurements PIAP
In partnership with: Paradigm Publishing Services
Publication frequency: 4 issues per year

© 2024 Bartłomiej Kulecki, published by Łukasiewicz Research Network – Industrial Research Institute for Automation and Measurements PIAP
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.