Three-Dimensional Path-Following Control of an Autonomous Underwater Vehicle Based on Deep Reinforcement Learning

Zhenyu Liang; Xingru Qu; Zhao Zhang; Cong Chen

doi:10.2478/pomr-2022-0042

Abstract

In this article, a deep reinforcement learning based three-dimensional path following control approach is proposed for an underactuated autonomous underwater vehicle (AUV). To be specific, kinematic control laws are employed by using the three-dimensional line-of-sight guidance and dynamic control laws are employed by using the twin delayed deep deterministic policy gradient algorithm (TD3), contributing to the surge velocity, pitch angle and heading angle control of an underactuated AUV. In order to solve the chattering of controllers, the action filter and the punishment function are built respectively, which can make control signals stable. Simulations are carried out to evaluate the performance of the proposed control approach. And results show that the AUV can complete the control mission successfully.

References

1. I. Stenius et al., “A System for Autonomous Seaweed Farm Inspection with an Underwater Robot,” Sensors, vol. 22, no. 13, Jul 2022, doi: 10.3390/s22135064.926977835808560
Open DOI Search in Google Scholar Back to article
2. L. Rowinski and M. Kaczmarczyk, “Evaluation of Effectiveness of Waterjet Propulsor for a Small Underwater Vehicle,” Polish Marit. Res., vol. 28, no. 4, 2022, doi: 10.2478/pomr-2021-0047.
Open DOI Search in Google Scholar Back to article
3. H. Choukri and L. Z. Qidan, “Path Following Control of Fully Actuated Autonomous Underwater Vehicle Based on LADRC,” Polish Marit. Res., vol. 25, no. 4, 2018, doi: 10.2478/pomr-2018-0130.
Open DOI Search in Google Scholar Back to article
4. L. Li, Z. Pei, J. Jin, and Y. Dai, “Control of Unmanned Surface Vehicle along the Desired Trajectory Using Improved Line of Sight and Estimated Sideslip Angle,” Polish Marit. Res., vol. 28, no. 2, 2021, doi: 10.2478/pomr-2021-0017.
Open DOI Search in Google Scholar Back to article
5. E. Vidal, N. Palomeras, M. Carreras, “Online 3D Underwater Exploration and Coverage,” in IEEE/OES Autonomous Underwater Vehicle Workshop (AUV), Rectory Univ Porto, Porto, Portugal, 2018.10.1109/AUV.2018.8729736
Search in Google Scholar Back to article
6. J. H. Wan et al., “Motion Control of Autonomous Underwater Vehicle Based on Fractional Calculus Active Disturbance Rejection,” Journal of Marine Science and Engineering, vol. 9, no. 11, Nov 2021, doi: 10.3390/jmse9111306.
Open DOI Search in Google Scholar Back to article
7. J. J. Zhou, X. Y. Zhao, T. Chen, Z. P. Yan, and Z. W. Yang, “Trajectory Tracking Control of an Underactuated AUV Based on Backstepping Sliding Mode With State Prediction,” IEEE Access, vol. 7, 2019, doi: 10.1109/access.2019.2958360.
Open DOI Search in Google Scholar Back to article
8. M. P. R. Prasad and A. Swarup, “Model predictive control of an AUV using de-coupled approach,” International Journal of Maritime Engineering, vol. 160, Jan‒Mar 2018, doi: 10.3940/rina.ijme.2018.a1.459.
Open DOI Search in Google Scholar Back to article
9. 9. J. H. Wan et al., “Multi-strategy fusion based on sea state codes for AUV motion control,” Ocean Engineering, vol. 248, Mar 2022, doi: 10.1016/j.oceaneng.2022.110600.
Open DOI Search in Google Scholar Back to article
10. Y. K. Xia, K. Xu, Z. M. Huang, W. J. Wang, G. H. Xu, and Y. Li, “Adaptive energy-efficient tracking control of a X rudder AUV with actuator dynamics and rolling restriction,” Applied Ocean Research, vol. 118, Jan 2022, doi: 10.1016/j.apor.2021.102994.
Open DOI Search in Google Scholar Back to article
11. K. Fang, H. L. Fang, J. W. Zhang, J. Q. Yao, and J. W. Li, “Neural adaptive output feedback tracking control of underactuated AUVs,” Ocean Engineering, vol. 234, Aug 2021, doi: 10.1016/j. oceaneng.2021.109211.
Open DOI Search in Google Scholar Back to article
12. J. L. Zhang, X. B. Xiang, Q. Zhang, and W. J. Li, “Neural network-based adaptive trajectory tracking control of underactuated AUVs with unknown asymmetrical actuator saturation and unknown dynamics,” Ocean Engineering, vol. 218, Dec 2020, doi: 10.1016/j.oceaneng.2020.108193.
Open DOI Search in Google Scholar Back to article
13. H. N. Esfahani and R. Szlapczynski, “Model Predictive Super-Twisting Sliding Mode Control for an Autonomous Surface Vehicle,” Polish Marit. Res., vol. 26, no. 3, 2019, doi: 10.2478/pomr-2019-0057.
Open DOI Search in Google Scholar Back to article
14. C. X. Cheng, Q. X. Sha, B. He, and G. L. Li, “Path planning and obstacle avoidance for AUV: A review,” Ocean Engineering, vol. 235, Sep 2021, doi: 10.1016/j.oceaneng.2021.109355.
Open DOI Search in Google Scholar Back to article
15. S. Brandi, M. Fiorentini, and A. Capozzoli, “Comparison of online and offline deep reinforcement learning with model predictive control for thermal energy management,” Automation in Construction, vol. 135, Mar 2022, doi: 10.1016/j.autcon.2022.104128.
Open DOI Search in Google Scholar Back to article
16. Y. Fang, Z. W. Huang, J. Y. Pu, and J. S. Zhang, “AUV position tracking and trajectory control based on fast-deployed deep reinforcement learning method,” Ocean Engineering, vol. 245, Feb 2022, doi: 10.1016/j.oceaneng.2021.110452.
Open DOI Search in Google Scholar Back to article
17. P. Zielinski and U. Markowska-Kaczmar, “3D robotic navigation using a vision-based deep reinforcement learning model,” Applied Soft Computing, vol. 110, Oct 2021, doi: 10.1016/j.asoc.2021.107602.
Open DOI Search in Google Scholar Back to article
18. D. L. Song, W. H. Gan, P. Yao, W. C. Zang, Z. X. Zhang, and X. Q. Qu, “Guidance and control of autonomous surface underwater vehicles for target tracking in ocean environment by deep reinforcement learning,” Ocean Engineering, vol. 250, Apr 2022, doi: 10.1016/j.oceaneng.2022.110947.
Open DOI Search in Google Scholar Back to article
19. E. Meyer, H. Robinson, A. Rasheed, and O. San, “Taming an Autonomous Surface Vehicle for Path Following and Collision Avoidance Using Deep Reinforcement Learning,” IEEE Access, vol. 8, 2020, doi: 10.1109/access.2020.2976586.
Open DOI Search in Google Scholar Back to article
20. A. B. Martinsen and A. M. Lekkas, “Straight-Path Following for Underactuated Marine Vessels using Deep Reinforcement Learning,” in 11th IFAC Conference on Control Applications in Marine Systems, Robotics, and Vehicles (CAMS), Opatija, Croatia, 2018, vol. 51.10.1016/j.ifacol.2018.09.502
Search in Google Scholar Back to article
21. T. I. Fossen, Handbook of Marine Craft Hydrodynamics and Motion Control, John Wiley, Chichester, UK, doi: 10.1002/9781119994138.
Open DOI Search in Google Scholar Back to article
22. M. Breivik, T. I. Fossen, “Principles of guidance-based path following in 2D and 3D,” in 44th IEEE Conference on Decision Control/European Control Conference (CCDECC), Seville, Spain, 2005.
Search in Google Scholar Back to article
23. T. Liu, Y. L. Hu, and H. Xu, “Deep Reinforcement Learning for Vectored Thruster Autonomous Underwater Vehicle Control,” Complexity, vol. 2021, Apr 2021, doi: 10.1155/2021/6649625.
Open DOI Search in Google Scholar Back to article
24. S. Fujimoto, H. V. Hoof, D. Meger, Addressing Function Approximation Error in Actor-Critic Methods, ICML, 2018.
Search in Google Scholar Back to article

Three-Dimensional Path-Following Control of an Autonomous Underwater Vehicle Based on Deep Reinforcement Learning

Abstract

Paradigm

My account