Reinforcement learning in discrete and continuous domains applied to ship trajectory generation

Rak, Andrzej; Gierusz, Witold

doi:10.2478/v10012-012-0020-8

Abstract

This paper presents the application of the reinforcement learning algorithmsto the task of autonomous determination of the ship trajectory during thein-harbour and harbour approaching manoeuvres. Authors used Markovdecision processes formalism to build up the background of algorithmpresentation. Two versions of RL algorithms were tested in the simulations:discrete (Q-learning) and continuous form (Least-Squares Policy Iteration).The results show that in both cases ship trajectory can be found. Howeverdiscrete Q-learning algorithm suffered from many limitations (mainly curseof dimensionality) and practically is not applicable to the examined task. On the other hand, LSPI gavepromising results. To be fully operational, proposed solution should be extended by taking into accountship heading and velocity and coupling with advanced multi-variable controller.

References

1. Busoniu L., Babuska R., De Schutter B., Ernst D.: Reinforcement Learning and Dynamic Programming UsingFunction Approximators. CRC Press. Automation and ControlEngineering Series. 2010.
Search in Google Scholar
2. Cichosz P.: Learning Systems. WNT. Warszawa 2000. (in Polish).
Search in Google Scholar
3. Gierusz W.: Synthesis of Multi-variable systems of Precise ShipMotion Control Using Selected Robust Control Design Methods. Gdynia Maritime University. 2005. (in Polish).
Search in Google Scholar
4. Gierusz W., Nguyen Cong V., Rak A.: Maneuvering Controland Trajectory Tracking of Very Large Crude Carrier, Ocean Engineering, 2007. No 34, pp. 932-945.10.1016/j.oceaneng.2006.06.003
Search in Google Scholar
5. Kudrewicz J.: Functional Analysis for Control and ElectronicsEngineers. PWN. Warszawa 1976. (in Polish).
Search in Google Scholar
6. Lagoudakis M. G., Parr R.: Least-Squares Policy Iteration. Journal of Machine Learning Research. 2003. Vol. 4, pp. 1107-1149.
Search in Google Scholar
7. Mitsubori K., Kamio T., Tanaka T.: On a Course Determinationbased on the Reinforcement Learning in Maneuvering motion ofa ship with the tidal current effect. International Symposium on Nonlinear Theory and its Applications. Xi’an 2002.
Search in Google Scholar
8. Morawski L., Nguyen Cong V., Rak A.: Full-Mission MarineAutopilot Based on Fuzzy Logic Techniques. Gdynia Maritime University. 2008.
Search in Google Scholar
9. Rak A.: Application of Reinforcement Learning to Ship MotionControl Systems. Zeszyty Naukowe AM w Gdyni. 2009. No 62, pp. 133-140. (in Polish).
Search in Google Scholar
10. Sutton R. S., Barto A. G.: Reinforcement Learning AnIntroduction. MIT Press. 1998.10.1109/TNN.1998.712192
Search in Google Scholar
11. Watkins C. J. C. H., Dayan P.: Q-learning. Machine Learning. 1992. Vol. 8, no 3-4, pp. 279-292.
Search in Google Scholar
12. Zhipeng S., Chen G., Jianbo S.: Reinforcement learning controlfor ship steering based on general fuzzified CMAC. Proc. of 5-th Asian Control Conference. 2005. Vol. 3, pp. 1552-1557.
Search in Google Scholar

Reinforcement learning in discrete and continuous domains applied to ship trajectory generation

Abstract

Paradigm

My account