An Information Based Approach to Stochastic Control Problems

Bania, Piotr

doi:10.34768/amcs-2020-0002

Abstract

An information based method for solving stochastic control problems with partial observation is proposed. First, information-theoretic lower bounds of the cost function are analysed. It is shown, under rather weak assumptions, that reduction in the expected cost with closed-loop control compared with the best open-loop strategy is upper bounded by a non-decreasing function of mutual information between control variables and the state trajectory. On the basis of this result, an information based control (IBC) method is developed. The main idea of IBC consists in replacing the original control task by a sequence of control problems that are relatively easy to solve and such that information about the system state is actively generated. Two examples of the IBC operation are given. It is shown that the method is able to find an optimal solution without using dynamic programming at least in these examples. Hence the computational complexity of IBC is substantially smaller than that of dynamic programming, which is the main advantage of the proposed method.

References

Alpcan, T., Shames, I., Cantoni, M. and Nair, G. (2015). An information-based learning approach to dual control, IEEE Transactions on Neural Networks and Learning Systems26(11): 2736–2748.10.1109/TNNLS.2015.239212225730828
Search in Google Scholar Back to article
Alspach, D. and Sorenson, H. (1972). Nonlinear Bayesian estimation using Gaussian sum approximations, IEEE Transactions on Automatic Control17(4): 439–448.10.1109/TAC.1972.1100034
Search in Google Scholar Back to article
Åström, K. and Wittenmark, B. (1995). Adaptive Control, Second Edition, Dover Publications, NewYork, NY.
Search in Google Scholar Back to article
Banek, T. (2010). Incremental value of information for discrete-time partially observed stochastic systems, Control and Cybernetics39(3): 769–781.
Search in Google Scholar Back to article
Bania, P. (2017). Simple example of dual control problem with almost analytical solution, Proceedings of the 19th Polish Control Conference, Krakow, Poland, pp. 55–64, DOI: 10.1007/978-3-319-60699-6-7.
Search in Google Scholar Back to article
Bania, P. (2018). Example for equivalence of dual and information based optimal control, International Journal of Control38(5): 787–803, DOI: 10.1080/00207179.2018.1436775.10.1080/00207179.2018.1436775
Search in Google Scholar Back to article
Bania, P. (2019). Bayesian input design for linear dynamical model discrimination, Entropy21(4): 1–13, DOI: 10.3390/e21040351.10.3390/e21040351751483533267065
Search in Google Scholar Back to article
Bania, P. and Baranowski, J. (2016). Field Kalman filter and its approximation, 55th IEEE Conference on Decision and Control, Las Vegas, NV, USA, pp. 2875–2880, DOI: 10.1109/CDC.2016.7798697.10.1109/CDC.2016.7798697
Search in Google Scholar Back to article
Bania, P. and Baranowski, J. (2017). Bayesian estimator of a faulty state: Logarithmic odds approach, 22nd International Conference on Methods and Models in Automation and Robotics (MMAR), Miedzyzdroje, Poland, pp. 253–257, DOI: 10.1109/MMAR.2017.8046834.10.1109/MMAR.2017.8046834
Search in Google Scholar Back to article
Baranowski, J., Bania, P., Prasad, I. and T., C. (2017). Bayesian fault detection and isolation using field Kalman filter, EURASIP Journal on Advances in Signal Processing79(1), DOI: 10.1186/s13634-017-0514-8.10.1186/s13634-017-0514-8
Search in Google Scholar Back to article
BarShalom, Y. and Tse, E. (1976). Caution, probing, and the value of information in the control of uncertain systems, Annals of Economic and Social Measurement5(3): 323–337.
Search in Google Scholar Back to article
Brechtel, S., Gindele, T. and Dillmann, R. (2013). Solving continuous POMDPs: Value iteration with incremental learning of an efficient space representation, Proceedings of the 30th International Conference on International Conference on Machine Learning, ICML’13, Atlanta, GA, USA, Vol. 28, pp. III–370–III–378.
Search in Google Scholar Back to article
Byrd, R., Hansen, S., Nocedal, J. and Singer, Y. (2016). A stochastic quasi-Newton method for large-scale optimization, SIAM Journal on Optimization26(2): 1008–1031.10.1137/140954362
Search in Google Scholar Back to article
Cover, T.M. and Thomas, J.A. (2006). Elements of Information Theory, Second Edition, John Wiley & Sons, Inc., Hoboken, NJ.
Search in Google Scholar Back to article
Delvenne, J.C. and Sandberg, H. (2013). Towards a thermodynamics of control: Entropy, energy and Kalman filtering, 52nd IEEE Conference on Decision and Control, Florence, Italy, pp. 3109–3114.
Search in Google Scholar Back to article
Dolgov, M. (2017). Approximate Stochastic Optimal Control of Smooth Nonlinear Systems and Piecewise Linear Systems, PhD thesis, Karlsruhe Institute of Technology, Karlsruhe.
Search in Google Scholar Back to article
Feldbaum, A.A. (1965). Optimal Control Systems, Academic Press, New York, NY.
Search in Google Scholar Back to article
Filatov, N.M. and Unbehauen, H. (2004). Adaptive Dual Control: Theory and Applications, Springer-Verlag, Berlin/Heidelberg.10.1007/b96083
Search in Google Scholar Back to article
Hijab, O. (1984). Entropy and dual control, 23rd Conference on Decision and Control, Las Vegas, NV, USA, pp. 45–50.
Search in Google Scholar Back to article
Huang, C., Ho, D.W.C., Lu, J. and Kurths, J. (2012). Partial synchronization in stochastic dynamical networks with switching communication channels, Chaos: An Interdisciplinary Journal of Nonlinear Science22(2): 023108, DOI: 10.1063/1.3702576.10.1063/1.370257622757515
Search in Google Scholar Back to article
Jiang, H. (2017). Uniform convergence rates for kernel density estimation, Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 1694–1703.
Search in Google Scholar Back to article
Joe, H. (1989). Estimation of entropy and other functionals of a multivariate density, Annals of the Institute of Statistical Mathematics41(4): 683–697.10.1007/BF00057735
Search in Google Scholar Back to article
Kolchinsky, A. and Tracey, B.D. (2017). Estimating mixture entropy with pairwise distances, Entropy19(361): 1–17.10.3390/e19070361
Search in Google Scholar Back to article
Korbicz, J., Koscielny, J.M., Kowalczuk, Z. and Cholewa, W. (2004). Fault Diagnosis: Models, Artificial Intelligence, Applications, Springer-Verlag, Berlin/Heidelberg.10.1007/978-3-642-18615-8
Search in Google Scholar Back to article
Kozlowski, E. and Banek, T. (2011). Active learning in discrete time stochastic systems, in J. Jozefczyk and D. Orski (Eds), Knowledge-Based Intelligent System Advancements: Systemic and Cybernetic Approaches, Information Science References, New York, NY, pp. 350–371.
Search in Google Scholar Back to article
Mitter, S.K. and Newton, N.J. (2005). Information and entropy flow in the Kalman–Bucy filter, Journal of Statistical Physics118(1): 145–176.10.1007/s10955-004-8781-9
Search in Google Scholar Back to article
Porta, J.M., Vlassis, N., Spaan, M.T. and Poupart, P. (2006). Point-based value iteration for continuous POMDPs, Journal of Machine Learning Research7(1): 2329–2367.
Search in Google Scholar Back to article
Sagawa, T. and Ueda, M. (2013). Role of mutual information in entropy production under information exchanges, New Journal of Physics15(125012): 2–23.10.1088/1367-2630/15/12/125012
Search in Google Scholar Back to article
Saridis, G.N. (1988). Entropy formulation of optimal and adaptive control, IEEE Transactions on Automatic Control33(8): 713–721.10.1109/9.1287
Search in Google Scholar Back to article
Särkä, S. (2013). Bayesian Filtering and Smoothing, Cambridge University Press, New York, NY.10.1017/CBO9781139344203
Search in Google Scholar Back to article
Taticonda, S. and Mitter, S.K. (2004). Control under communication constraints, IEEE Transactions on Automatic Control49(7): 1056–1068.10.1109/TAC.2004.831187
Search in Google Scholar Back to article
Thrun, S. (2000). Monte Carlo POMDPs, in S. Solla et al. (Eds), Advances in Neural Information Processing Systems, MIT Press, Cambridge, MA, pp. 1064–1070.
Search in Google Scholar Back to article
Touchette, H. (2000). Information-theoretic Aspects in the Control of Dynamical Systems Master’s thesis, MIT, Cambridge, MA, https://pdfs.semanticscholar.org/c915/088f514d937f5d1c666221c95d731532101e.pdf.
Search in Google Scholar Back to article
Touchette, H. and Lloyd, S. (2000). Information-theoretic limits of control, Physical Review Letters84(6): 1156–1159.10.1103/PhysRevLett.84.115611017467
Search in Google Scholar Back to article
Touchette, H. and Lloyd, S. (2004). Information-theoretic approach to the study of control systems, Physica A331(1): 140–172.10.1016/j.physa.2003.09.007
Search in Google Scholar Back to article
Tsai, Y.A., Casiello, F.A. and Loparo, K.A. (1992). Discrete-time entropy formulation of optimal and adaptive control problems, IEEE Transactions on Automatic Control37(7): 1083–1088.10.1109/9.148379
Search in Google Scholar Back to article
Tse, E. (1974). Adaptive dual control methods, Annals of Economic and Social Measurement3(1): 65–82.
Search in Google Scholar Back to article
Uciński, D. (2004). Optimal Measurement Methods for Distributed Parameter System Identification, CRC Press, Boca Raton, FL.10.1201/9780203026786
Search in Google Scholar Back to article
Zabczyk, J. (1996). Chance and Decision. Stochastic Control in Discrete Time, Quaderni Scuola Normale di Pisa, Pisa.
Search in Google Scholar Back to article
Zhao, D., Liu, J., Wu, R., Cheng, D. and Tang, X. (2019). An active exploration method for data efficient reinforcement learning, International Journal of Applied Mathematics and Computer Science29(2): 351–362, DOI: 10.2478/amcs-2019-0026.10.2478/amcs-2019-0026
Search in Google Scholar Back to article

An Information Based Approach to Stochastic Control Problems

Abstract

Paradigm

My account