Self-Adaptive LSAC-PID Approach Based on Lyapunov Reward Shaping for Mobile Robots

doi:10.1007/s12204-023-2631-x

Abstract

Abstract: In order to solve the control problem of multiple-input multiple-output (MIMO) systems in complex and variable control environments, a model-free adaptive LSAC-PID method based on deep reinforcement learning (RL) is proposed in this paper for automatic control of mobile robots. According to the environmental feedback, the RL agent of the upper controller outputs the optimal parameters to the lower MIMO PID controllers, which can realize the real-time PID optimal control. First, a model-free adaptive MIMO PID hybrid control strategy is presented to realize real-time optimal tuning of control parameters in terms of soft-actor-critic (SAC) algorithm, which is state-of-the-art RL algorithm. Second, in order to improve the RL convergence speed and the control performance, a Lyapunov-based reward shaping method for off-policy RL algorithm is designed, and a self-adaptive LSAC-PID tuning approach with Lyapunov-based reward is then determined. Through the policy evaluation and policy improvement of the soft policy iteration, the convergence and optimality of the proposed LSAC-PID algorithm are proved mathematically. Finally, based on the proposed reward shaping method, the reward function is designed to improve the system stability for the line-following robot. The simulation and experiment results show that the proposed adaptive LSAC-PID approach has good control performance such as fast convergence speed, high generalization and high real-time performance, and achieves real-time optimal tuning of MIMO PID parameters without the system model and control loop decoupling.

Key words: multiple-input multiple-output (MIMO), PID tuning, reinforcement learning (RL), Lyapunov-based reward shaping, soft actor-critic (SAC), mobile robot

摘要： 为了解决复杂多变环境下多输入多输出（MIMO）系统的控制问题，本文针对移动机器人的自动控制提出了一种基于深度强化学习（RL）的无模型自适应LSAC-PID方法。根据环境反馈，作为上层控制器的RL智能体将最优参数输出给下层MIMO PID控制器，实现实时最优控制。首先，提出了一种无模型自适应MIMO PID混合控制策略，通过强化学习中最先进的软演员评论家（SAC）算法来实现控制参数的实时最优整定。其次，为了提高RL的收敛速度和控制性能，基于李雅普诺夫理论设计了一种适用于非策略RL算法的奖励塑造方法，并基于此确定了一种基于李雅普诺夫奖励的自适应LSAC-PID整定方法。通过软策略迭代的策略评估和策略改进，从数学上证明了提出的LSAC-PID算法的收敛性和最优性。最后，基于所提出的奖励塑造方法，针对循线机器人系统设计了奖励函数来提高其稳定性。仿真和实验结果表明，在不依赖控制系统模型和解耦控制回路的情况下，提出的自适应LSAC-PID方法可以实现MIMO PID参数的实时最优整定，具有收敛速度快、泛化性强和实时性高等优点。

关键词: 多输入多输出，PID整定，强化学习，基于李雅普诺夫奖励塑造，软演员评论家，移动机器人

CLC Number:

TP273
TP242

YU Xinyi, XU Siyu, FAN Yuehai, OU Linlin. Self-Adaptive LSAC-PID Approach Based on Lyapunov Reward Shaping for Mobile Robots[J]. J Shanghai Jiaotong Univ Sci, 2025, 30(6): 1085-1102.

References

[1] FU Z, CHEN Z P, ZHENG C, et al. A cable-tunnel inspecting robot for dangerous environment [J]. International Journal of Advanced Robotic Systems, 2008, 5(3): 243-248.
[2] NAGATANI K, KIRIBAYASHI S, OKADA Y, et al. Redesign of rescue mobile robot Quince [C]//2011 IEEE International Symposium on Safety, Security, and Rescue Robotics. Kyoto: IEEE, 2011: 13-18.
[3] TUBA E, STRUMBERGER I, ZIVKOVIC D, et al. Mobile robot path planning by improved brain storm optimization algorithm [C]//2018 IEEE Congress on Evolutionary Computation. Rio de Janeiro: IEEE, 2018: 1-8.
[4] WANG Q G, NIE Z Y. PID control for MIMO processes [M]//PID control in the third millennium. London: Springer, 2012: 177-204.
[5] KATEBI R. Robust multivariable tuning methods [M]//PID control in the third millennium. London: Springer, 2012: 255-280.
[6] GIL P, LUCENA C, CARDOSO A, et al. Gain tuning of fuzzy PID controllers for MIMO systems: A performance-driven approach [J]. IEEE Transactions on Fuzzy Systems, 2015, 23(4): 757-767.
[7] BOYD S, HAST M, ASTROM K J. MIMO PID tuning via iterated LMI restriction [J]. International Journal of Robust and Nonlinear Control, 2016, 26(8): 1718-1731.
[8] SONG Y D, HUANG X C, WEN C Y. Robust adaptive fault-tolerant PID control of MIMO nonlinear systems with unknown control direction [J]. IEEE Transactions on Industrial Electronics, 2017, 64(6): 4876-4884.
[9] HOWELL M N, BEST M C. On-line PID tuning for engine idle-speed control using continuous action reinforcement learning automata [J]. Control Engineering Practice, 2000, 8(2): 147-154.
[10] CARLUCHO I, DE PAULA M, VILLAR S A, et al. Incremental Q-learning strategy for adaptive PID control of mobile robots [J]. Expert Systems With Applications, 2017, 80: 183-199.

[11] CARLUCHO I, DE PAULA M, ACOSTA G G, et al. Double Q-PID algorithm for mobile robot control [J]. Expert Systems With Applications, 2019, 137: 292-307.

[12] KONDA V, TSITSIKLIS J. Actor-critic algorithms [C]//12th International Conference on Neural Information Processing Systems. Denver: NIPS, 1999: 1008-1014.
[13] WANG X S, CHENG Y H, WEI S, et al. A proposal of adaptive PID controller based on reinforcement learning [J]. Journal of China University of Mining and Technology, 2007, 17(1): 40-44.
[14] AKBARIMAJD A. Reinforcement learning adaptive PID controller for an under-actuated robot arm [J]. International Journal of Integrated Engineering, 2015, 7(2): 20-27.
[15] CARLUCHO I, DE PAULA M, ACOSTA G G, et al. An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots [J]. ISA Transactions, 2020, 102: 280-294.
[16] HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor [C]//35th International Conference on Machine Learning. Stockholm: IMLS, 2018: 1861-1870.
[17] YU X Y, FAN Y H, XU S Y, et al. A self-adaptive SACPID control approach based on reinforcement learning for mobile robots [J]. International Journal of Robust and Nonlinear Control, 2022, 32(18): 9625-9643.
[18] WEISZ G, BUDZIANOWSKI P, SU P H, et al. Sample efficient deep reinforcement learning for dialogue systems with large action spaces [J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018, 26(11): 2083-2097.
[19] YE D H, CHEN G B, ZHANG W, et al. Towards playing full MOBA games with deep reinforcement learning [C]//34th Conference on Neural Information Processing Systems. Vancouver: NIPS, 2020: 1-12.
[20] YE F, CHENG X X, WANG P, et al. Automated lane change strategy using proximal policy optimizationbased deep reinforcement learning [C]//2020 IEEE Intelligent Vehicles Symposium. Las Vegas: IEEE, 2020: 1746-1752.
[21] NG A Y, HARADA D, RUSSELL S. Policy invariance under reward transformations: Theory and application to reward shaping [C]//16th International Conference on Machine Learning. Bled: IMLS, 1999: 278-287.
[22] ASMUTH J, LITTMAN M L, ZINKOV R. Potentialbased shaping in model-based reinforcement learning [C]//23rd AAAI Conference on Artificial Intelligence. Chicago: AAAI, 2008: 604-609.
[23] DEVLIN S M, KUDENKO D. Dynamic potentialbased reward shaping [C]//11th International Conference on Autonomous Agents and Multiagent Systems. Valenci: IFAAMAS, 2012: 433-440.
[24] DEVLIN S, YLINIEMI L, KUDENKO D, et al. Potential-based difference rewards for multiagent reinforcement learning [C]//13th International Conference on Autonomous Agents and Multi-Agent Systems. Paris: IFAAMAS, 2014: 165-172.
[25] WIEWIORA E, COTTRELL G W, ELKAN C. Principled methods for advising reinforcement learning agents [C]//20th International Conference on Machine Learning. Washington: IMLS, 2003: 792-799.
[26] HARUTYUNYAN A, DEVLIN S, VRANCX P, et al. Expressing arbitrary reward functions as potentialbased advice [J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2015, 29(1): 2652-2658.
[27] BRYS T, HARUTYUNYAN A, SUAY H B, et al. Reinforcement learning from demonstration through shaping [C]//24th International Conference on Artificial Intelligence. Buenos Aires: AAAI, 2015: 3352-3358.
[28] DONG Y L, et al. Principled reward shaping for reinforcement learning via Lyapunov stability theory [J]. Neurocomputing, 2020, 393: 83-90.
[29] SUTTON R S, BARTO A G. Reinforcement learning: An introduction [M]. 2nd ed. Cambridge: MIT Press, 2018.
[30] BALTES J, LIN Y M. Path tracking control of nonholonomic car-like robot with reinforcement learning [M]//RoboCup-99: Robot Soccer World Cup III. Berlin, Heidelberg: Springer, 1999: 162-173.
[31] SAADATMAND S, AZIZI S, KAVOUSI M, et al. Autonomous control of a line follower robot using a Qlearning controller [C]//2020 10th Annual Computing and Communication Workshop and Conference. Las Vegas: IEEE, 2020: 556-561.
[32] MARTINSEN A B, LEKKAS A M. Curved path following with deep reinforcement learning: Results from three vessel models [C]//OCEANS 2018 MTS/IEEE Charleston. Charleston: IEEE, 2018: 1-8.
[33] KUMAR V, NAKRA B C, MITTAL A P. A review on classical and fuzzy PID controllers [J]. International Journal of Intelligent Control and Systems, 2011, 16(3): 170-181.
[34] XU Q, KAN J M, CHEN S N, et al. Fuzzy PID based trajectory tracking control of mobile robot and its simulation in simulink [J]. International Journal of Control and Automation, 2014, 7(8): 233-244.
[35] TIEP D K, LEE K, IM D Y, et al. Design of fuzzy-PID controller for path tracking of mobile robot with differential drive [J]. International Journal of Fuzzy Logic and Intelligent Systems, 2018, 18(3): 220-228.
[36] FURUKAWA S, KONDO S, TAKANISHI A, et al. Radial basis function neural network based PID control for quad-rotor flying robot [C]//2017 17th International Conference on Control, Automation and Systems. Jeju: IEEE, 2017: 580-584.
[37] LIU Q, LI D, GE S S, et al. Adaptive bias RBF neural network control for a robotic manipulator [J]. Neurocomputing, 2021, 447: 213-223.