Optimization Method of Underwater Flapping Foil Propulsion Performance Based on Gaussian Process Regression and Deep Reinforcement Learning

doi:10.16183/j.cnki.jsjtu.2023.188

Abstract

Abstract:

In order to overcome the complexity and variability of underwater working environments, as well as the difficulty of controlling the flapping motion due to the significant nonlinear characteristics and numerous variables involved, a direct exploration approach is proposed to search for the optimal flapping foil propulsion parameters in the environment. The Latin hypercube sampling technique is utilized to obtain the samples of multi-dimensional flapping parameters in actual water pool data, and a Gaussian process regression (GPR) machine learning model is established based on these samples to generalize the working environment. Under different propulsion performance requirements, the TD3 algorithm in deep reinforcement learning (DRL) is trained for maximizing rewards and obtaining the optimal combination of multiple parameter actions in continuous intervals. The experimental results demonstrate that the GPR-TD3 method is capable of learning the globally optimal solution for flapping propulsion in the experimental environment, including maximum speed and maximum efficiency. Furthermore, the accuracy of this optimal solution can be intuitively verified through a two-dimensional contour plot in the GPR. Meanwhile, with 290 sets of real samples provided for any given propulsion speed requirement, the agent can recommend a set of action combinations with an error range of 0.23% to 6.68%, which can provide reference for practical applications.

Key words: underwater flapping foil, Gaussian process regression (GPR), deep reinforcement learning (DRL), propulsion performance optimization

CLC Number:

U664.35

YANG Yinghe, WEI Handi, FAN Dixia, LI Ang. Optimization Method of Underwater Flapping Foil Propulsion Performance Based on Gaussian Process Regression and Deep Reinforcement Learning[J]. Journal of Shanghai Jiao Tong University, 2025, 59(1): 70-78.

Figures/Tables 14

Tab.1

Fig.1

Fig.2

Tab.2

Common kernel functions for GPR

核函数	函数关系式
Matern 3/2	k(x_i, x_j\|θ)=σ²exp $1 + 3 r σ l$ exp $- 3 r σ l$ , r= $(x i - x j) T (x i - x j)$
Matern 5/2	k(x_i, x_j\|θ)=σ² $1 + 3 r σ l + 5 r 2 3 σ 2$ exp $- 5 r σ l$
ARD Matern 3/2	k(x_i, x_j\|θ)=σ²(1+ $3 r$ )exp(- $3 r$ ), r= $∑ m = 1 d (x i m - x j m) 2 σ m 2$
ARD Matern 5/2	k(x_i, x_j\|θ)=σ² $1 + 5 r + 5 r 2 3$ exp(- $5 r$ )
Squared exponential	k(x_i, x_j\|θ)=σ²exp $- 12 (x i - x j) T (x i - x j) σ l 2$
Absolute exponential	k(x_i, x_j\|θ)=σ²exp $- r σ l$ , r= $(x i - x j) T (x i - x j)$

Tab.2

Fig.3

Fig.4

Fig.5

Tab.3

Tab.4

Fig.6

Fig.7

Fig.8

Fig.9

Tab.5

Number of samples and learned actions required by traditional reinforcement learning and GPR-TD3 methods

类别	传统强化学习样本数量	GPR-TD3样本数量	动作向量
推进速度局部最优	9 200	290	[ $0.700 5565 2.860$ ]
推进效率局部最优	6 200	290	[ $0.700 5565 2.837$ ]
推进速度100 mm/s	5 300	290	[ $0.501 5565 3.630$ ]
推进速度80 mm/s	5 900	290	[ $0.500 55 53.615 4.500$ ]
推进速度70 mm/s	4 300	290	[ $0.675 55 52.178 2.734$ ]

Tab.5

References 11

[1]	MANNAM N, KRISHNANKUTTY P, VIJAYAKUMARAN H, et al. Experimental and numerical study of penguin mode flapping foil propulsion system for ships[J]. Journal of Bionic Engineering, 2017, 14(4): 770-780.
[2]	WU X, ZHANG X, TIAN X, et al. A review on fluid dynamics of flapping foils[J]. Ocean Engineering, 2020, 195: 106712.
[3]	ASHRAF M A, YOUNG J, LAI J C A, et al. Oscillation frequency and amplitude effects on plunging airfoil propulsion and flow periodicity[J]. AIAA Journal, 2012, 50(11): 2308-2324.
[4]	KHALID M, AKHTAR I, IMTIAZ H, et al. On the hydrodynamics and nonlinear interaction between fish in tandem configuration[J]. Ocean Engineering, 2018, 157: 108-120.
[5]	DAS A, SHUKLA R K, GOVARDHAN R N. Existence of a sharp transition in the peak propulsive efficiency of a low-Re pitching foil[J]. Journal of Fluid Mechanics, 2016, 800: 307-326.
[6]	MACKOWSKI A W, WILLIAMSON C H K. Direct measurement of thrust and efficiency of an airfoil undergoing pure pitching[J]. Journal of Fluid Mechanics, 2015, 765: 524-543.
[7]	AMIRALAEI M R, ALIGHANBARI H, HASHEMI S M. An investigation into the effects of unsteady parameters on the aerodynamics of a low Reynolds number pitching airfoil[J]. Journal of Fluids and Structures, 2010, 26(6): 979-993.
[8]	SCOTT F, HERKE H, DAVID M. Addressing function approximation error in actor-critic methods[C]// Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden: PMLR, 2018: 1587-1596.
[9]	THAKOR M, KUMAR G, DAS D, et al. Investigation of asymmetrically pitching airfoil at high reduced frequency[J]. Physics of Fluids, 2020, 32(5): 053607.
[10]	CHENG M, JIAO L, YAN P, et al. Prediction of surface residual stress in end milling with Gaussian process regression[J]. Measurement, 2021, 178(11): 109333.
[11]	GARNIER P, VIQUERAT J, RABAULT J, et al. A review on deep reinforcement learning for fluid mechanics[J]. Computers & Fluids, 2021, 225: 104973.

参数	下边界	上边界
横荡(首摇)频率,f/Hz	0.5	0.7
首摇运动幅值,z₀/(°)	25	55
横荡运动幅值,y₀/mm	20	65
运动相位差,ϕ/rad	0.52	5.76

核函数	L_s	N_l	O_n	M	R	P
Matern 3/2	0.000 688	71 523	16	2.128	3.108	0.864
Matern 5/2	0.000 562	11 281	12	12.597	16.011	0.640
ARD Matern 3/2	[2.385 7.976 6.998 4.158]	39 948	1	2.128	3.108	0.864
ARD Matern 5/2	[5.088 1.123 8.345 5.086]	32 307	18	12.597	16.011	0.640
Squared exponential	0.008 38	19 108	7	1.274	1.516	0.956
Absolute exponential	0.007 72	32 667	100	1.006	1.832	0.957

核函数	L_s	N_l	O_n	M	R	P
Matern 3/2	0.689	11 820	13	0.010 9	0.014 6	-0.645
Matern 5/2	0.946	79 233	3	0.010 9	0.014 7	-0.621
Squared exponential	0.001 12	39 666	2	0.010 9	0.014 6	0.250
Absolute exponential	0.043 3	46 725	5	0.010 9	0.014 7	-0.931