Modeling of Multi-Modal Knowledge Graph for Assembly Process of Wind Turbines with Multi-Source Heterogeneous Data

doi:10.16183/j.cnki.jsjtu.2023.062

Abstract

Abstract:

The assembly process information of wind turbines is usually scattered in process documents consisting of multi-modal information, such as 3D models, natural texts, and images. Therefore, the cost of maintaining data and extracting process knowledge is high while the efficiency is low. To solve this problem, a multi-modal knowledge graph-based modeling method for the assembly process knowledge of wind turbines is proposed with multi-source heterogeneous data. First, the concepts in multi-modal process knowledge graph of wind turbine (MPKG-WT) are defined by analyzing the process characteristics of wind turbines to complete the construction of ontology. Then, based on the characteristics of multi-source heterogeneous data and multi-modal information, data analysis, knowledge extraction, and semantic similarity calculation are leveraged to realize the automatic instantiation of the graph. Finally, taking the process data of a wind turbine enterprise as an example, MPKG-WT is constructed and verified by implementing an auxiliary system for process design. The results show that MPKG-WT is more informative than the single-modal graph, and the data in different modals can complement each other, which leads to significant improvements in the efficiency of process design.

Key words: multi-modal knowledge graph, wind turbines, assembly process knowledge, knowledge modeling, knowledge reuse

CLC Number:

TP391

HU Zhiqiang, LIU Mingfei, LI Qi, LI Xinyu, BAO Jinsong. Modeling of Multi-Modal Knowledge Graph for Assembly Process of Wind Turbines with Multi-Source Heterogeneous Data[J]. Journal of Shanghai Jiao Tong University, 2024, 58(8): 1249-1263.

Figures/Tables 24

Fig.1

Fig.2

Tab.1

Tab.2

Tab.3

Fig.3

Fig.4

Fig.5

Tab.4

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Tab.5

Tab.6

Fig.13

Fig.14

Tab.7

Fig.15

Tab.8

Fig.16

References 22

[1]	许淳瑶, 葛立超, 冯红翠, 等. 风力发电现状及叶片组成与回收利用综述[J]. 热力发电, 2022, 51(9): 29-41.
	XU Chunyao, GE Lichao, FENG Hongcui, et al. Review on status of wind power generation and composition and recycling of wind turbine blades[J]. Thermal Power Generation, 2022, 51(9): 29-41.
[2]	刘检华, 孙清超, 程晖, 等. 产品装配技术的研究现状、技术内涵及发展趋势[J]. 机械工程学报, 2018, 54(11): 2-28. doi: 10.3901/JME.2018.11.002
	LIU Jianhua, SUN Qingchao, CHENG Hui, et al. The state-of-the-art, connotation and developing trends of the products assembly technology[J]. Journal of Mechanical Engineering, 2018, 54(11): 2-28. doi: 10.3901/JME.2018.11.002
[3]	宋邓强, 周彬, 申兴旺, 等. 面向船舶分段制造过程的动态知识图谱建模方法[J]. 上海交通大学学报, 2021, 55(5): 544-556. doi: 10.16183/j.cnki.jsjtu.2020.241
	SONG Dengqiang, ZHOU Bin, SHEN Xingwang, et al. Dynamic knowledge graph modeling method for ship block manufacturing process[J]. Journal of Shanghai Jiao Tong University, 2021, 55(5): 544-556.
[4]	XU L D, WANG C G, BI Z M, et al. Object-oriented templates for automated assembly planning of complex products[J]. IEEE Transactions on Automation Science & Engineering, 2014, 11(2): 492-503.
[5]	SONG L J, FU Y Y, SU J F, et al. A novel modeling method of the crowdsourcing design process for complex products-based an object-oriented petri net[J]. IEEE Access, 2021, 9: 41430-41440.
[6]	董晨阳, 郑小云, 余建波. 基于过程挖掘与复杂网络集成的制造过程资源建模与关键加工节点识别[J]. 机械工程学报, 2019, 55(3): 169-180. doi: 10.3901/JME.2019.03.169
	DONG Chenyang, ZHENG Xiaoyun, YU Jianbo. Resource modeling of manufacturing process and critical nodes recognition based on the integration of process mining and complex network[J]. Journal of Mechanical Engineering, 2019, 55(3): 169-180. doi: 10.3901/JME.2019.03.169
[7]	RUDNITCKAIA J, VENKATACHALAM H S, ESSMANN R, et al. Screening process mining and value stream techniques on industrial manufacturing processes: Process modelling and bottleneck analysis[J]. IEEE Access, 2022, 10: 24203-24214.
[8]	DAS S K, SWAIN A K. An ontology-based modelling and reasoning framework for assembly process selection[J]. The International Journal of Advanced Manufacturing Technology, 2022, 120(7): 4863-4887.
[9]	施昭, 曾鹏, 于海斌. 基于本体的制造知识建模方法及其应用[J]. 计算机集成制造系统, 2018, 24(11): 2653-2664.
	SHI Zhao, ZENG Peng, YU Haibin. Ontology-based modeling method for manufacturing knowledge and its application[J]. Computer Integrated Manufacturing Systems, 2018, 24(11): 2653-2664.
[10]	李秀玲, 张树生, 黄瑞, 等. 面向工艺重用的工艺知识图谱构建方法[J]. 西北工业大学学报, 2019, 37(6): 1174-1183.
	LI Xiuling, ZHANG Shusheng, HUANG Rui, et al. Process knowledge graph construction method for process reuse[J]. Journal of Northwestern Polytechnical University, 2019, 37(6): 1174-1183.
[11]	吴闯, 张亮, 唐希浪, 等. 航空发动机润滑系统故障知识图谱构建及应用[J]. 北京航空航天大学学报, 2024, 50(4): 1336-1346.
	WU Chuang, ZHANG Liang, TANG Xilang, et al. Construction and application of fault knowledge graph for aero-engine lubrication system[J]. Journal of Beijing University of Aeronautics and Astronautics, 2024, 50(4): 1336-1346.
[12]	顾星海, 鲍劲松, 吕超凡. 基于知识图谱的装配语义信息建模[J]. 航空制造技术, 2021, 64(4): 74-81.
	GU Xinghai, BAO Jinsong, LÜ Chaofan. Assembly semantic information modeling based on knowledge graph[J]. Aeronautical Manufacturing Technology, 2021, 64(4): 74-81.
[13]	ZHU X R, LI Z X, WANG X D, et al. Multi-modal knowledge graph construction and application: A survey[J]. IEEE Transactions on Knowledge & Data Engineering, 2024, 36(2): 715-735.
[14]	KANNAN A V, FRADKIN D, AKROTIRIANAKIS I, et al. Multimodal knowledge graph for deep learning papers and code[C]// Proceedings of the 29th ACM International Conference on Information & Knowledge Management. Virtual Event, Ireland: ACM, 2020: 3417-3420.
[15]	李星原, 汪鹏, 申牧, 等. 癫痫病相关论文多模态知识图谱的构建初探[J]. 北京邮电大学学报, 2022, 45(4): 19-24. doi: 10.13190/j.jbupt.2021-187
	LI Xingyuan, WANG Peng, SHEN Mu, et al. Construction of multi-modal knowledge graph for epilepsy related papers[J]. Journal of Beijing University of Posts & Telecommunications, 2022, 45(4): 19-24.
[16]	CLARK K, LUONG M T, LE Q V, et al. ELECTRA: Pre-training text encoders as discriminators rather than generators[DB/OL]. (2020-03-24)[2023-02-01]. https://arxiv.org/abs/2003.10555.
[17]	HUANG Z H, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging[DB/OL]. (2015-08-09)[2023-02-01]. https://arxiv.org/abs/1508.01991.
[18]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 770-778.
[19]	于勇, 周阳, 曹鹏, 等. 基于MBD模型的工序模型构建方法[J]. 浙江大学学报(工学版), 2018, 52(6): 1025-1034.
	YU Yong, ZHOU Yang, CAO Peng, et al. In-process model construction method based on model-based definition model[J]. Journal of Zhejiang University (Engineering Science), 2018, 52(6): 1025-1034.
[20]	ZHANG P F, LI T R, WANG G Q, et al. Multi-source information fusion based on rough set theory: A review[J]. Information Fusion, 2021, 68: 85-117.
[21]	REIMERS N, GUREVYCH I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks[C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong, China: Association for Computational Linguistics, 2019: 3982-3992.
[22]	ARTSTEIN R, POESIO M. Inter-coder agreement for computational linguistics[J]. Computational Linguistics, 2008, 34(4): 555-596.

多模态实体类型	实体类别	实体定义
E_W	AC	利用工具对原材料或半成品按照设计预期要求进行加工及处理的方法和过程
E_W	AP	在同一地点利用同一设备完成的装配工作
E_W	AS	在同一位置用同一工具,不改变工作方法完成的装配工作
E_W	AE	组成装配机器或产品的最小单元
E_W	AT	装配过程中用到的工具
E_V	VC	可视化装配零件的图像
E_V	VS	指导装配过程的图像
E_M	MS	三维模型自身及其层级结构
E_M	MF	三维模型的几何组成单元

关系类型	关系类别	关系定义	联结的模态
R_st	HasImg	表示文本实体具有的相关图像	文本-图像
R_st	HasMod	表示文本或图像实体具有的相关三维模型	文本或图像-模型
R_st	HasGF	三维模型具有几何特征	模型-模型
R_st	HasPart	产品的装配关系,表示某装配体包含子装配体或是零件	模型-模型
R_tm	StepTo	表示同一工艺下不同工序之间的先后关系	文本-文本
R_tm	Next	表示同一工序下不同工步之间的先后关系	文本-文本
R_op	Prepare	表示进入工作之前需要准备的零件、工具	文本-文本
R_op	HasTool	表示操作过程中需要用到某种工具	文本-文本
R_op	Check	表示操作过程中的检查步骤	文本-文本
R_op	Operate	动作统一表示,表示对零件采取某种动作	文本-文本
R_op	On/ Under/ Before/ After/Sideof	位置关系表示,表示不同装配实体之间的相对位置	文本-文本

实体类别	属性名
AC	工艺名称;工艺规范
AP	工序名称;耗时;所属工艺
AS	工步名称;所属工序
AE	零件名称;规格;代号
AT	工具名称;规格
VC	URL;尺寸;描述;向量表示
VS	URL;尺寸;描述;向量表示
MS	URL;名称;组成结构
MF	名称;模型材质

组合类别	图谱关系
AE/AT,AE/AT + POB ATT	On/Under/Before/After/Sideof
AS/AE,AT + SBV/IOB/FOB	HasTool
AS,AE + SBV/IOB/FOB	Operate
AC,AP或AP,AP	StepTo
AP,AS或AS,AS	Next
v:准备/预备/备用/筹备	Prepare
v:检查/检测/核验/查验/查看+	Check
nd:上/顶部+ POB	On
nd:下/底部+ POB	Under
nd:前+ POB	Before
nd:后+ POB	After
nd:旁边/周围/一侧/一边+ POB	Sideof

模型	实验参数	取值
Electra-BiLSTM-CRF	词嵌入维度	768
Electra-BiLSTM-CRF	迭代次数	60
Electra-BiLSTM-CRF	学习率	0.0001
Electra-BiLSTM-CRF	DropOut	0.5
Electra-BiLSTM-CRF	Batch Size	32
Electra-BiLSTM-CRF	最大句长	256
Resnet50	迭代次数	30
Resnet50	学习率	0.001
Resnet50	Batch Size	32
跨模态对齐模型	迭代次数	80
跨模态对齐模型	输入嵌入维度	1768
Sentence-BERT	相似度阈值	0.6