Journal of Shanghai Jiao Tong University

Journal of Shanghai Jiaotong University >

2024 , Vol. 58 >Issue 8: 1249 - 1263

DOI: https://doi.org/10.16183/j.cnki.jsjtu.2023.062

Mechanical Engineering

Modeling of Multi-Modal Knowledge Graph for Assembly Process of Wind Turbines with Multi-Source Heterogeneous Data

Expand
  • College of Mechanical Engineering, Donghua University, Shanghai 201620, China

Received date: 2023-02-22

  Revised date: 2023-03-22

  Accepted date: 2023-04-06

  Online published: 2023-04-20

Fold

Abstract

The assembly process information of wind turbines is usually scattered in process documents consisting of multi-modal information, such as 3D models, natural texts, and images. Therefore, the cost of maintaining data and extracting process knowledge is high while the efficiency is low. To solve this problem, a multi-modal knowledge graph-based modeling method for the assembly process knowledge of wind turbines is proposed with multi-source heterogeneous data. First, the concepts in multi-modal process knowledge graph of wind turbine (MPKG-WT) are defined by analyzing the process characteristics of wind turbines to complete the construction of ontology. Then, based on the characteristics of multi-source heterogeneous data and multi-modal information, data analysis, knowledge extraction, and semantic similarity calculation are leveraged to realize the automatic instantiation of the graph. Finally, taking the process data of a wind turbine enterprise as an example, MPKG-WT is constructed and verified by implementing an auxiliary system for process design. The results show that MPKG-WT is more informative than the single-modal graph, and the data in different modals can complement each other, which leads to significant improvements in the efficiency of process design.

Key words： multi-modal knowledge graph; wind turbines; assembly process knowledge; knowledge modeling; knowledge reuse

Cite this article

HU Zhiqiang, LIU Mingfei, LI Qi, LI Xinyu, BAO Jinsong . Modeling of Multi-Modal Knowledge Graph for Assembly Process of Wind Turbines with Multi-Source Heterogeneous Data[J]. Journal of Shanghai Jiaotong University, 2024 , 58(8) : 1249 -1263 . DOI: 10.16183/j.cnki.jsjtu.2023.062

References

[1] 许淳瑶, 葛立超, 冯红翠, 等. 风力发电现状及叶片组成与回收利用综述[J]. 热力发电, 2022, 51(9): 29-41.
  XU Chunyao, GE Lichao, FENG Hongcui, et al. Review on status of wind power generation and composition and recycling of wind turbine blades[J]. Thermal Power Generation, 2022, 51(9): 29-41.
[2] 刘检华, 孙清超, 程晖, 等. 产品装配技术的研究现状、技术内涵及发展趋势[J]. 机械工程学报, 2018, 54(11): 2-28.
  LIU Jianhua, SUN Qingchao, CHENG Hui, et al. The state-of-the-art, connotation and developing trends of the products assembly technology[J]. Journal of Mechanical Engineering, 2018, 54(11): 2-28.
[3] 宋邓强, 周彬, 申兴旺, 等. 面向船舶分段制造过程的动态知识图谱建模方法[J]. 上海交通大学学报, 2021, 55(5): 544-556.
  SONG Dengqiang, ZHOU Bin, SHEN Xingwang, et al. Dynamic knowledge graph modeling method for ship block manufacturing process[J]. Journal of Shanghai Jiao Tong University, 2021, 55(5): 544-556.
[4] XU L D, WANG C G, BI Z M, et al. Object-oriented templates for automated assembly planning of complex products[J]. IEEE Transactions on Automation Science & Engineering, 2014, 11(2): 492-503.
[5] SONG L J, FU Y Y, SU J F, et al. A novel modeling method of the crowdsourcing design process for complex products-based an object-oriented petri net[J]. IEEE Access, 2021, 9: 41430-41440.
[6] 董晨阳, 郑小云, 余建波. 基于过程挖掘与复杂网络集成的制造过程资源建模与关键加工节点识别[J]. 机械工程学报, 2019, 55(3): 169-180.
  DONG Chenyang, ZHENG Xiaoyun, YU Jianbo. Resource modeling of manufacturing process and critical nodes recognition based on the integration of process mining and complex network[J]. Journal of Mechanical Engineering, 2019, 55(3): 169-180.
[7] RUDNITCKAIA J, VENKATACHALAM H S, ESSMANN R, et al. Screening process mining and value stream techniques on industrial manufacturing processes: Process modelling and bottleneck analysis[J]. IEEE Access, 2022, 10: 24203-24214.
[8] DAS S K, SWAIN A K. An ontology-based modelling and reasoning framework for assembly process selection[J]. The International Journal of Advanced Manufacturing Technology, 2022, 120(7): 4863-4887.
[9] 施昭, 曾鹏, 于海斌. 基于本体的制造知识建模方法及其应用[J]. 计算机集成制造系统, 2018, 24(11): 2653-2664.
  SHI Zhao, ZENG Peng, YU Haibin. Ontology-based modeling method for manufacturing knowledge and its application[J]. Computer Integrated Manufacturing Systems, 2018, 24(11): 2653-2664.
[10] 李秀玲, 张树生, 黄瑞, 等. 面向工艺重用的工艺知识图谱构建方法[J]. 西北工业大学学报, 2019, 37(6): 1174-1183.
  LI Xiuling, ZHANG Shusheng, HUANG Rui, et al. Process knowledge graph construction method for process reuse[J]. Journal of Northwestern Polytechnical University, 2019, 37(6): 1174-1183.
[11] 吴闯, 张亮, 唐希浪, 等. 航空发动机润滑系统故障知识图谱构建及应用[J]. 北京航空航天大学学报, 2024, 50(4): 1336-1346.
  WU Chuang, ZHANG Liang, TANG Xilang, et al. Construction and application of fault knowledge graph for aero-engine lubrication system[J]. Journal of Beijing University of Aeronautics and Astronautics, 2024, 50(4): 1336-1346.
[12] 顾星海, 鲍劲松, 吕超凡. 基于知识图谱的装配语义信息建模[J]. 航空制造技术, 2021, 64(4): 74-81.
  GU Xinghai, BAO Jinsong, Lü Chaofan. Assembly semantic information modeling based on knowledge graph[J]. Aeronautical Manufacturing Technology, 2021, 64(4): 74-81.
[13] ZHU X R, LI Z X, WANG X D, et al. Multi-modal knowledge graph construction and application: A survey[J]. IEEE Transactions on Knowledge & Data Engineering, 2024, 36(2): 715-735.
[14] KANNAN A V, FRADKIN D, AKROTIRIANAKIS I, et al. Multimodal knowledge graph for deep learning papers and code[C]// Proceedings of the 29th ACM International Conference on Information & Knowledge Management. Virtual Event, Ireland: ACM, 2020: 3417-3420.
[15] 李星原, 汪鹏, 申牧, 等. 癫痫病相关论文多模态知识图谱的构建初探[J]. 北京邮电大学学报, 2022, 45(4): 19-24.
  LI Xingyuan, WANG Peng, SHEN Mu, et al. Construction of multi-modal knowledge graph for epilepsy related papers[J]. Journal of Beijing University of Posts & Telecommunications, 2022, 45(4): 19-24.
[16] CLARK K, LUONG M T, LE Q V, et al. ELECTRA: Pre-training text encoders as discriminators rather than generators[DB/OL]. (2020-03-24)[2023-02-01]. https://arxiv.org/abs/2003.10555.
[17] HUANG Z H, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging[DB/OL]. (2015-08-09)[2023-02-01]. https://arxiv.org/abs/1508.01991.
[18] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 770-778.
[19] 于勇, 周阳, 曹鹏, 等. 基于MBD模型的工序模型构建方法[J]. 浙江大学学报(工学版), 2018, 52(6): 1025-1034.
  YU Yong, ZHOU Yang, CAO Peng, et al. In-process model construction method based on model-based definition model[J]. Journal of Zhejiang University (Engineering Science), 2018, 52(6): 1025-1034.
[20] ZHANG P F, LI T R, WANG G Q, et al. Multi-source information fusion based on rough set theory: A review[J]. Information Fusion, 2021, 68: 85-117.
[21] REIMERS N, GUREVYCH I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks[C]// Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong, China: Association for Computational Linguistics, 2019: 3982-3992.
[22] ARTSTEIN R, POESIO M. Inter-coder agreement for computational linguistics[J]. Computational Linguistics, 2008, 34(4): 555-596.
Options
Outlines

/

〈 〉