融合LLM与TLP的跨工艺文档多模态知识图谱建模方法

展开
  • 1.东华大学 计算机科学与技术学院,上海  201620;

    2.东华大学 机械工程学院,上海  201620
黄好阳(2001—),硕士生,从事知识图谱建模研究
鲍劲松,教授,博士生导师;E-mail:bao@dhu.edu.cn

网络出版日期: 2025-05-28

基金资助

国家自然科学基金(5247551)资助项目

A Multimodal Knowledge Graph Modeling Method for Cross-Process Documents via Integration of LLM and TLP

Expand
  • 1.School of Computer Science and Technology, Donghua University, Shanghai 201620, China;

    2.School of Mechanical Engineering, Donghua University, Shanghai 201620, China

Online published: 2025-05-28

摘要

针对汽轮机叶片加工工艺领域多模态数据分散、文档间缺乏联系导致的数据管理效率低下和工艺复用困难问题,提出一种基于大语言模型(LLM)和技术语言处理(TLP)的多模态跨文档图谱建模技术。首先设计多模态图谱本体框架,然后利用TLP技术解决工艺文档中复杂术语、非标准化文本和跨文档语义不一致性问题;针对LLM在理解特定工艺领域知识时面临的术语歧义、上下文缺失和跨文档语义关联困难等问题,提出一种基于模板分层构建的提示工程方法,实现工艺信息的自动抽取与融合。结果表明,TLP解决了LLM领域适应性差的问题,显著提高了术语识别准确率;LLM的上下文学习增强规则解决了TLP泛化能力弱的问题,提升了处理新术语能力,两者融合形成双向增强;同时构建出的图谱能直观展示工艺文档多模态信息及其关联,显著提升知识整合与复用效率。

本文引用格式

黄好阳1, 李飞2, 张恒郡2, 俞佳毅2, 鲍劲松2 . 融合LLM与TLP的跨工艺文档多模态知识图谱建模方法[J]. 上海交通大学学报, 0 : 1 . DOI: 10.16183/j.cnki.jsjtu.2025.068

Abstract

In response to the challenges of inefficient data management and difficult process reuse caused by the dispersion of multimodal data and the lack of inter-document connections in the field of steam turbine blade machining processes, this paper proposes a multimodal cross-document graph modeling technique based on Large Language Models (LLM) and Technical Language Processing (TLP). Initially, a multimodal graph ontology framework is designed, followed by the application of TLP technology to address issues such as complex terminology, non-standardized text, and cross-document semantic inconsistencies in process documentation. To tackle the problems faced by LLMs in understanding domain-specific process knowledge, including term ambiguity, lack of contextual information, and difficulties in cross-document semantic association, a template-based hierarchical prompt engineering method is introduced to facilitate the automatic extraction and integration of process information. The results show that TLP solves the problem of poor adaptability in the field of LLM and significantly improves the accuracy of term recognition. The context learning augmentation rules of LLM solve the problem of weak TLP generalization ability and improve the ability to deal with new terms, and the two are fused to form two-way enhancement. At the same time, the constructed map can intuitively display the multimodal information and its correlation of process documents, which significantly improves the efficiency of knowledge integration and reuse.
文章导航

/