Named Entity Recognition of Design Specification Integrated with High-Quality Topic and Attention Mechanism

doi:10.1007/s12204-022-2534-2

Abstract

Abstract: Automatic extraction of key data from design specifications is an important means to assist in engineering design automation. Considering the characteristics of diverse data types, small scale, insufficient character information content and strong contextual relevance of design specification, a named entity recognition model integrated with high-quality topic and attention mechanism, namely Quality Topic-Char Embedding-BiLSTMAttention-CRF, was proposed to automatically identify entities in design specification. Based on the topic model,an improved algorithm for high-quality topic extraction was proposed first, and then the high-quality topic information obtained was added into the distributed representation of Chinese characters to better enrich character features. Next, the attention mechanism was used in parallel on the basis of the BiLSTM-CRF model to fully mine the contextual semantic information. Finally, the experiment was performed on the collected corpus of Chinese ship design specification, and the model was compared with multiple sets of models. The results show that F-score (harmonic mean of precision and recall) of the model is 80.24%. The model performs better than other models in design specification, and is expected to provide an automatic means for engineering design.

Key words: named entity recognition, design specification, topic model, high-quality topic, attention mechanism

摘要： 从设计规范中自动提取关键数据是辅助工程设计自动化的重要手段。针对设计规范数据类型多、规模小、字符信息含量不足、上下文相关性强等特点，提出了一种集成高质量主题与注意力机制的命名实体识别模型，即“高质量主题-字符嵌入- BiLSTM- CRF”，用于设计规范实体的自动识别。在主题模型的基础上，提出了一种改进的高质量主题提取算法，然后将获得的高质量主题信息加入到汉字的分布式表示中，以更好地丰富汉字特征。其次，在BiLSTM-CRF模型的基础上并行使用注意机制，充分挖掘上下文语义信息。最后，在收集到的中国船舶设计规范语料上进行了实验，并与多组模型进行了比较。结果表明：该模型的F-score（召回率和准确率的调和平均值）为80.24%。该模型在设计规范方面优于其他模型，有望为工程设计提供一种自动化手段。

关键词: 命名实体识别，设计规范，主题模型，优质主题，注意力机制

CLC Number:

TP391

ZHOU Cheng (周成), JIANG Zuhua^∗ (蒋祖华). Named Entity Recognition of Design Specification Integrated with High-Quality Topic and Attention Mechanism[J]. J Shanghai Jiaotong Univ Sci, 2024, 29(6): 1169-1180.

References

[1] LAI T Q, CHENG L L, WANG D P, et al. RMAN: Relational multi-head attention neural network for joint extraction of entities and relations [J]. Applied Intelligence, 2022, 52(3): 3132-3142.
[2] SHI D M, LIN Y G, CHEN E H. A NER-based feature selection method for text classification [J]. Computer Engineering & Science, 2007, 29(11): 152-156 (in Chinese).
[3] WU W T, LI P F, ZHU Q M. Joint extraction of entities and events by a hybrid neural network [J]. Journal of Chinese Information Processing, 2019, 33(8): 77-83(in Chinese).
[4] WANG Q L, LIANG Y, YAN L F, et al. Research on named entity recognition algorithm based on Power Intelligent Question Answering System [J]. Electronic Technology and Software Engineering, 2022(3): 95-98(in Chinese).
[5] BORTHWICK A, STERLING J, AGICHTEIN E, et al. Description of the MENE named entity system as used in MUC-7 [C]//7th Message Understanding Conference. Fairfax: ACL, 1998: 1-7.
[6] BURGER J D, HENDERSON J C, MORGAN W T. Statistical named entity recognizer adaptation [C]//6th Conference on Natural Language Learning. Stroudsburg: ACL, 2002: 1-4.
[7] LAFFERTY J D, MCCALLUM A, PEREIRA F C N. Conditional random fields: Probabilistic models for segmenting and labeling sequence data [C]//18th International Conference on Machine Learning. San Francisco: ACM, 2001: 282-289.
[8] PENG N Y, DREDZE M. Improving named entity recognition for Chinese social media with word segmentation representation learning [C]//54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Berlin: ACL, 2016:149-155.
[9] HUANG Z H, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging [DB/OL]. (2015-08-09).https://arxiv.org/abs/1508.01991.
[10] BHARADWAJ A, MORTENSEN D, DYER C, et al. Phonologically aware neural model for named entity recognition in low resource transfer settings [C]//2016 Conference on Empirical Methods in Natural Language Processing. Austin: ACL, 2016: 1462-1472.
[11] MA X Z, HOVY E. End-to-end sequence labeling via Bi-directional LSTM-CNNs-CRF [C]//54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin: ACL,2016: 1064-1074.
[12] GU Y. Research on complex Chinese named entity recognition based on BiLSTM-CRF [D]. Nanjing: Nanjing University, 2019 (in Chinese).
[13] DEVLIN J, CHANG M W, LEE K, et al.Bert: Pre-training of deep bidirectional transformers for language understanding [DB/OL]. (2018-10-11).https://arxiv.org/abs/1810.04805.
[14] ZHANG W T, JIANG S H, ZHAO S, et al. A BERTBiLSTM-CRF model for Chinese electronic medical records named entity recognition [C]//2019 12th International Conference on Intelligent Computation Technology and Automation. Xiangtan: IEEE, 2019: 166-169.
[15] KIM Y. Convolutional neural networks for sentence classification [C]//2014 Conference on Empirical Methods in Natural Language Processing. Doha: ACL, 2014: 1746-1751.
[16] FREY B J, DUECK D. Clustering by passing messages between data points [J]. Science, 2007, 315(5814):972-976.
[17] FU R J, QIN B, LIU T. Open-categorical text classification based on multi-LDA models [J]. Soft Computing, 2015, 19(1): 29-38.
[18] LIU L Y, JIANG H M, HE P C, et al. On the variance of the adaptive learning rate and beyond [DB/OL].(2019-08-08). https://arxiv.org/abs/1908.03265.
[19] ZHANG M R, LUCAS J, HINTON G E, et al. Lookahead optimizer: k steps forward, 1 step back [DB/OL].(2019-07-19). https://arxiv.org/abs/1907.08610.