Journal of Shanghai Jiaotong University >
Named Entity Recognition of Enterprise Annual Report Integrated with BERT
Received date: 2020-01-08
Online published: 2021-03-03
Automatically extracting key data from annual reports is an important means of business assessments. Aimed at the characteristics of complex entities, strong contextual semantics, and small scale of key entities in the field of corporate annual reports, a BERT-BiGRU-Attention-CRF model was proposed to automatically identify and extract entities in the annual reports of enterprises. Based on the BiGRU-CRF model, the BERT pre-trained language model was used to enhance the generalization ability of the word vector model to capture long-range contextual information. Furthermore, the attention mechanism was used to fully mine the global and local features of the text. The experiment was performed on a self-constructed corporate annual report corpus, and the model was compared with multiple sets of models. The results show that the value of F1 (harmonic mean of precision and recall) of the BERT-BiGRU-Attention-CRF model is 93.69%. The model has a better performance than other traditional models in annual reports, and is expected to provide an automatic means for enterprise assessments.
Key words： named entity recognition; enterprise annual report; BERT; attention mechanism; BiGRU
ZHANG Jingyi, HE Guanghui, DAI Zhou, LIU Yadong . Named Entity Recognition of Enterprise Annual Report Integrated with BERT[J]. Journal of Shanghai Jiaotong University, 2021 , 55(2) : 117 -123 . DOI: 10.16183/j.cnki.jsjtu.2020.009
| [1] | ZHENG S C, WANG F, BAO H Y, et al. Joint extraction of entities and relations based on a novel tagging scheme[C]∥Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. (Volume 1: Long Papers). Stroudsburg, PA, USA: ACL, 2017: 1227-1236. |
| [2] | 吴文涛，李培峰，朱巧明．基于混合神经网络的实体和事件联合抽取方法[J]. 中文信息学报，2019, 33(8): 77-83. |
| [2] | WU Wentao, LI Peifeng, ZHU Qiaoming. Joint extraction of entities and events by a hybrid neural network[J]. Journal of Chinese Information Processing, 2019, 33(8): 77-83. |
| [3] | HUANG Z H, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging[OL]. (2015-08-09)[2019-11-20]. . |
| [4] | CHIU J P C, NICHOLS E. Named entity recognition with bidirectional LSTM-CNNs[J]. Transactions of the Association for Computational Linguistics, 2016, 4(1): 357-370. |
| [5] | MA X Z, HOVY E. End-to-end sequence labeling via Bi-directional LSTM-CNNs-CRF[C]∥Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. (Volume 1: Long Papers). Stroudsburg, PA, USA: ACL, 2016: 1064-1074. |
| [6] | CHO K, VAN M B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]∥Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA, USA: ACL, 2014: 1724-1734. |
| [7] | 王洁，张瑞东，吴晨生．基于GRU的命名实体识别方法[J]. 计算机系统应用，2018, 27(9): 18-24. |
| [7] | WANG Jie, ZHANG Ruidong, WU Chensheng. Named entity recognition method based on GRU[J]. Computer Systems & Applications, 2018, 27(9): 18-24. |
| [8] | BHARADWAJ A, MORTENSEN D, DYER C, et al. Phonologically aware neural model for named entity recognition in low resource transfer settings[C]∥Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA, USA: ACL, 2016: 1462-1472. |
| [9] | CAO P F, CHEN Y B, LIU K, et al. Adversarial transfer learning for Chinese named entity recognition with self-attention mechanism[C]∥Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA, USA: ACL, 2018: 182-192. |
| [10] | VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]∥Proceedings of the 31st Conference on Neural Information Processing Systems. Long Beach, CA, USA: NIPS, 2017: 5998-6008. |
| [11] | DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[OL]. (2019-05-24)[2019-11-25]. . |
| [12] | WITTEN I H, PAYNTER G W, FRANK E, et al. KEA: Practical automatic keyphrase extraction[C]∥Proceedings of the Fourth ACM Conference on Digital Libraries. New York, USA: ACM, 1999: 254-255. |
| [13] | KANDOLA E J, HOFMANN T, POGGIO T, et al. A neural probabilistic language model[J]. Studies in Fuzziness and Soft Computing, 2006, 194: 137-186. |
| [14] | KIROS R, ZHU Y K, SALAKHUTDINOV R R, et al. Skip-thought vectors[C]∥Proceedings of the 29th Conference on Neural Information Processing Systems. Montreal, Canada: NIPS, 2015: 3294-3302. |
| [15] | STRUBELL E, VERGA P, BELANGER D, et al. Fast and accurate entity recognition with iterated dilated convolutions[C]∥Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA, USA: ACL, 2017: 2670-2680. |
/
| 〈 |
|
〉 |
