Research of Clinical Named Entity Recognition Based on Bi-LSTM-CRF

doi:10.1007/s12204-018-1954-5

Abstract

Abstract: Electronic Medical Records (EMR) with unstructured sentences and various conceptual expressions provide rich information for medical information extraction. However, common Named Entity Recognition (NER) in Natural Language Processing (NLP) are not well suitable for clinical NER in EMR. This study aims at applying neural networks to clinical concept extractions. We integrate Bidirectional Long Short-Term Memory Networks (Bi-LSTM) with a Conditional Random Fields (CRF) layer to detect three types of clinical named entities. Word representations fed into the neural networks are concatenated by character-based word embeddings and Contin- uous Bag of Words (CBOW) embeddings trained both on domain and non-domain corpus. We test our NER system on i2b2/VA open datasets and compare the performance with six related works, achieving the best result of NER with F1 value 0.853 7. We also point out a few speciˉc problems in clinical concept extractions which will give some hints to deeper studies.

Key words: clinical named entity recognition| bidirectional long short-term memory networks| conditional random fields

摘要： Electronic Medical Records (EMR) with unstructured sentences and various conceptual expressions provide rich information for medical information extraction. However, common Named Entity Recognition (NER) in Natural Language Processing (NLP) are not well suitable for clinical NER in EMR. This study aims at applying neural networks to clinical concept extractions. We integrate Bidirectional Long Short-Term Memory Networks (Bi-LSTM) with a Conditional Random Fields (CRF) layer to detect three types of clinical named entities. Word representations fed into the neural networks are concatenated by character-based word embeddings and Contin- uous Bag of Words (CBOW) embeddings trained both on domain and non-domain corpus. We test our NER system on i2b2/VA open datasets and compare the performance with six related works, achieving the best result of NER with F1 value 0.853 7. We also point out a few speciˉc problems in clinical concept extractions which will give some hints to deeper studies.

关键词: clinical named entity recognition| bidirectional long short-term memory networks| conditional random fields

CLC Number:

TP 391.4

QIN Ying (秦颖), ZENG Yingfei (曾颖菲). Research of Clinical Named Entity Recognition Based on Bi-LSTM-CRF[J]. Journal of Shanghai Jiao Tong University (Science), 2018, 23(3): 392-.

References 19

[1]	SAGER N, FRIEDMAN C, LYMAN M S. Review ofmedical language processing: computer managementof narrative data [J]. Computational Linguistics, 1989,15(3): 195-198.
[2]	UZUNER O, SOUTH B R, SHEN S, et al. 2010i2b2/VA challenge on concepts, assertions, and rela-tions in clinical text [J]. Journal of the American Med-ical Informatics Association. 2011, 18(5): 552-556.
[3]	CURRAN J R, CLARK S. Language indepen-dent NER using a maximum entropy tagger[C]//Proceedings of the 7th Conference on Natu-ral Language Learning at HLT-NAACL. Edmonton,Canada: ACL, 2003: 164-167.
[4]	TJONG KIM SANG E F, DE MEULDER F.Introduction to the CoNLL-2003 shared task:Language-Independent named entity recognition[C]//Proceedings of the 7th Conference on NaturalLanguage Learning at HLT-NAACL. Edmonton,Canada: ACL, 2003: 142-147.
[5]	COLLOBERT R, WESTON J, BOTTOU L, et al.Natural language processing (almost) from scratch [J].Journal of Machine Learning Research, 2011, 12(8):2493-2537.
[6]	HUANG Z, XU W, YU K. Bidirectional LSTM-CRFmodels for sequence tagging [EB/OL]. (2015-08-19).[2017-06-21]. https://arxiv.org/pdf/1508.01991v1.pdf.
[7]	LAMPLE G, BALLESTEROS M, SUBRAMANIAN S, et al. Neural architectures for named entity recog-nition [C]//Proceedings of NAACL-2016, San Diego,US: ACL, 2016: 260-270.
[8]	HOCHREITER S, SCHMIDHUBER J. Long shortterm memory [J].Neural Computation, 1997, 9(8):1735-1780.
[9]	LAFFERTY J, MCCALLUM A, PEREIRA F C N. Conditional random ˉelds: Probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conferenceon Machine Learning. Williamstown, US: IMLS, 2001:282-289.
[10]	BOAG W, WACOME K, NAUMANN T, et al.CliNER: A lightweight tool for clinical named entityrecognition [C]//AMIA Joint Summits on Clinical Research Informatics. San Francisco, CA: AMIA, 2015.
[11]	DE BRUIJN B, CHERRY C, KIRITCHENKO S, et al.Machine-Learned solutions for three stages of clinicalinformation extraction: the state of the art at i2b22010 [J].Journal of the American Medical InformaticsAssociation, 2011, 18(5): 557-562.
[12]	WU Y H, XU J, JIANG M, et al. A study of neuralword embeddings for named entity recognition in clinical text [C]//AMIA Annual Symposium Proceedings.2015: 1326-1333.
[13]	JONNALAGADDA S, COHEN T, WU S, et al.Enhancing clinical concept extraction with distribu-tional semantics [J]. Journal of Biomedical Informatics, 2012,45(1): 129-140.
[14]	CHALAPATHY R, BORZESHI, E Z, PICCARDIM. Bidirectional LSTM-CRF for clinical conceptextraction [EB/OL]. (2016-10-19). [2017-06-21].https://arxiv.org/pdf/1610.05858.pdf.
[15]	MIKOLOV T, CHEN K, CORRADO G, et al.E±cient estimation of word representations invector space [EB/OL]. (2013-09-07). [2017-06-21].https://arxiv.org/pdf/1301.3781v3.pdf.
[16]	BENGIO Y, SIMARD P, FRASCONI P. Learninglong-term dependencies with gradient descent is di±-cult [J].IEEE Transactions on Neural Networks, 1994,5(2): 157-166.
[17]	GRAVES A, SCHMIDHUBER J. Framewise phonemeclassiˉcation with bidirectional LSTM and other neu-ral network architectures [J]. Neural Networks, 2005,18(5/6): 602-610.
[18]	MIKOLOV T, SUTSKEVER I, CHEN K, et al. Dis-tributed representations of words and phrases andtheir compositionality [EB/OL]. (2013-10-16). [2017-06-21]. https://arxiv.org/pdf/1310.4546.pdf.
[19]	FU X, ANANIADOU S. Improving the extraction ofclinical concepts from clinical records [C]//Proceedingsof the 4th Workshop on Building and Evaluating Re-sources for Health and Biomedical Text Processing.Reykjavik, Iceland: ELRA, 2014.