Multi-Feature Extraction of Pulmonary Nodules Based on LSTM and Attention Structure

doi:10.16183/j.cnki.jsjtu.2021.113

Abstract

Abstract:

The accurate classification of shape, edge, and internal features of pulmonary nodules can not only assist the radiologists in their daily diagnosis, but also improve the writing efficiency of imaging reports. This paper proposes a multi-task classification model based on long-short term memory (LSTM) and attention structure, which merges the shared features among different classification tasks through attention mechanism to improve the feature extraction performance of the current task. The classifier based on temporal sequence LSTM structure can effectively screen the shared features and improve the efficiency of information transmission in the multi-task model. Experimental results show that compared with the traditional multi-task structure, the proposed model can achieve better classification results on the public dataset LIDC-IDRI, and assist doctors to quickly obtain nodule characteristics.

Key words: pulmonary nodule, attention structure, long-short term memory(LSTM)network, multi-task classification

CLC Number:

R318
TP181

NI Yangfan, YANG Yuanyuan, XIE Zhe, ZHENG Dezhong, WANG Weidong. Multi-Feature Extraction of Pulmonary Nodules Based on LSTM and Attention Structure[J]. Journal of Shanghai Jiao Tong University, 2022, 56(8): 1078-1088.

Figures/Tables 10

Fig.1

Fig.2

Fig.3

Tab.1

Tab.2

Fig.4

Tab.3

Fig.5

Tab.4

Fig.6

References 27

[1]	VANDENHENDE S, GEORGOULIS S, VAN GANSBEKE W, et al. Multi-task learning for dense prediction tasks: A survey[DB/OL]. (2021-01-26) [2021-03-01]. https://ieeexplore.ieee.org/abstract/document/9336293.
[2]	HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]// 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2980-2988.
[3]	LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018: 8759-8768.
[4]	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. doi: 10.1109/TPAMI.2016.2577031 URL
[5]	LIU W, ANGUELOV D, ERHAN D, et al. SSD: Single shot MultiBox detector[M]// Computer vision-ECCV 2016. Cham, Switzerland: Springer International Publishing, 2016: 21-37.
[6]	DUNNICK N R, LANGLOTZ C P. The radiology report of the future: A summary of the 2007 intersociety conference[J]. Journal of the American College of Radiology, 2008, 5(5): 626-629. doi: 10.1016/j.jacr.2007.12.015 URL
[7]	KOKKINOS I. UberNet:Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017: 5454-5463.
[8]	MISRA I, SHRIVASTAVA A, GUPTA A, et al. Cross-stitch networks for multi-task learning[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016: 3994-4003.
[9]	SHEN S W, HAN S X, ABERLE D R, et al. An interpretable deep hierarchical semantic convolutional neural network for lung nodule malignancy classification[J]. Expert Systems With Applications, 2019, 128: 84-95. doi: 10.1016/j.eswa.2019.01.048 URL
[10]	AMYAR A, MODZELEWSKI R, LI H, et al. Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: Classification and segmentation[J]. Computers in Biology and Medicine, 2020, 126: 104037. doi: 10.1016/j.compbiomed.2020.104037 URL
[11]	CHEN Z V. BADRINARAYANAN V, LEE C Y, et al. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks[C]// International Conference on Machine Learning. Stockholm, Sweden: PMLR, 2018: 794-803.
[12]	LIU S K, JOHNS E, DAVISON A J. End-to-end multi-task learning with attention[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019: 1871-1880.
[13]	GUO M, HAQUE A, HUANG D A, et al. Dynamic task prioritization for multitask learning[M]// Computer vision-ECCV 2018. Cham, Switzerland: Springer International Publishing, 2018: 282-299.
[14]	JALALI A, SANGHAVI S, RUAN C, et al. A dirty model for multi-task learning[J]. Advances in Neural Information Processing Systems, 2010, 23: 964-972.
[15]	ANDO R K, ZHANG T, BARTLETT P. A framework for learning predictive structures from multiple tasks and unlabeled data[J]. Journal of Machine Learning Research, 2005, 6: 1817-1853.
[16]	CHEN J H, LIU J, YE J P. Learning incoherent sparse and low-rank patterns from multiple tasks[J]. ACM Transactions on Knowledge Discovery from Data, 2012, 5(4): 1-31.
[17]	JACOBS R A, JORDAN M I, NOWLAN S J, et al. Adaptive mixtures of local experts[J]. Neural Computation, 1991, 3(1): 79-87. doi: 10.1162/neco.1991.3.1.79 URL
[18]	SHAZEER N, MIRHOSEINI A, MAZIARZ K, et al. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer[DB/OL]. (2017-01-23) [2021-03-01]. https://arxiv.org/abs/1701.06538.
[19]	MA J Q, ZHAO Z, YI X Y, et al. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts[C]// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York, NY, USA: ACM, 2018: 1930-1939.
[20]	CHEN S H, QIN J, JI X, et al. Automatic scoring of multiple semantic attributes with multi-task feature leverage: A study on pulmonary nodules in CT images[J]. IEEE Transactions on Medical Imaging, 2017, 36(3): 802-814. doi: 10.1109/TMI.2016.2629462 URL
[21]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]// 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2999-3007.
[22]	ARMATO S G I, MCLENNAN G, BIDAUT L, et al. The lung image database consortium (LIDC) and image database resource initiative (IDRI): A completed reference database of lung nodules on CT scans[J]. Medical Physics, 2011, 38(2): 915-931. doi: 10.1118/1.3528204 URL
[23]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, Nevada, USA: IEEE, 2016: 770-778.
[24]	MCNITT-GRAY M F, ARMATO S G III, MEYER C R, et al. The lung image database consortium (LIDC) data collection process for nodule detection and annotation[J]. Academic Radiology, 2007, 14(12): 1464-1474. doi: 10.1016/j.acra.2007.07.021 URL
[25]	ISENSEE F, JAEGER P F, KOHL S A A, et al. nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation[J]. Nature Methods, 2021, 18(2): 203-211. doi: 10.1038/s41592-020-01008-z URL
[26]	CIPOLLA R, GAL Y, KENDALL A. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Shenzhen, China: IEEE, 2018: 7482-7491.
[27]	KENDALL A, GAL Y, CIPOLLA R, et al. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. Salt Lake City, UT, USA: IEEE, 2018: 7482-7491.

语义特征	描述	分级	数量
恶性风险	结节的恶性概率	1.可能性极低(0) 2.可能性较低(0) 3.不确定(0) 4.可能性较高(1) 5.可能性极高(1)	(0) 1002 (1) 559
形状	结节的三维圆度	1.索条型(2) 2. -(2) 3.椭球形(1) 4. -(1) 5.球形(0)	(0) 357 (1) 541 (2) 663
边缘	结节边缘是否清晰	1.边缘模糊(0) 2. -(0) 3. -(0) 4.边缘可分(1) 5.边缘十分清晰(1)	(0) 599 (1) 962
毛刺	毛刺出现的密集程度	1~4分级代表毛刺密度(0) 5.未出现毛刺(1)	(0) 1057 (1) 504
分叶	分叶出现的密集程度	1~4分级代表分叶密度(0) 5.未出现分叶(1)	(0) 1287 (1) 274
纹理	结节的内部纹理	1.纯磨玻璃(2) 2. - 3.半实性(1) 4. - 5.实性(0)	(0) 1184 (1) 161 (2) 216
钙化	结节是否出现钙化	1.爆米花型(0) 2.板层状(0) 3.实性(0) 4.非中心型(0) 5.中心型(0) 6.无钙化(1)	176 1385
内部结构	结节的内部组成	1.软组织(0) 2.流体(1) 3. - 4. - 5.空气(2)	(0) 1054 (1) 29 (2) 478

分类特征	Res50			OMoE			Res50+LSTM			Res50+ATT+LSTM
分类特征	θ%	F₁%	e	θ%	F₁%	e	θ%	F₁%	e	θ%	F₁%	e
形状	66.89	45.25	0.79	67.22	46.52	0.82	68.24	48.61	0.75	70.01	50.97	0.69
边界	78.93	87.12	0.78	78.26	87.22	0.85	78.95	88.64	0.76	79.21	88.01	0.80
毛刺	94.98	97.26	0.58	93.97	96.89	0.57	94.31	97.86	0.53	94.32	97.38	0.53
分叶	94.98	97.42	0.67	94.98	97.42	0.68	95.00	97.46	0.62	95.55	97.89	0.52
内部成分	99.33	34.53	0.03	99.33	34.44	0.02	99.33	34.81	0.07	99.32	33.59	0.05
实性程度	68.56	73.30	0.59	68.22	74.09	0.64	75.57	75.67	0.57	78.62	71.94	0.48
钙化	88.63	94.55	0.32	84.61	91.84	0.41	89.97	94.71	0.35	90.00	94.93	0.33
恶性程度	83.27	66.89	0.77	79.60	66.92	0.90	84.60	67.95	0.70	85.10	75.98	0.63

分类特征	Res50			OMoE			Res50+LSTM			Res50+ATT+Bi-LSTM					Res50+ATT+LSTM
分类特征	θ%	F₁%	e	θ%	F₁%	e	θ%	F₁%	e	θ%	F₁%		e		θ%	F₁%	e
形状	67.22	47.56	0.80	68.23	47.66	0.81	69.20	49.66	0.73	70.98		51.15		0.68	71.21	51.99	0.70
边界	78.66	87.59	0.82	77.93	87.96	0.83	79.20	87.59	0.75	81.25		88.56		0.75	81.11	89.10	0.74
毛刺	94.31	97.07	0.54	94.31	97.10	0.54	94.32	97.38	0.56	94.31		97.42		0.53	94.31	97.65	0.50
分叶	94.98	97.42	0.64	94.98	97.43	0.64	95.10	97.11	0.59	95.12		97.89		0.52	95.66	97.88	0.50
内部成分	99.33	34.36	0.03	99.33	34.67	0.04	99.33	35.01	0.06	99.32		34.89		0.06	99.32	34.41	0.06
实性程度	78.26	75.36	0.57	78.01	67.05	0.62	78.72	75.90	0.51	78.77		71.56		0.58	78.88	71.28	0.48
钙化	89.97	94.88	0.42	89.96	94.71	0.43	89.99	97.48	0.40	92.29		95.15		0.37	92.21	95.12	0.31
恶性程度	81.27	67.65	0.80	80.60	66.29	0.83	84.88	71.29	0.66	86.59		78.05		0.54	86.61	78.11	0.59

特征	文献[9]			文献[20]			Res50+ATT+LSTM
特征	θ%	F₁%	e	θ%	F₁%	e	θ%	F₁%	e
形状	-	-	-	-	-	0.86	71.21	51.99	0.70
边界	72.5	68.97	-	-	-	0.92	81.11	89.10	0.74
毛刺	-	-	-	-	-	0.64	94.31	97.65	0.50
分叶	-	-	-	-	-	0.80	95.66	97.88	0.50
内部成分	-	-	-	-	-	0.02	99.32	34.41	0.06
实性程度	-	-	-	-	-	0.18	78.88	71.28	0.48
钙化	90.8	84.74	-	-	-	0.87	92.21	95.12	0.31
恶性程度	84.2	78.64	-	-	-	0.87	86.61	68.11	0.59