卷积神经网络的正交性特征提取方法及其应用

doi:10.16183/j.cnki.jsjtu.2020.276

摘要/Abstract

摘要：

针对卷积神经网络中存在的特征冗余问题,将正交性向量的概念引入特征中,从强化特征之间差异性的角度,提出一种适用于卷积神经网络的正交性特征提取方法.通过搭建并列的卷积神经网络支路结构,设计正交损失函数,从而促使卷积核提取出相互正交的样本特征,丰富特征多样性,消除特征冗余,提升特征用于分类识别的效果.在一维样本数据集上的实验结果表明,相比于普通的卷积神经网络,所提方法能够监督不同卷积核,挖掘出更为全面的正交性特征信息,进而提升卷积神经网络的性能效率,为后续模式识别和紧凑型神经网络的研究奠定基础.

关键词: 卷积神经网络, 特征冗余, 正交性, 特征向量

Abstract:

In view of feature redundancy in the convolutional neural network, the concept of orthogonal vectors is introduced into features. Then, a method for orthogonal features extraction of convolutional neural network is proposed from the perspective of enhancing the differences between features. By building the structure of parallel branches and designing the orthogonal loss function, the convolution kernels can extract the orthogonal features, enrich the feature diversity, eliminate the feature redundancy, and improve the results of classification. The experiment results on one-dimensional sample dataset show that compared with the traditional convolution neural network, the proposed method can supervise the convolution kernels with different sizes to mine more comprehensive information of orthogonal features, which improves the efficiency of convolutional neural network and lays the foundation for subsequent researches on pattern recognition and compact neural network.

Key words: convolution neural network, feature redundancy, orthogonality, feature vector

中图分类号:

TP183

李辰, 李建勋. 卷积神经网络的正交性特征提取方法及其应用[J]. 上海交通大学学报, 2021, 55(10): 1320-1329.

LI Chen, LI Jianxun. Orthogonal Features Extraction Method and Its Application in Convolution Neural Network[J]. Journal of Shanghai Jiao Tong University, 2021, 55(10): 1320-1329.

图/表 14

图1

图2

图3

图4

图5

图6

图7

表1

图8

图9

图10

图11

图12

表2

参考文献 19

[1]	KANG X D, XIANG X L, LI S T, et al. PCA-based edge-preserving features for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(12):7140-7151. doi: 10.1109/TGRS.2017.2743102 URL
[2]	WU X H, ZUO W M, LIN L, et al. F-SVM: Combination of feature transformation and SVM learning via convex relaxation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(11):5185-5199. doi: 10.1109/TNNLS.2018.2791507 URL
[3]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition [EB/OL]. (2014-09-04) [2020-05-18]. https://arxiv.org/abs/1409.1556 .
[4]	刘娟宏, 胡彧, 黄鹤宇. 端到端的深度卷积神经网络语音识别[J]. 计算机应用与软件, 2020, 37(4):192-196.
	LIU Juanhong, HU Yu, HUANG Heyu. End-to-end speech recognition based on deep convolution neural network[J]. Computer Applications and Software, 2020, 37(4):192-196.
[5]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE, 2015: 3431-3440.
[6]	ADLER P, FALK C, FRIEDLER S A, et al. Auditing black-box models for indirect influence[J]. Knowledge and Information Systems, 2018, 54(1):95-122. doi: 10.1007/s10115-017-1116-3 URL
[7]	ZHANG Q L, JIANG Z Q, LU Q S, et al. Split to be slim: An overlooked redundancy in vanilla convolution[C]// Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. California, USA: International Joint Conferences on Artificial Intelligence Organization, 2020: 3195-3201.
[8]	HE Y H, ZHANG X Y, SUN J. Channel pruning for accelerating very deep neural networks[C]// 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 17453051.
[9]	CHEN C F, LEE G G, SRITAPAN V, et al. Deep convolutional neural network on iOS mobile devices[C]// 2016 IEEE International Workshop on Signal Processing Systems (SiPS). Dallas, TX, USA: IEEE, 2016: 130-135.
[10]	QIU J X, CHEN C, LIU S C,, et al. SlimConv: Reducing channel redundancy in convolutional neural networks by weights flipping [EB/OL].(2020-03-08) [2020-05-18]. https://www.researchgate.net/publication/339997878_SlimConv_Reducing_Channel_Redundancy_in_Convolutional_Neural_Networks_by_Weights_Flipping .
[11]	HAN K, WANG Y H, TIAN Q, et al. GhostNet: More features from cheap operations[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020: 19874590.
[12]	卢泓宇, 张敏, 刘奕群, 等. 卷积神经网络特征重要性分析及增强特征选择模型[J]. 软件学报, 2017, 28(11):2879-2890.
	LU Hongyu, ZHANG Min, LIU Yiqun, et al. Convolution neural network feature importance analysis and feature selection enhanced model[J]. Journal of Software, 2017, 28(11):2879-2890.
[13]	BRO R, SMILDE A K. Principal component analysis[J]. Analytical Methods, 2014, 6(9):2812-2831. doi: 10.1039/C3AY41907J URL
[14]	CHIANG H L, KADUR T, FETTWEIS G. Analyses of orthogonal and non-orthogonal steering vectors at millimeter wave systems[C]// 2016 IEEE 17th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM). Coimbra, Portugal: IEEE, 2016: 16192551.
[15]	孙文赟, 宋昱, 陈昌盛. 基于卷积-反卷积网络的正交人脸特征学习算法[J]. 深圳大学学报(理工版), 2020, 37(5):474-481.
	SUN Wenyun, SONG Yu, CHEN Changsheng. An orthogonal facial feature learning method based on convolutional-deconvolutional network[J]. Journal of Shenzhen University (Science and Engineering), 2020, 37(5):474-481.
[16]	GU J X, WANG Z H, KUEN J, et al. Recent advances in convolutional neural networks[J]. Pattern Recognition, 2018, 77:354-377. doi: 10.1016/j.patcog.2017.10.013 URL
[17]	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, Massachusetts, USA: IEEE, 2015: 1-9.
[18]	LUO W J, LI Y J, URTASUN R, et al. Understanding the effective receptive field in deep convolutional neural networks [EB/OL].(2017-01-15) [2020-05-18]. https://www.researchgate.net/publication/312461699_Understanding_the_Effective_Receptive_Field_in_Deep_Convolutional_Neural_Networks .
[19]	MURUGAN P. Feed forward and backward run in deep convolution neural network[EB/OL]. (2017-11-20) [2020-05-18]. https://www.researchgate.net/publication/320975739_Feed_Forward_and_Backward_Run_in_Deep_Convolution_Neural_Network .

模型名称	J₃	ζ/%
不含正交层的神经网络(正常CNN)	0.499	83.33
含2条支路的CNN模型(图5模型)	0.434	89.89
含4条支路的CNN模型(图6模型)	0.375	91.65

模型名称	J₃	ζ/%
不含正交模块的神经网络(正常CNN)	0.453	82.41
含2条支路的神经网络(图5模型)	0.347	87.57