Orthogonal Features Extraction Method and Its Application in Convolution Neural Network

doi:10.16183/j.cnki.jsjtu.2020.276

Abstract

Abstract:

In view of feature redundancy in the convolutional neural network, the concept of orthogonal vectors is introduced into features. Then, a method for orthogonal features extraction of convolutional neural network is proposed from the perspective of enhancing the differences between features. By building the structure of parallel branches and designing the orthogonal loss function, the convolution kernels can extract the orthogonal features, enrich the feature diversity, eliminate the feature redundancy, and improve the results of classification. The experiment results on one-dimensional sample dataset show that compared with the traditional convolution neural network, the proposed method can supervise the convolution kernels with different sizes to mine more comprehensive information of orthogonal features, which improves the efficiency of convolutional neural network and lays the foundation for subsequent researches on pattern recognition and compact neural network.

Key words: convolution neural network, feature redundancy, orthogonality, feature vector

CLC Number:

TP183

LI Chen, LI Jianxun. Orthogonal Features Extraction Method and Its Application in Convolution Neural Network[J]. Journal of Shanghai Jiao Tong University, 2021, 55(10): 1320-1329.

Figures/Tables 14

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Tab.1

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Tab.2

References 19

[1]	KANG X D, XIANG X L, LI S T, et al. PCA-based edge-preserving features for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(12):7140-7151. doi: 10.1109/TGRS.2017.2743102 URL
[2]	WU X H, ZUO W M, LIN L, et al. F-SVM: Combination of feature transformation and SVM learning via convex relaxation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(11):5185-5199. doi: 10.1109/TNNLS.2018.2791507 URL
[3]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition [EB/OL]. (2014-09-04) [2020-05-18]. https://arxiv.org/abs/1409.1556 .
[4]	刘娟宏, 胡彧, 黄鹤宇. 端到端的深度卷积神经网络语音识别[J]. 计算机应用与软件, 2020, 37(4):192-196.
	LIU Juanhong, HU Yu, HUANG Heyu. End-to-end speech recognition based on deep convolution neural network[J]. Computer Applications and Software, 2020, 37(4):192-196.
[5]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE, 2015: 3431-3440.
[6]	ADLER P, FALK C, FRIEDLER S A, et al. Auditing black-box models for indirect influence[J]. Knowledge and Information Systems, 2018, 54(1):95-122. doi: 10.1007/s10115-017-1116-3 URL
[7]	ZHANG Q L, JIANG Z Q, LU Q S, et al. Split to be slim: An overlooked redundancy in vanilla convolution[C]// Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. California, USA: International Joint Conferences on Artificial Intelligence Organization, 2020: 3195-3201.
[8]	HE Y H, ZHANG X Y, SUN J. Channel pruning for accelerating very deep neural networks[C]// 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 17453051.
[9]	CHEN C F, LEE G G, SRITAPAN V, et al. Deep convolutional neural network on iOS mobile devices[C]// 2016 IEEE International Workshop on Signal Processing Systems (SiPS). Dallas, TX, USA: IEEE, 2016: 130-135.
[10]	QIU J X, CHEN C, LIU S C,, et al. SlimConv: Reducing channel redundancy in convolutional neural networks by weights flipping [EB/OL].(2020-03-08) [2020-05-18]. https://www.researchgate.net/publication/339997878_SlimConv_Reducing_Channel_Redundancy_in_Convolutional_Neural_Networks_by_Weights_Flipping .
[11]	HAN K, WANG Y H, TIAN Q, et al. GhostNet: More features from cheap operations[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020: 19874590.
[12]	卢泓宇, 张敏, 刘奕群, 等. 卷积神经网络特征重要性分析及增强特征选择模型[J]. 软件学报, 2017, 28(11):2879-2890.
	LU Hongyu, ZHANG Min, LIU Yiqun, et al. Convolution neural network feature importance analysis and feature selection enhanced model[J]. Journal of Software, 2017, 28(11):2879-2890.
[13]	BRO R, SMILDE A K. Principal component analysis[J]. Analytical Methods, 2014, 6(9):2812-2831. doi: 10.1039/C3AY41907J URL
[14]	CHIANG H L, KADUR T, FETTWEIS G. Analyses of orthogonal and non-orthogonal steering vectors at millimeter wave systems[C]// 2016 IEEE 17th International Symposium on A World of Wireless, Mobile and Multimedia Networks (WoWMoM). Coimbra, Portugal: IEEE, 2016: 16192551.
[15]	孙文赟, 宋昱, 陈昌盛. 基于卷积-反卷积网络的正交人脸特征学习算法[J]. 深圳大学学报(理工版), 2020, 37(5):474-481.
	SUN Wenyun, SONG Yu, CHEN Changsheng. An orthogonal facial feature learning method based on convolutional-deconvolutional network[J]. Journal of Shenzhen University (Science and Engineering), 2020, 37(5):474-481.
[16]	GU J X, WANG Z H, KUEN J, et al. Recent advances in convolutional neural networks[J]. Pattern Recognition, 2018, 77:354-377. doi: 10.1016/j.patcog.2017.10.013 URL
[17]	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, Massachusetts, USA: IEEE, 2015: 1-9.
[18]	LUO W J, LI Y J, URTASUN R, et al. Understanding the effective receptive field in deep convolutional neural networks [EB/OL].(2017-01-15) [2020-05-18]. https://www.researchgate.net/publication/312461699_Understanding_the_Effective_Receptive_Field_in_Deep_Convolutional_Neural_Networks .
[19]	MURUGAN P. Feed forward and backward run in deep convolution neural network[EB/OL]. (2017-11-20) [2020-05-18]. https://www.researchgate.net/publication/320975739_Feed_Forward_and_Backward_Run_in_Deep_Convolution_Neural_Network .

模型名称	J₃	ζ/%
不含正交层的神经网络(正常CNN)	0.499	83.33
含2条支路的CNN模型(图5模型)	0.434	89.89
含4条支路的CNN模型(图6模型)	0.375	91.65

模型名称	J₃	ζ/%
不含正交模块的神经网络(正常CNN)	0.453	82.41
含2条支路的神经网络(图5模型)	0.347	87.57