Transfer Learning Based on Joint Feature Matching and Adversarial Networks

doi:10.1007/s12204-019-2132-0

Abstract

Abstract: Domain adaptation and adversarial networks are two main approaches for transfer learning. Domain adaptation methods match the mean values of source and target domains, which requires a very large batch size during training. However, adversarial networks are usually unstable when training. In this paper, we propose a joint method of feature matching and adversarial networks to reduce domain discrepancy and mine domaininvariant features from the local and global aspects. At the same time, our method improves the stability of training. Moreover, the method is embedded into a unified convolutional neural network that can be easily optimized by gradient descent. Experimental results show that our joint method can yield the state-of-the-art results on three common public datasets.

Key words: transfer learning| adversarial networks| feature matching| domain-invariant features

摘要： Domain adaptation and adversarial networks are two main approaches for transfer learning. Domain adaptation methods match the mean values of source and target domains, which requires a very large batch size during training. However, adversarial networks are usually unstable when training. In this paper, we propose a joint method of feature matching and adversarial networks to reduce domain discrepancy and mine domaininvariant features from the local and global aspects. At the same time, our method improves the stability of training. Moreover, the method is embedded into a unified convolutional neural network that can be easily optimized by gradient descent. Experimental results show that our joint method can yield the state-of-the-art results on three common public datasets.

关键词: transfer learning| adversarial networks| feature matching| domain-invariant features

CLC Number:

TP 181

ZHONG Haowen (钟昊文), WANG Chao (王超), TUO Hongya (庹红娅), HU Jian (胡健), QIAO Lingfeng (乔凌峰), JING Zhongliang (敬忠良). Transfer Learning Based on Joint Feature Matching and Adversarial Networks[J]. Journal of Shanghai Jiao Tong University (Science), 2019, 24(6): 699-705.

References 24

[1]	PAN S J, YANG Q. A survey on transfer learning [J].IEEE Transactions on Knowledge and Data Engineering,2010, 22(10): 1345-1359.
[9]	TZENG E, HOFFMAN J, SAENKO K, et al. Adversarialdiscriminative domain adaptation [C]//IEEEConference on Computer Vision and Pattern Recognition.Honolulu, HI, USA: IEEE, 2017: 7167-7176.
[10]	LIU M Y, TUZEL O. Coupled generative adversarialnetworks [C]//29th Conference on Neural InformationProcessing Systems (NIPS 2016). Barcelona, Spain:NIPS, 2016: 469-477.
[2]	PAN S J, TSANG I W, KWOK J T, et al. Domainadaptation via transfer component analysis [J]. IEEETransactions on Neural Networks, 2011, 22(2): 199-210.
[11]	SHEN J, QU Y R, ZHANGWN, et al.Wasserstein distanceguided representation learning for domain adaptation[C]//Thirty-Second AAAI Conference on ArtificialIntelligence. New Orleans, USA: AAAI, 2018:4058-4065.
[3]	GONG B, SHI Y, SHA F, et al. Geodesic flow kernelfor unsupervised domain adaptation [C]//IEEE Conferenceon Computer Vision and Pattern Recognition.Providence, RI, USA: IEEE, 2012: 2066-2073.
[12]	GOODFELLOW I J, POUGET-ABADIE J, MIRZAM, et al. Generative adversarial nets [C]//28th AnnualConference on Neural Information Processing Systems.Montreal, Canada: NIPS, 2014: 2672-2680.
[4]	LONG M S, CAO Y, WANG J M, et al. Learningtransferable features with deep adaptation networks[C]//32nd International Conference on MachineLearning. Lille, France: PMLR, 2015: 97-105.
[13]	GHIFARY M, KLEIJN W B, ZHANG M J. Domainadaptive neural networks for object recognition [M].Lecture Notes in Computer Science. Cham, Switzerland:Springer International Publishing Switzerland,2014: 898-904.
[5]	LONG M S, ZHU H, WANG J M, et al. Unsuperviseddomain adaptation with residual transfer networks[C]//30th Conference on Neural InformationProcessing Systems (NIPS 2016). Barcelona, Spain:NIPS, 2016: 136-144.
[6]	LONG M S, ZHU H, WANG J M, et al. Deep transferlearning with joint adaptation networks [C]//34th InternationalConference on Machine Learning. Sydney,Australia: PMLR, 2017: 2208-2217.
[14]	TZENG E, HOFFMAN J, ZHANG N, et al.Deep domain confusion: Maximizing for domaininvariance [EB/OL]. [2019-04-23]. https://arxiv.org/pdf/1412.3474v1.pdf.
[7]	GRETTON A, BORGWARDT K M, RASCH M J, etal. A kernel two-sample test [J]. Journal of MachineLearning Research, 2012, 13(3): 723-773.
[15]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet classification with deep convolutional neuralnetworks [C]//26th Annual Conference on NeuralInformation Processing Systems. Lake Tahoe, Nevada,USA: NIPS, 2012: 1097-1105.
[8]	GANIN Y, USTINOVA E, AJAKAN H, et al. Domainadversarialtraining of neural networks [J]. Journal ofMachine Learning Research, 2016, 17(1): 1-35.
[9]	TZENG E, HOFFMAN J, SAENKO K, et al. Adversarialdiscriminative domain adaptation [C]//IEEEConference on Computer Vision and Pattern Recognition.Honolulu, HI, USA: IEEE, 2017: 7167-7176.
[16]	YOSINSKI J, CLUNE J, BENGIO Y, et al. How transferableare features in deep neural networks? [C]//28thAnnual Conference on Neural Information ProcessingSystems. Montreal, Canada: NIPS, 2014: 3320-3328.
[17]	MA S, FU J L, CHEN W C, et al. DA-GAN: Instancelevelimage translation by deep attention generativeadversarial networks [C]//IEEE ICVF Conference onComputer Vision and Pattern Recognition. Salt LakeCity, USA: IEEE, 2018: 5657-5666.
[10]	LIU M Y, TUZEL O. Coupled generative adversarialnetworks [C]//29th Conference on Neural InformationProcessing Systems (NIPS 2016). Barcelona, Spain:NIPS, 2016: 469-477.
[18]	SALIMANS T, GOODFELLOW I, ZAREMBA W, etal. Improved techniques for training GANs [C]//29thConference on Neural Information Processing Systems(NIPS 2016). Barcelona, Spain: NIPS, 2016: 2234-2242.
[11]	SHEN J, QU Y R, ZHANGWN, et al.Wasserstein distanceguided representation learning for domain adaptation[C]//Thirty-Second AAAI Conference on ArtificialIntelligence. New Orleans, USA: AAAI, 2018:4058-4065.
[19]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired imageto-image translation using cycle-consistent adversarialnetworks [C]//IEEE International Conference onComputer Vision. Venice, Italy: IEEE, 2017: 2242-2251.
[12]	GOODFELLOW I J, POUGET-ABADIE J, MIRZAM, et al. Generative adversarial nets [C]//28th AnnualConference on Neural Information Processing Systems.Montreal, Canada: NIPS, 2014: 2672-2680.
[20]	BOUSMALIS K, TRIGEORGIS G, SILBERMAN N,et al. Domain separation networks [C]//29th Conferenceon Neural Information Processing Systems (NIPS2016). Barcelona, Spain: NIPS, 2016: 343-351.
[13]	GHIFARY M, KLEIJN W B, ZHANG M J. Domainadaptive neural networks for object recognition [M].Lecture Notes in Computer Science. Cham, Switzerland:Springer International Publishing Switzerland,2014: 898-904.
[21]	LI C L, CHANG W C, CHENG Y, et al. MMD GAN:Towards deeper understanding of moment matchingnetwork [C]//31st Annual Conference on Neural InformationProcessing Systems (NIPS 2017). Long Beach,CA, USA: NIPS, 2017: 2203-2213.
[14]	TZENG E, HOFFMAN J, ZHANG N, et al.Deep domain confusion: Maximizing for domaininvariance [EB/OL]. [2019-04-23]. https://arxiv.org/pdf/1412.3474v1.pdf.
[22]	DZIUGAITE G K, ROY D M, GHAHRAMANI Z.Training generative neural networks via maximummean discrepancy optimization [C]//31st Conferenceon Uncertainty in Artificial Intelligence. Amsterdam,the Netherlands: AUAI, 2015: 258-267.
[15]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet classification with deep convolutional neuralnetworks [C]//26th Annual Conference on NeuralInformation Processing Systems. Lake Tahoe, Nevada,USA: NIPS, 2012: 1097-1105.
[23]	SUN B C, FENG J S, SAENKO K. Return of frustratinglyeasy domain adaptation [C]//Association for theAdvance of Artificial Intelligence Conference. Phoenix,Arizona, USA: AAAI Press, 2016: 2058-2065.
[16]	YOSINSKI J, CLUNE J, BENGIO Y, et al. How transferableare features in deep neural networks? [C]//28thAnnual Conference on Neural Information ProcessingSystems. Montreal, Canada: NIPS, 2014: 3320-3328.
[24]	DONAHUE J, JIA Y, VINYALS O, et al. DeCAF: Adeep convolutional activation feature for generic visualrecognition [C]//31st International Conference on MachineLearning. Beijing, China: ICML, 2014: 647-655.
[17]	MA S, FU J L, CHEN W C, et al. DA-GAN: Instancelevelimage translation by deep attention generativeadversarial networks [C]//IEEE ICVF Conference onComputer Vision and Pattern Recognition. Salt LakeCity, USA: IEEE, 2018: 5657-5666.
[18]	SALIMANS T, GOODFELLOW I, ZAREMBA W, etal. Improved techniques for training GANs [C]//29thConference on Neural Information Processing Systems(NIPS 2016). Barcelona, Spain: NIPS, 2016: 2234-2242.
[19]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired imageto-image translation using cycle-consistent adversarialnetworks [C]//IEEE International Conference onComputer Vision. Venice, Italy: IEEE, 2017: 2242-2251.
[20]	BOUSMALIS K, TRIGEORGIS G, SILBERMAN N,et al. Domain separation networks [C]//29th Conferenceon Neural Information Processing Systems (NIPS2016). Barcelona, Spain: NIPS, 2016: 343-351.
[21]	LI C L, CHANG W C, CHENG Y, et al. MMD GAN:Towards deeper understanding of moment matchingnetwork [C]//31st Annual Conference on Neural InformationProcessing Systems (NIPS 2017). Long Beach,CA, USA: NIPS, 2017: 2203-2213.
[22]	DZIUGAITE G K, ROY D M, GHAHRAMANI Z.Training generative neural networks via maximummean discrepancy optimization [C]//31st Conferenceon Uncertainty in Artificial Intelligence. Amsterdam,the Netherlands: AUAI, 2015: 258-267.
[23]	SUN B C, FENG J S, SAENKO K. Return of frustratinglyeasy domain adaptation [C]//Association for theAdvance of Artificial Intelligence Conference. Phoenix,Arizona, USA: AAAI Press, 2016: 2058-2065.
[24]	DONAHUE J, JIA Y, VINYALS O, et al. DeCAF: Adeep convolutional activation feature for generic visualrecognition [C]//31st International Conference on MachineLearning. Beijing, China: ICML, 2014: 647-655.