| Citation: | LAN Lingqiang, LI Xin, LIU Qiyuan, et al. Facial expression recognition method based on a joint normalization strategy[J]. Journal of Beijing University of Aeronautics and Astronautics, 2020, 46(9): 1797-1806. doi: 10.13700/j.bh.1001-5965.2020.0073(in Chinese)
|
As for that end-to-end feature extraction and classification based on deep learning often used in facial expression recognition, a new method of depth model optimization has been proposed. This paper proposes the joint optimization strategies learned from ResNet18 residual network and normalization ideas, that is, filter response normalization and batch normalization, instance normalization and group normalization, as well as group normalization and batch normalization were embedded in the network, respectively, to balance and improve the distribution of feature data, make up for the shortcomings of single regularization, and improve model performance. The validation and test were carried out on the two public datasets FER2013 and CK+, and the highest accuracy rates are 73.558% and 94.9%, respectively. The experimental results indicate that the joint optimization strategy enhances the performance of the basic network, which is better than most of the latest facial expression recognition methods.
| [1] |
JAIN D K, SHAMSOLMOALI P, SEHDEV P.Extended deep neural network for facial emotion recognition[J].Pattern Recogntion Letters, 2019, 120:69-74. doi: 10.1016/j.patrec.2019.01.008
|
| [2] |
HU S H, HU Y M, LI J Q, et al.Natural scene facial expression recognition based on differential features[C]//2019 Chinese Automation Congress(CAC).Piscataway: IEEE Press, 2019: 2840-2844.
|
| [3] |
LI Y, CAO G T, CAO W M.Stacking-based deep neural network for facial expression recognition[C]//2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).Piscataway: IEEE Press, 2019: 1338-1342.
|
| [4] |
HE J, CAI J F, FANG L Z, et al.A method of facial expression recognition based on LBP fusion of key expressions areas[C]//The 27th Chinese Control and Decision Conference(2015 CCDC).Piscataway: IEEE Press, 2015: 4200-4204.
|
| [5] |
OJALA T, PIETIKÄINEN M, MÄENPÄÄ T.Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7):971-987. doi: 10.1109/TPAMI.2002.1017623
|
| [6] |
IOFFE S, SZEGEDY C.Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on Machine Learning (ICML).New York: ACM, 2015: 448-456.
|
| [7] |
ULYANOV D, VEDALDI A, LEMPITSKY V.Instance normalization: The missing ingredient for fast stylization[EB/OL].(2017-11-06)[2020-02-20].
|
| [8] |
WU Y X, HE K M.Group normalization[C]//The European Conference on Computer Vision (ECCV).Berlin: Springer, 2018: 3-19.
|
| [9] |
SINGH S, KRISHNAN S.Filter response normalization layer: Fliminating batch dependence in the training of deep neural networks[EB/OL].(2019-11-21)[2020-02-20].
|
| [10] |
GOODFELLOW I J, ERHAN D, CARRIER P L, et al.Challenges in representation learning: A report on three machine learning contests[C]//International Conference on Neural Information Processing.Berlin: Springer, 2013: 117-124.
|
| [11] |
LUCEY P, COHN J F, KANADE T, et al.The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR).Piscataway: IEEE Presss, 2010: 94-10.
|
| [12] |
DALAL N, TRIGGS B.Histograms of oriented gradients for human detection[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway: IEEE Press, 2005, 1: 886-893.
|
| [13] |
LOWE D G.Object recognition from local scale-invariant features[C]//Proceedings of the 7th IEEE International Conference on Computer Vision.Piscataway: IEEE Press, 1999, 2: 1150-1157.
|
| [14] |
LYONS M J, AKAMATSU S, KAMACHI M, et al.The Japanese female facial expression (JAFFE) database[C]//Proceedings of 3rd International Conference on Automatic Face and Gesture Recognition.Piscataway: IEEE Press, 1998: 14-16.
|
| [15] |
ZHAO G, HUANG X, TAINI M, et al.Facial expression recognition from near-infrared videos[J].Image and Vision Computing, 2011, 29(9):607-619. doi: 10.1016/j.imavis.2011.07.002
|
| [16] |
ELSAYED A, MAHMOOD A, SOBH T.Effect of super resolution on high dimensional features for unsupervised face recognition in the wild[C]//2017 IEEE Applied Imagery Pattern Recognition Workshop(AIPR).Piscataway: IEEE Press, 2017: 1-5.
|
| [17] |
WANG P Y, SU F, ZHAO Z C.Joint multi-feature fusion and attribute relationships for facial attribute prediction[C]//2017 IEEE Visual Communications and Image Processing (VCIP).Piscataway: IEEE Press, 2017: 1-4.
|
| [18] |
TAHERKHANI F, NASRABADI N M, DAWSON J.A deep face identification network enhanced by facial attributes prediction[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway: IEEE Press, 2018: 553-560.
|
| [19] |
KUO C M, LAI S H, MICHEL S.A compact deep learning model for robust facial expression recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway: IEEE Press, 2018: 2121-2129.
|
| [20] |
YANG J, ZHANG F, CHEN B, et al.Facial expression recognition based on facial action unit[C]//2019 10th International Green and Sustainable Computing Conference (IGSC).Piscataway: IEEE Press, 2019: 1-6.
|
| [21] |
YU K, SALZMANN M.Second-order convolution neural networks[J].Clinical Immunology & Immunopathology, 2017, 66(3):230-238.
|
| [22] |
GAO Z, XIE J, WANG Q, et al.Global second-order pooling convolutional networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Piscataway: IEEE Press, 2019: 3024-3033.
|
| [23] |
HUANG Z, LUC V G.A Riemannian network for SPD matrix learning[C]//31st AAAI Conference on Artificial Intelligence.San Francisco: AAAI Press, 2017: 2036-2042.
|
| [24] |
ACHARYA D, HUANG Z, PAUDEL D P, et al.Covariance pooling for facial expression recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Piscataway: IEEE Press, 2018: 367-374.
|
| [25] |
HAMESTER D, BARROS P, WERMTER S.Face expression recognition with a 2-channel convolutional neural network[C]//2015 International Joint Conference on Neural Networks (IJCNN).Piscataway: IEEE Press, 2015: 1-8.
|
| [26] |
MOLLAHOSSEINI A, CHAN D, MAHOOR M H.Going deeper in facial expression recognition using deep neural networks[C]//2016 IEEE Winter Conference on Applications of Computer Vision (WACV).Piscataway: IEEE Press, 2016: 1-10.
|
| [27] |
CHEN Y, HU H.Facial expression recognition by inter-class relational learning[J].IEEE Access, 2019, 7:94106-94117. doi: 10.1109/ACCESS.2019.2928983
|
| [28] |
LIU M, LI S, SHAN S, et al.Deeply learning deformable facial action parts model for dynamic expression analysis[C]//Asian Conference on Computer Vision(ACCV).Berlin: Springer, 2014: 143-157.
|
| [29] |
NGUYEN D H, KIM S H, LEE G S, et al.Facial expression recognition using a temporal ensemble of multi-level convolutional neural networks[EB/OL].(2019-10-10)[2020-02-20].
|
| [30] |
MONTAVON G, ORR G B, MULLER K R.Neural networks:Tricks of the trade[M].Berlin:Springer, 1998:9-50.
|
| [31] |
XIE S Y, GIRSHICK R, DOLLAR P, et al.Aggregated residual transformations for deep neural networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway: IEEE Press, 2017: 1492-1500.
|
| [32] |
SZEGEDY C, LIU W, JIA Y, et al.Going deeper with convolutions[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway: IEEE Press, 2015: 1-9.
|
| [33] |
HUANG G, LIU Z, WEINBERGER K Q, et al.Densely connected convolutional networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway: IEEE Press, 2017: 4700-4708.
|
| [34] |
ULYANOV D, VEDALDI A, LEMPITSKY V.Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Piscataway: IEEE Press, 2017: 6924-6932.
|
| [35] |
PAN X, LUO P, SHI J, et al.Two at once: Enhancing learning and generalization capacities via IBN-net[C]//The European Conference on Computer Vision (ECCV).Berlin: Springer, 2018: 464-479.
|
| [36] |
EKMAN P, FRIESEN W V.Facial action coding system:A technique for the measurement of facial movement[M].Palo Alto:Consulting Psychologists Press, 1978:32-96.
|
| [37] |
HE K, ZHANG X, REN S, et al.Identity mappings in deep residual networks[C]//The European Conference on Computer Vision.Berlin: Springer, 2016: 630-645.
|
| [38] |
QIN Z Y, WU J.Visual saliency maps can apply to facial expression recognition[EB/OL].(2018-11-12)[2020-02-20].
|
| [39] |
MIAO S, XU H Y, HAN Z Q, et al.Recognizing facial expressions using a shallow convolutional neural network[J].IEEE Access, 2019, 7:78000-78011. doi: 10.1109/ACCESS.2019.2921220
|
| [40] |
ZHOU J C, JIA X, SHEN L L, et al.Improved softmax loss for deep learning-based face and expression recognition[J].Cognitive Computation and Systems, 2019, 1(4):97-102. doi: 10.1049/ccs.2019.0010
|
| [41] |
TIAN Y, WEN Z W, XIE W C, et al.Outlier-suppressed triplet loss with adaptive class-aware margins for facial expression recognition[C]//2019 IEEE International Conference on Image Processing (ICIP).Piscataway: IEEE Press, 2019: 46-50.
|
| [42] |
XIE S Y, HU H F, WU Y B.Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition[J].Pattern Recognition, 2019, 92:177-191. doi: 10.1016/j.patcog.2019.03.019
|
Figures(5) / Tables(5)
Copyright © Journal of Beijing University of Aeronautics and Astronautics
Address: Editorial Department of Journal of Beijing University of Aeronautics and Astronautics, 37 Xueyuan Road, Haidian District, Beijing Post Code: 100191 Email: jbuaa@cqjj8.com
Supported by:
Beijing Renhe Information Technology Co., Ltd.
DownLoad:
