Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Pose-invariant face recognition with multitask cascade networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In this work, a face recognition method is proposed for face under pose variations using a multitask convolutional neural network (CNN). Furthermore, a pose estimation method followed by a face identification module is combined in a cascaded structure and used separately. In the presence of various facial expressions as well as low illuminations, datasets that include separated face poses can enhance the robustness of face recognition. The proposed method relies on a pose estimation module using a convolutional neural network model and trained on three categories of face image capture such as the left side, frontal, and right side. Second, three CNN models are used for face identification according to the estimated pose. The Left-CNN model, Front-CNN model, and Right-CNN model are used to identify the face for the left, frontal, and right pose of the face, respectively. Because face images may contain some useless information (e.g., background content), we propose a skin-based face segmentation method using structure–texture decomposition and the color-invariant descriptor. Experimental evaluation has been conducted using the proposed cascade-based face recognition system that consists of the aforementioned steps (i.e., pose estimation, face segmentation, and face identification) and is assessed on four different datasets and its superiority has been shown over related state-of-the-art techniques. Results reveal the contribution of the separate representation, skin segmentation, and pose estimation in the recognition robustness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Elharrouss O, Almaadeed N, Al-Maadeed S (2021) A Review of Video Surveillance Systems. J Vis Commun Image Represent p 103116

  2. Akbari Y, Almaadeed N, Al-maadeed S, Elharrouss O (2021) Applications, databases and open computer vision research from drone videos and images: a survey. Artif Intell Rev 1–52

  3. Wang Z et al (2014) Low-resolution face recognition: a review. Vis Comput 30(4):359–386

    Article  Google Scholar 

  4. Yi D, Lei Z, Li SZ (2013) Towards pose robust face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  5. Yu X, Xu F (2020) Random inverse packet information and its acquisition. Appl Math Nonlinear Sci 5(2):357–366

    Article  MathSciNet  Google Scholar 

  6. Xie T, Liu R, Wei Z (2020) Improvement of the fast clustering algorithm improved by k-means in the big data. Appl Math Nonlinear Sci 5(1):1–10

    Article  MathSciNet  Google Scholar 

  7. Li T, Yang W (2020) Solution to chance constrained programming problem in swap trailer transport organisation based on improved simulated annealing algorithm. Appl Math Nonlinear Sci 5(1):47–54

    Article  MathSciNet  Google Scholar 

  8. Crosswhite N et al (2018) Template adaptation for face verification and identification. Image Vis Comput 79:35–48

    Article  Google Scholar 

  9. Abbad A et al (2018) Application of MEEMD in post-processing of dimensionality reduction methods for face recognition. IET Biometrics 8(1):59–68

    Article  MathSciNet  Google Scholar 

  10. Dadi HS, Pillutla GKM, Makkena ML (2017) Face recognition and human tracking using GMM, HOG and SVM in surveillance videos. Ann Data Sci 1–23

  11. Xu W et al (2018) Sensor-assisted multi-view face recognition system on smart glass. IEEE Trans Mobile Comput 17(1):197–210

    Article  Google Scholar 

  12. Bhowmik MK et al (2019) Enhancement of robustness of face recognition system through reduced gaussianity in Log-ICA. Expert Syst Appl 116:96–107

    Article  Google Scholar 

  13. Li Y et al (2019) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439–2450

    Article  MathSciNet  Google Scholar 

  14. Ge S et al (2019) Low-resolution face recognition in the wild via selective knowledge distillation. IEEE Trans Image Process 28(4):2051–2062

    Article  MathSciNet  Google Scholar 

  15. Masi I, et al (2016) Pose-aware face recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  16. Pentland A, Moghaddam B, Starner T (1994) View-based and modular eigenspaces for face recognition

  17. Sun Y, Wang X, Tang X (2014) Deep learning face representation from predicting 10,000 classes. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  18. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  19. Taigman Y, et al (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  20. Bashbaghi, S et al (2019) Deep learning architectures for face recognition in video surveillance. Deep Learning in Object Detection and Recognition. Springer, Singapore, pp 133–154

  21. Jourabloo A, et al (2017) Pose-invariant face alignment with a single CNN. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV)

  22. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: bmvc

  23. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  24. Banerjee S, Das S (2018) Mutual variation of information on transfer-CNN for face recognition with degraded probe samples. Neurocomputing 310:299–315

    Article  Google Scholar 

  25. Zhang MM, Shang K, Wu H (2019) Deep Compact Discriminative representation for unconstrained face recognition. Signal Process Image Commun

  26. He R et al (2018) Wasserstein CNN: learning invariant features for nir-vis face recognition. IEEE Trans Pattern Analysis Mach Intell

  27. Hsu G-S et al (2018) Fast Landmark localization with 3D component reconstruction and CNN for cross-pose recognition. IEEE Trans Circuits Syst Video Technol 28(11):3194–3207

    Article  Google Scholar 

  28. Huang GB, Learned-Miller E (2014) Labeled faces in the wild: updates and new reporting procedures. Dept. Comput. Sci., Univ. Massachusetts Amherst, Amherst, MA, USA, Tech. Rep, : p. 14-003

  29. Littwin E, Wolf L (2016) The multiverse loss for robust transfer learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 3957–3966

  30. Tadmor O, Wexler Y, Rosenwein T, Shalev-Shwartz S, Shashua A (2016) Learning a metric embedding for face recognition using the multibatch method. In: Proceedings of the NIPS, pp 10–11

  31. Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: Proceedings of the European conference on computer vision (ECCV), pp 499–515

  32. Wang D, Otto C, Jain AK (2017) Face search at scale. IEEE Trans Pattern Anal Mach Intell 39(6):1122–1136

    Article  Google Scholar 

  33. Sankaranarayanan, Swami AA, Castillo CD, Chellappa R (2016) Triplet probabilistic embedding for face verification and clustering. In: 2016 IEEE 8th international conference on biometrics theory, applications and systems (BTAS), pp 1-8. IEEE

  34. Tran L, Yin X, Liu X (2017) Disentangled representation learning GAN for pose-invariant face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1–10

  35. Peng X, Yu X, Sohn K, Metaxas DN, Chandraker M (2017) Reconstruction-based disentanglement for pose-invariant face recognition. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1–10

  36. Patacchiola M, Cangelosi A (2017) Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods. Pattern Recognit 71:132–143

    Article  Google Scholar 

  37. Sreekanth P, Kulkarni U, Shetty S, S M M (2018) Head Pose Estimation using Transfer Learning, 2018 International Conference on Recent Trends in Advance Computing (ICRTAC), Chennai, India, , pp. 73-79

  38. Zavan FHDB, Bellon OR, Silva L, Medioni GG (2019) Benchmarking parts-based face processing in-the-wild for gender recognition and head pose estimation. Pattern Recognit Lett 123:104–110

    Article  Google Scholar 

  39. Ranjan R, Patel VM, Chellappa R (2017) Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Trans Pattern Anal Mach Intell 41(1):121–135

    Article  Google Scholar 

  40. Xie W, Zisserman A (2018) Multicolumn networks for face recognition. arXiv preprint arXiv:1807.09192

  41. Kemelmacher-Shlizerman I, Seitz SM, Miller D, Brossard E (2016) The MegaFace benchmark: 1 million faces for recognition at scale. In: IEEE conference on computer vision and pattern recognition (CVPR)

  42. Masi I, Tran AT, Hassner T, Leksut JT, Medioni G (2016) Do we really need to collect millions of faces for effective face recognition? In European Conference on Computer Vision (ECCV)

  43. Huang C, Li Y, Loy CC, Tang X (2016) Learning deep representation for imbalanced classification. In: CVPR

  44. Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: European conference on computer vision, pp 499–515. Springer

  45. Liu W, Wen Y, Yu Z, Yang M (2016) Large-margin softmax loss for convolutional neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 507–516

  46. Wu Y, Liu H, Li J, Fu Y (2017) Deep face recognition with center invariant loss. In: Thematic Workshop of ACM-MM

  47. Liu W, Wen Y, Yu Z, Li M, Raj B, Song L (2017) Sphereface: Deep hypersphere embedding for face recognition. In: IEEE conf Comput Vis Pattern Recogt, volume 1

  48. Liu W, Wen Y, Yu Z, Li M, Raj B, Song L (2017) Sphereface: deep hypersphere embedding for face recognition. In: IEEE conference on computer vision and pattern recognition (CVPR)

  49. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) Cosface: large margin cosine loss for deep face recognition. In: CVPR

  50. Zhang X, Fang Z, Wen Y, Li Z, Qiao Y (2017) Range loss for deep face recognition with long-tailed training data. In: IEEE international conference on computer vision (ICCV)

  51. Yin X, Yu X, Sohn K, Liu X, Chandraker M (2018) Feature transfer learning for deep face recognition with long-tail data, arXiv preprint, arXiv:1803.09014

  52. Zhong Y, Deng W, Wang M, et al (2019) Unequal-training for deep face recognition with long-tailed noisy data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7812–7821

  53. Huang C, Li Y, Chen CL, et al (2019) Deep imbalanced learning for face recognition and attribute prediction. IEEE Trans Pattern Anal Mach Intell

  54. Zhao K, Xu J, Cheng M-M (2019) Regularface: Deep face recognition via exclusive regularization. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 1136–1144

  55. Deng J, Guo J, Xue N, et al (2019) Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 4690–4699

  56. Liu B, Deng W, Zhong Y, et al (2019) Fair Loss: margin-Aware Reinforcement Learning for Deep Face Recognition. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 10052–10061

  57. Schroff F, Kalenichenko D, Philbin J (2015) FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) , pp 815–82

  58. Elharrouss O, Almaadeed N, Al-Maadeed S (2020) February. LFR face dataset: Left-Front-Right dataset for pose-invariant face recognition in the wild. In: 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT) (pp 124-130). IEEE

  59. Elharrouss O, Moujahid D, Tairi H (2015) Motion detection based on the combining of the background subtraction and the structure–texture decomposition. Optik-Int JLight Electron Opt 126(24):5992–5997

    Article  Google Scholar 

  60. Geusebroek J-M, van den Boomgaard R, Smeulders AWM, Dev A (2000) Color and scale: the spatial structure of color images. In: Proceeding of the 6th European conference on computer vision (ECCV), vol 1, pp 331–341

  61. Zhou H, Chen Y, Feng R (2013) A novel background subtraction method based on color invariants. Comput Vis Image Understand 117(11):1589–1597

    Article  Google Scholar 

  62. Simonyan K, Zisserman A (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  63. Wang L, Ge L, Li R et al (2017) Three-stream CNNs for action recognition. Pattern Recognit Lett 92:33–40

    Article  Google Scholar 

  64. Gourier N, Hall D, James L (2004) Crowley. Estimating face orientation from robust detection of salient facial structures. In: FG Net workshop on visual observation of deictic gestures, vol. 6, p. 7. FGnet (IST-2000-26434) Cambridge, UK

  65. Sengupta S et al (2016) Frontal to profile face verification in the wild. In: 2016 IEEE winter conference on applications of computer vision (WACV). IEEE

  66. Wolf L, Hassner T, Maoz, FI (2011) recognition in unconstrained videos with matched background similarity. In: IEEE conference on computer vision and pattern recognition (CVPR)

  67. Masi I, Rawls S, Medioni G, Natarajan P (2016) Pose-aware face recognition in the wild. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 4838–4846

  68. Lee H, et al (2017) Structure-texture decomposition of images with interval gradient. In: Computer graphics forum. Wiley Online Library

  69. Yin X, Liu X (2018) Multi-task convolutional neural network for pose-invariant face recognition. IEEE Trans Image Process 27(2):964–975

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This publication was made by NPRP Grant # NPRP8-140-2-065 from the Qatar National Research Fund (a member of the Qatar Foundation). The statements made herein are solely the responsibility of the authors.

Funding

This publication was made by NPRP Grant # NPRP8-140-2-065 from the Qatar National Research Fund (a member of the Qatar Foundation). The statements made herein are solely the responsibility of the authors.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Qatar University, Doha, Qatar

    Omar Elharrouss, Noor Almaadeed & Somaya Al-Maadeed

  2. Department of Computer Science and Digital Technologies, Northumbria University, Newcastle Upon Tyne, Newcastle, UK

    Fouad Khelifi

Authors
  1. Omar Elharrouss
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Noor Almaadeed
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Somaya Al-Maadeed
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fouad Khelifi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Omar Elharrouss.

Ethics declarations

Conflict of interest

I certify that there is no actual or potential conflict of interest in relation to this article.

Conflict of interest

The authors declare that they have no competing interests.

Financial interests

I certify that there is no financial interest in relation to this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Elharrouss, O., Almaadeed, N., Al-Maadeed, S. et al. Pose-invariant face recognition with multitask cascade networks. Neural Comput & Applic 34, 6039–6052 (2022). https://doi.org/10.1007/s00521-021-06690-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06690-4

Keywords

Search

Navigation