Deep Feature Fusion for Robust Demographic Group Classification in Facial Images
Keywords:
Race classification, transfer learning, FaceNet, MTCNN, ethnicity classification, deep learningAbstract
Classifying demographic groups from facial images is challenging because differences between groups can be subtle, while variations in lighting, pose, or expression often dominate the visual signal. To address this, we propose a hybrid deep feature fusion approach that combines two complementary strengths. Faces are first detected and aligned using MTCNN to ensure consistency across samples. We then extract features in two ways: a pretrained CNN captures dataset-specific appearance cues, while FaceNet provides compact embeddings that preserve identity-related information. Instead of relying on either representation alone, we integrate both through a multilayer perceptron classifier, which learns more flexible decision boundaries in the fused space. Experiments on a curated dataset as well as UTKFace and FairFace benchmarks demonstrate that the fusion approach not only improves accuracy but also generalizes better across datasets compared to single-feature methods. These results highlight the value of combining CNN-derived and embedding-based representations, offering a more robust framework for demographic group classification and fairness-aware face analysis.
References
[1] Jain, A., Ross, A., Nandakumar, K.: 50 Years of Biometric Research. Springer, Cham (2016)
[2] Hong, J., Suh, B., Kim, S.: Culturally adaptive user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 3543–3552. ACM, New York, NY, USA (2014)
[3] Kamran, S., Adeel, M., et al.: Demographic profiling from social media images. Multimedia Tools and Applications 78(8), 10783–10804 (2019)
[4] Buolamwini, J., Gebru, T.: Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Proceedings of the 1st Conference on Fairness, Accountability and Transparency (FAT), pp. 77–91 (2018)
[5] Wang, M., Deng, W.: Addressing bias in facial recognition algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(3), 435–447 (2019)
[6] Shan, C.: Illumination normalization for robust face recognition: A survey. In: Proceedings of the ACM International Conference on Multimedia, pp. 49–56 (2011)
[7] Lu, Z., Ding, C., Tao, D.: Cross-domain race classification from facial images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6781–6790 (2020)
[8] Ojala, T., Pietikainen, M., Maenpaaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7), 971–987 (2002)
[9] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 886–893 (2005). IEEE
[10] Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning 20(3), 273–297 (1995)
[11] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248–255 (2009)
[12] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)
[13] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
[14] Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. In: CVPR (2015)
[15] Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: Additive angular margin loss for deep face recognition. CVPR (2019)
[16] Izadi, M., Saberi, A., Aghaeinia, H.: Feature fusion for gender classification in the wild. Multimedia Tools and Applications 78(3), 3649–3670 (2019)
[17] Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters 23(10), 1499–1503 (2016)
[18] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15(1), 1929–1958 (2014)
[19] Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L.: Sphereface: Deep hyper-sphere embedding for face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6738–6746 (2017)
[20] Robinson, J.P., Livitz, G., Henon, Y., Qin, C., Fu, Y., Timoner, S.: Face recognition: Too bias, or not too bias? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1–10. IEEE, Seattle, WA, USA (2020). https://doi.org/10.1109/CVPRW50498.2020.
00008
[21] Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. In: IEEE Signal Processing Letters, vol. 23, pp. 1499–1503 (2016)
[22] Zafeiriou, S., Zhang, C., Zhang, Z.: A survey on face detection in the wild: Past, present and future. Computer Vision and Image Understanding 138, 1–24 (2015)
[23] Izadi, S., Abdollahi, A., Demirel, H.: Feature-level fusion from facial attributes for face recognition. arXiv preprint arXiv:1909.13126 (2019)
[24] Ranjan, R., Castillo, C., Chellappa, R.: Deep learning for understanding faces. In: Li, S.Z., Jain, A.K. (eds.) Handbook of Face Recognition
[25] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016)
[26] Baig, T.I., Alam, T.M., Anjum, T., Naseer, S., Wahab, A., Imtiaz, M., Raza, M.M.: Classification of human face: Asian and non-asian people. In: 2019 International Conference on Innovative Computing (ICIC), pp. 1–6 (2019). https:
//doi.org/10.1109/ICIC48496.2019.8966721
[27] Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5810–5818 (2017). IEEE
[28] Karkkainen, K., Joo, J.: Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1548–1558
(2021)
Additional Files
Published
Issue
Section
License
Copyright (c) 2025 Tayaba Anjum, Naila Hamid
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright and grant the journal the right of first publication, with the work simultaneously licensed under a CC-BY 4.0 License.
