Blind Face Restoration Survey

Sait Sharipov, Bulat Nutfullin, Narek Maloyan

Abstract


The importance of researching methods for blind face restoration (BFR) arises from their potential practical applications in various domains. Examples of such areas include digital art and computer graphics for character face reconstruction and animation, as well as social networks and mobile applications, where they contribute to improving the quality of images and videos. In this paper, we conduct a review of contemporary methods and approaches used for solving the BFR problem. We examine various types of models based on generative adversarial networks, autoencoders, and diffusion models, which have demonstrated significant progress in this field. Specifically, we analyze key aspects such as network architecture, loss functions, quality metrics, and datasets. Furthermore, we discuss the issues and limitations of existing methods, as well as possible directions for future research. In particular, we emphasize the need for developing algorithms that are robust to various degradations and capable of adapting to different lighting conditions, poses, and facial expressions. In conclusion, we provide a systematic comparison of existing methods and summarize their merits and drawbacks.

Full Text:

PDF (Russian)

References


Image super-resolution using deep convolutional networks / Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang // IEEE transactions on pattern analysis and machine intelligence. — 2015. — Vol. 38, no. 2. — P. 295–307.

Enhanced deep residual networks for single image super-resolution / Bee Lim, Sanghyun Son, Heewon Kim et al. //Proceedings of the IEEE conference on computer vision and pattern recognition workshops. — 2017. — P. 136–144.

Esrgan: Enhanced super-resolution generative adversarial networks / Xintao Wang, Ke Yu, Shixiang Wu et al. // Proceedings of the European conference on computer vision (ECCV) workshops. — 2018. — P. 0–0.

Image super-resolution using very deep residual channel attention networks / Yulun Zhang, Kunpeng Li, Kai Li et al. // Proceedings of the European conference on computer vision (ECCV). — 2018. — P. 286–301.

Photo-realistic single image super-resolution using a generative adversarial network / Christian Ledig, Lucas Theis, Ferenc Huszár et al. //Proceedings of the IEEE conference on computer vision and pattern recognition. — 2017. — P. 4681–4690.

Sajjadi Mehdi SM, Scholkopf Bernhard, Hirsch Michael. Enhancenet: Single image super-resolution through automated texture synthesis //Proceedings of the IEEE international conference on computer vision. — 2017. — P. 4491–4500.

Variational denoising network: Toward blind noise modeling and removal / Zongsheng Yue, Hongwei Yong, Qian Zhao et al. // Advances in neural information processing systems. — 2019. — Vol. 32.

Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising / Kai Zhang, Wangmeng Zuo, Yunjin Chen et al. // IEEE transactions on image processing. — 2017. — Vol. 26, no. 7. — P. 3142–3155.

Zhang Kai, Zuo Wangmeng, Zhang Lei. Ffdnet: Toward a fast and flexible solution for cnn-based image denoising // IEEE Transactions on Image Processing. — 2018. — Vol. 27, no. 9. — P. 4608–4622.

Deblurgan: Blind motion deblurring using conditional adversarial networks / Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych et al. // Proceedings of the IEEE conference on computer vision and pattern recognition. — 2018. — P. 8183–8192.

Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better / Orest Kupyn, Tetiana Martyniuk, Junru Wu, Zhangyang Wang // Proceedings of the IEEE/CVF international conference on computer vision. — 2019. — P. 8878–8887.

Deblurring by realistic blurring / Kaihao Zhang, Wenhan Luo, Yiran Zhong et al. // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. — 2020. — P. 2737–2746.

Adversarial spatio-temporal learning for video deblurring / Kaihao Zhang, Wenhan Luo, Yiran Zhong et al. // IEEE Transactions on Image Processing. — 2018. — Vol. 28, no. 1. — P. 291–301.

Compression artifacts reduction by a deep convolutional network / Chao Dong, Yubin Deng, Chen Change Loy, Xiaoou Tang // Proceedings of the IEEE international conference on computer vision. — 2015. — P. 576–584.

Jpeg artifacts reduction via deep convolutional sparse coding / Xueyang Fu, Zheng-Jun Zha, Feng Wu et al. // Proceedings of the IEEE/CVF International Conference on Computer Vision. — 2019. — P. 2501–2510.

Deep learning vs. traditional computer vision / Niall O’Mahony, Sean Campbell, Anderson Carvalho et al. // Advances in Computer Vision: Proceedings of the 2019 Computer Vision Conference (CVC), Volume 1 1 / Springer. — 2020. — P. 128–144.

Wang Zhou, Bovik Alan C. A universal image quality index // IEEE signal processing letters. — 2002. — Vol. 9, no. 3. — P. 81–84.

The unreasonable effectiveness of deep features as a perceptual metric / Richard Zhang, Phillip Isola, Alexei A Efros et al. // Proceedings of the IEEE conference on computer vision and pattern recognition. — 2018. — P. 586–595.

Gans trained by a two time-scale update rule converge to a local nash equilibrium / Martin Heusel, Hubert Ramsauer, Thomas Unterthiner et al. // Advances in neural information processing systems. — 2017. — Vol. 30.

Mittal Anish, Soundararajan Rajiv, Bovik Alan C. Making a “completely blind” image quality analyzer // IEEE Signal processing letters. — 2012. — Vol. 20, no. 3. — P. 209–212.

Attention-aware face hallucination via deep reinforcement learning / Qingxing Cao, Liang Lin, Yukai Shi et al. // Proceedings of the IEEE conference on computer vision and pattern recognition. — 2017. — P. 690–698.

Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution / Huaibo Huang, Ran He, Zhenan Sun, Tieniu Tan // Proceedings of the IEEE international conference on computer vision. — 2017. — P. 1689–1697.

Learning to super-resolve blurry face and text images / Xiangyu Xu, Deqing Sun, Jinshan Pan et al. // Proceedings of the IEEE international conference on computer vision. — 2017. — P. 251–260.

Fsrnet: End-to-end learning face super-resolution with facial priors / Yu Chen, Ying Tai, Xiaoming Liu et al. // Proceedings of the IEEE conference on computer vision and pattern recognition. — 2018. — P. 2492–2501.

Face super-resolution guided by facial component heatmaps / Xin Yu, Basura Fernando, Bernard Ghanem et al. // Proceedings of the European conference on computer vision (ECCV). — 2018. — P. 217–233.

Progressive semantic-aware style transformation for blind face restoration / Chaofeng Chen, Xiaoming Li, Lingbo Yang et al. // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. — 2021. — P. 11896–11905.

Deep semantic face deblurring / Ziyi Shen, Wei-Sheng Lai, Tingfa Xu et al. // Proceedings of the IEEE conference on computer vision and pattern recognition. — 2018. — P. 8260–8269.

Hifacegan: Face renovation via collaborative suppression and replenishment / Lingbo Yang, Shanshe Wang, Siwei Ma et al. // Proceedings of the 28th ACM international conference on multimedia. — 2020. — P. 1551–1560.

Face super-resolution guided by 3d facial priors / Xiaobin Hu, Wenqi Ren, John LaMaster et al. // Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16 / Springer. — 2020. — P. 763–780.

Face video deblurring using 3d facial priors / Wenqi Ren, Jiaolong Yang, Senyou Deng et al. // Proceedings of the IEEE/CVF international conference on computer vision. — 2019. — P. 9388– 9397.

Learning warped guidance for blind face restoration / Xiaoming Li, Ming Liu, Yuting Ye et al. // Proceedings of the European conference on computer vision (ECCV). — 2018. — P. 272–289.

Blind face restoration via deep multi-scale component dictionaries / Xiaoming Li, Chaofeng Chen, Shangchen Zhou et al. // Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16 / Springer. — 2020. — P. 399– 415.

Zhou Shangchen, Chan Kelvin C. K., Li Chongyi, Loy Chen Change. Towards robust blind face restoration with codebook lookup transformer. — 2022. — 2206.11253.

Vqfr: Blind face restoration with vector-quantized dictionary and parallel decoder / Yuchao Gu, Xintao Wang, Liangbin Xie et al. // Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVIII / Springer. — 2022. — P. 126–143.

Restoreformer: High-quality blind face restoration from undegraded key-value pairs / Zhouxia Wang, Jiawei Zhang, Runjian Chen et al. // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. — 2022. — P. 17512–17521.

Van Den Oord Aaron, Vinyals Oriol et al. Neural discrete representation learning // Advances in neural information processing systems. — 2017. — Vol. 30.

Esser Patrick, Rombach Robin, Ommer Bjorn. Taming transformers for high-resolution image synthesis // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. — 2021. — P. 12873–12883.

Pulse: Self-supervised photo upsampling via latent space exploration of generative models / Sachit Menon, Alexandru Damian, Shijia Hu et al. // Proceedings of the ieee/cvf conference on computer vision and pattern recognition. — 2020. — P. 2437–2445.

Karras Tero, Laine Samuli, Aila Timo. A style-based generator architecture for generative adversarial networks // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. — 2019. — P. 4401–4410.

Gan prior embedded network for blind face restoration in the wild / Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. — 2021. — P. 672–681.

Towards real-world blind face restoration with generative facial prior / Xintao Wang, Yu Li, Honglun Zhang, Ying Shan // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. — 2021. — P. 9168–9178.

Dhariwal Prafulla, Nichol Alexander. Diffusion models beat gans on image synthesis // Advances in Neural Information Processing Systems. — 2021. — Vol. 34. — P. 8780–8794.

Wang Yinhuai, Yu Jiwen, Zhang Jian. Zero-shot image restoration using denoising diffusion null-space model // arXiv preprint arXiv:2212.00490. — 2022.

Yue Zongsheng, Loy Chen Change. Difface: Blind face restoration with diffused error contraction // arXiv preprint arXiv:2212.06512. — 2022.

Learning spatial attention for face super-resolution / Chaofeng Chen, Dihong Gong, Hao Wang et al. // IEEE Transactions on Image Processing. — 2020. — Vol. 30. — P. 1219–1231.

Glean: Generative latent bank for large-factor image super-resolution / Kelvin CK Chan, Xintao Wang, Xiangyu Xu et al. //Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. — 2021. — P. 14245–14254.

Faceformer: Scale-aware blind face restoration with transformers / Aijin Li, Gen Li, Lei Sun, Xintao Wang // arXiv preprint arXiv:2207.09790. — 2022.

Blind face restoration: Benchmark datasets and a baseline model / Puyang Zhang, Kaihao Zhang, Wenhan Luo et al. // arXiv preprint arXiv:2206.03697. — 2022.

Denoising diffusion restoration models / Bahjat Kawar, Michael Elad, Stefano Ermon, Jiaming Song // arXiv preprint arXiv:2201.11793. — 2022.

Dr2: Diffusion-based robust degradation remover for blind face restoration / Zhixin Wang, Xiaoyun Zhang, Ziying Zhang et al. //arXiv preprint arXiv:2303.06885. — 2023.

Swin transformer: Hierarchical vision transformer using shifted windows / Ze Liu, Yutong Lin, Yue Cao et al. // Proceedings of the IEEE/CVF international conference on computer vision. — 2021. — P. 10012–10022.

A new class of efficient adaptive filters for online nonlinear modeling / Danilo Comminiello, Alireza Nezamdoust, Simone Scardapane et al. // IEEE Transactions on Systems, Man, and Cybernetics: Systems. — 2023. — mar. — Vol. 53, no. 3. — P. 1384–1396.

Cai Changjiang, Mordohai Philippos. Do end-to-end stereo algorithms under-utilize information? — 2020. — 10.

Park Taesung, Liu Ming-Yu, Wang Ting-Chun, Zhu Jun-Yan. Semantic image synthesis with spatially-adaptive normalization. — 2019. — 1903.07291.

He Kaiming, Gkioxari Georgia, Dollár Piotr, Girshick Ross. Mask rcnn. — 2018. — 1703.06870.

Dai Jifeng, Qi Haozhi, Xiong Yuwen et al. Deformable convolutional networks. — 2017. — 1703.06211.

Compressed sensing using generative models / Ashish Bora, Ajil Jalal, Eric Price, Alexandros G Dimakis // International Conference on Machine Learning / PMLR. — 2017. — P. 537–546.

Vershynin Roman. Random vectors in high dimensions // Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press. — 2018. — Vol. 3. — P. 38–69.

Karras Tero, Laine Samuli, Aittala Miika et al. Analyzing and improving the image quality of stylegan. — 2020. — 1912.04958.

Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network / Wenzhe Shi, Jose Caballero, Ferenc Huszár et al. // Proceedings of the IEEE conference on computer vision and pattern recognition. — 2016. — P. 1874– 883.

Deep unsupervised learning using nonequilibrium thermodynamics / Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, Surya Ganguli // International Conference on Machine Learning / PMLR. — 2015. — P. 2256–2265.

Ho Jonathan, Jain Ajay, Abbeel Pieter. Denoising diffusion probabilistic models // Advances in Neural Information Processing Systems. — 2020. — Vol. 33. — P. 6840–6851.

Swinir: Image restoration using swin transformer / Jingyun Liang, Jiezhang Cao, Guolei Sun et al. // Proceedings of the IEEE/CVF international conference on computer vision. — 2021. — P. 1833– 1844.

Ilvr: Conditioning method for denoising diffusion probabilistic models / Jooyoung Choi, Sungwon Kim, Yonghyun Jeong et al. // arXiv preprint arXiv:2108.02938. — 2021.

Deep learning face attributes in the wild / Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang // Proceedings of the IEEE international conference on computer vision. — 2015. — P. 3730–3738.

Deep learning face representation by joint identification-verification / Yi Sun, Yuheng Chen, Xiaogang Wang, Xiaoou Tang // Advances in neural information processing systems. — 2014. — Vol. 27.

Learning face representation from scratch / Dong Yi, Zhen Lei, Shengcai Liao, Stan Z Li // arXiv preprint arXiv:1411.7923. — 2014.

Vggface2: A dataset for recognising faces across pose and age / Qiong Cao, Li Shen, Weidi Xie et al. // 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018) / IEEE. — 2018. — P. 67–74.

Joint face detection and alignment using multitask cascaded convolutional networks / Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, Yu Qiao // IEEE signal processing letters. — 2016. — Vol. 23, no. 10. — P. 1499–1503.

Rothe Rasmus, Timofte Radu, Van Gool Luc. Dex: Deep expectation of apparent age from a single image // Proceedings of the IEEE international conference on computer vision workshops. — 2015. — P. 10–15.

Interactive facial feature localization / Vuong Le, Jonathan Brandt, Zhe Lin et al. // Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part III 12 / Springer. — 2012. — P. 679–692.

Yang Shuo, Luo Ping, Loy Chen Change, Tang Xiaoou. Wider face: A face detection benchmark. — 2015. — 1511.06523.

Recognize complex events from static images by fusing deep channels / Yuanjun Xiong, Kai Zhu, Dahua Lin, Xiaoou Tang // Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on / IEEE. — 2015.

Labeled faces in the wild: A database for studying face recognition in unconstrained environments : Rep. : 07-49 / University of Massachusetts, Amherst ; Executor: Gary B. Huang, Manu Ramesh, Tamara Berg, Erik Learned-Miller : 2007. — October.

Jesorsky Oliver, Kirchberg Klaus J., Frischholz Robert W. Robust face detection using the hausdorff distance // Audio- and VideoBased Biometric Person Authentication / Ed. by Josef Bigun, Fabrizio Smeraldi. — Berlin, Heidelberg : Springer Berlin Heidelberg, 2001. — P. 90–95.

Annotated facial landmarks in the wild: A large-scale, realworld database for facial landmark localization / Martin Köstinger, Paul Wohlhart, Peter M. Roth, Horst Bischof // 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). — 2011. — P. 2144–2151.

Zhang Kaihao, Li Dongxu, Luo Wenhan et al. Edface-celeb-1m: Benchmarking face hallucination with a million-scale dataset. — 2022. — 2110.05031.


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность IT Congress 2024

ISSN: 2307-8162