A neural network approach for occlusion detection in video

Maxim Velikanov, Alexandra Anzina, Sergey Lavrushkin, Dmitry Vatolin

Abstract


Occlusions are a set of pixels, which are visible in a single frame of two sequential frames in a video. Finding occlusions is of great importance in the field of computer vision. Precise detection of occlusions will improve the accuracy of many video processing methods, such as: frame interpolation, optical flow calculation, color propagation etc. The majority of existing methods are based on optimization of an energy function, which is computationally expensive. It is also worth noting that accurate estimation of occlusions is hard with no information about movement between frames, and knowledge of occlusions during optical flow estimation allows the algorithm to avoid wrong correspondences between pixels of frames. Taking this into consideration we present a novel method of occlusion detection based on PWC-net, an optical flow calculation algorithm. The key idea is to construct a pyramid of features with different resolutions for frame processing. This way of processing originates from a common computer graphics technique and is widely adopted. We also performed a comparison of our method with 15 similar methods on the MPI-Sintel dataset.

Full Text:

PDF (Russian)

References


Moving gradients: a path-based method for plausible image

interpolation / Dhruv Mahajan, Fu-Chung Huang, Wojciech Matusik

et al. // ACM Transactions on Graphics (TOG) / ACM. –– Vol. 28. ––

–– P. 42.

Motion-compensated frame interpolation using bilateral motion

estimation and adaptive overlapped block motion compensation /

Byeong-Doo Choi, Jong-Woo Han, Chang-Su Kim, Sung-Jea Ko //

IEEE Transactions on Circuits and Systems for Video Technology. ––

–– Vol. 17, no. 4. –– P. 407–416.

Cremers Daniel, Soatto Stefano. Motion competition: A variational

approach to piecewise parametric motion segmentation // International

Journal of Computer Vision. –– 2005. –– Vol. 62, no. 3. –– P. 249–265.

Chang Michael M, Tekalp A Murat, Sezan M Ibrahim. Simultaneous

motion estimation and segmentation // IEEE transactions on image

processing. –– 1997. –– Vol. 6, no. 9. –– P. 1326–1333.

Ascenso Joao, Brites Catarina, Pereira Fernando. Improving frame

interpolation with spatial motion smoothing for pixel domain

distributed video coding // 5th EURASIP Conference on Speech

and Image Processing, Multimedia Communications and Services /

Citeseer. –– 2005. –– P. 1–6.

Puri A, Hang H-M, Schilling D. An efficient block-matching algorithm

for motion-compensated coding // ICASSP’87. IEEE International

Conference on Acoustics, Speech, and Signal Processing / IEEE. ––

Vol. 12. –– 1987. –– P. 1063–1066.

Symmetrical dense optical flow estimation with occlusions detection /

Luis Alvarez, Rachid Deriche, Théo Papadopoulo, Javier Sánchez //

International Journal of Computer Vision. –– 2007. –– Vol. 75, no. 3. ––

P. 371–385.

Computing visual correspondence with occlusions via graph cuts :

Rep. / Cornell University ; Executor: Vladimir Kolmogorov,

Ramin Zabih : 2001.

Strecha Christoph, Fransens Rik, Van Gool Luc. A probabilistic

approach to large displacement optical flow and occlusion detection //

International Workshop on Statistical Methods in Video Processing /

Springer. –– 2004. –– P. 71–82.

Hur Junhwa, Roth Stefan. Mirrorflow: Exploiting symmetries in joint

optical flow and occlusion estimation // Proceedings of the IEEE

International Conference on Computer Vision. –– 2017. –– P. 312–321.

Determining occlusions from space and time image reconstructions /

Juan-Manuel Pérez-Rúa, Tomas Crivelli, Patrick Bouthemy,

Patrick Pérez // Proceedings of the IEEE Conference on Computer

Vision and Pattern Recognition. –– 2016. –– P. 1382–1391.

Deepmatching: Hierarchical deformable dense matching /

Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui,

Cordelia Schmid // International Journal of Computer Vision. ––

–– Vol. 120, no. 3. –– P. 300–323.

Lee Kyong Joon, Yun Il Dong. Occlusion detecting window matching

scheme for optical flow estimation with discrete optimization // Pattern

Recognition Letters. –– 2017. –– Vol. 89. –– P. 73–80.

Humayun Ahmad, Mac Aodha Oisin, Brostow Gabriel J. Learning to

find occlusion regions // CVPR 2011 / IEEE. –– 2011. –– P. 2161–2168.

Ayvaci Alper, Raptis Michalis, Soatto Stefano. Sparse occlusion

detection with optical flow // International Journal of Computer

Vision. –– 2012. –– Vol. 97, no. 3. –– P. 322–338.

Sun Deqing, Sudderth Erik B, Black Michael J. Layered image motion

with explicit occlusions, temporal consistency, and depth ordering //

Advances in Neural Information Processing Systems. –– 2010. ––

P. 2226–2234.

Sun Deqing, Liu Ce, Pfister Hanspeter. Local layering for joint

motion estimation and occlusion detection // Proceedings of the IEEE

Conference on Computer Vision and Pattern Recognition. –– 2014. ––

P. 1098–1105.

Li Ang, Yuan Zejian. Symmnet: A symmetric convolutional neural

network for occlusion detection // arXiv preprint arXiv:1807.00959. ––

Occlusions, motion and depth boundaries with a generic network

for disparity, optical flow or scene flow estimation / Eddy Ilg,

Tonmoy Saikia, Margret Keuper, Thomas Brox // Proceedings of the

European Conference on Computer Vision. –– 2018. –– P. 614–630.

Ronneberger Olaf, Fischer Philipp, Brox Thomas. U-net:

Convolutional networks for biomedical image segmentation //

International Conference on Medical image computing and

computer-assisted intervention / Springer. –– 2015. –– P. 234–241.

Chaurasia Abhishek, Culurciello Eugenio. Linknet: Exploiting encoder

representations for efficient semantic segmentation // 2017 IEEE

Visual Communications and Image Processing (VCIP) / IEEE. ––

–– P. 1–4.

Bisenet: Bilateral segmentation network for real-time semantic

segmentation / Changqian Yu, Jingbo Wang, Chao Peng et al. //

Proceedings of the European Conference on Computer Vision. ––

–– P. 325–341.

Flownet 2.0: Evolution of optical flow estimation with deep networks /

Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia et al. // Proceedings of

the IEEE Conference on Computer Vision and Pattern Recognition. ––

–– P. 2462–2470.

Flownet: Learning optical flow with convolutional networks /

Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg et al. // Proceedings

of the IEEE International Conference on Computer Vision. –– 2015. ––

P. 2758–2766.

Xu Jia, Ranftl René, Koltun Vladlen. Accurate optical flow via direct

cost volume processing // Proceedings of the IEEE Conference on

Computer Vision and Pattern Recognition. –– 2017. –– P. 1289–1297.

Epicflow: Edge-preserving interpolation of correspondences for optical

flow / Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui,

Cordelia Schmid // Proceedings of the IEEE Conference on Computer

Vision and Pattern Recognition. –– 2015. –– P. 1164–1172.

Pwc-net: Cnns for optical flow using pyramid, warping, and cost

volume / Deqing Sun, Xiaodong Yang, Ming-Yu Liu, Jan Kautz //

Proceedings of the IEEE Conference on Computer Vision and Pattern

Recognition. –– 2018. –– P. 8934–8943.

Imagenet: A large-scale hierarchical image database / Jia Deng,

Wei Dong, Richard Socher et al. –– 2009.

Deep residual learning for image recognition / Kaiming He,

Xiangyu Zhang, Shaoqing Ren, Jian Sun // Proceedings of the IEEE

Conference on Computer Vision and Pattern Recognition. –– 2016. ––

P. 770–778.

A large dataset to train convolutional networks for disparity, optical

flow, and scene flow estimation / Nikolaus Mayer, Eddy Ilg,

Philip Hausser et al. // Proceedings of the IEEE Conference on

Computer Vision and Pattern Recognition. –– 2016. –– P. 4040–4048.

Egnal Geoffrey, Wildes Richard P. Detecting binocular

half-occlusions: empirical comparisons of four approaches //

Proceedings IEEE Conference on Computer Vision and Pattern

Recognition / IEEE. –– Vol. 2. –– 2000. –– P. 466–473.

Empirical evaluation of rectified activations in convolutional network / Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li // arXiv preprint

arXiv:1505.00853. –– 2015.

Densely connected convolutional networks / Gao Huang, Zhuang Liu,

Laurens Van Der Maaten, Kilian Q Weinberger // Proceedings of the

IEEE Conference on Computer Vision and Pattern Recognition. ––

–– P. 4700–4708.

Kingma Diederik P, Ba Jimmy. Adam: A method for stochastic

optimization // arXiv preprint arXiv:1412.6980. –– 2014.

Estellers Virginia, Soatto Stefano. Detecting occlusions as an inverse

problem // Journal of Mathematical Imaging and Vision. –– 2016. ––

Vol. 54, no. 2. –– P. 181–198.

Ha Synh Viet-Uyen, Vu Tuan-Anh, Tran Ha Manh. An extended

occlusion detection approach for video processing // REV Journal on

Electronics and Communications. –– 2019. –– Vol. 8, no. 3-4.


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность IT Congress 2024

ISSN: 2307-8162