Real-time target tracking using a PTZ camera with Delay-compensated IMM-UKF and Nonlinear Model Predictive Control

Khanh Le Van, Phong Luong Thu

Abstract


Real-time target tracking using pan–tilt–zoom (PTZ) cameras remains challenging because of nonlinear observation geometry, asynchronous visual measurements, and latency introduced by deep learning-based detection modules. These factors are particularly impactful in cases of highly maneuvering targets, where delayed measurements no longer accurately reflect the current state of the target. This paper proposes a latency-aware PTZ tracking and control architecture. The Interacting Multiple Model Unscented Kalman Filter (IMM-UKF) is utilized to estimate the target state under nonlinear measurement conditions and sudden changes in motion. To handle delayed visual measurements, a retroactive update and state repropagation mechanism (Rollback and Replay) is built on a history buffer, thereby reducing synchronization errors between the image processing pipeline and the control loop. The optical parameters of the camera are also dynamically updated according to the zoom level to maintain the accuracy of the measurement vector. From the estimated and delay-compensated target state, the Nonlinear Model Predictive Control (NMPC) on the augmented state generates Pan/Tilt commands, while an adaptive PID controller adjusts the Zoom axis. Incorporating actuator inertia and velocity-command dynamics into the predictive model improves tracking accuracy, smooths the control signals, and reduces abrupt command variations. The architecture is evaluated in simulation scenarios with highly maneuvering targets to verify tracking accuracy, closed-loop stability, and the capability to operate under delayed conditions.

Full Text:

PDF

References


S. Hutchinson, G. D. Hager, and P. I. Corke, “A tutorial on visual servo control,” IEEE Transactions on Robotics and Automation, vol. 12, no. 5, pp. 651–670, 1996.

F. Chaumette and S. Hutchinson, “Visual servo control. I. Basic approaches,” IEEE Robotics & Automation Magazine, vol. 13, no. 4, pp. 82–90, 2006.

F. Chaumette and S. Hutchinson, “Visual servo control. II. Advanced approaches [Tutorial],” IEEE Robotics & Automation Magazine, vol. 14, no. 1, pp. 109–118, 2007.

R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge, U.K.: Cambridge University Press, 2003.

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330–1334, 2000.

S. Wang, D. Liu, Y. Yang, X. Chen, P. Cao, L. Li, L. Yan, Z. Cheng, and Y. Shen, “Distortion correction in surface defects evaluating system of large fine optics,” Optics Communications, vol. 312, pp. 260–268, 2014.

Y. Zhang et al., “ByteTrack: Multi-object tracking by associating every detection box,” in Proceedings of the European Conference on Computer Vision, Cham: Springer Nature Switzerland, 2022.

N. Aharon, R. Orfaig, and B.-Z. Bobrovsky, “BoT-SORT: Robust associations multi-pedestrian tracking,” arXiv preprint arXiv:2206.14651, 2022.

D. Reis et al., “Real-time flying object detection with YOLOv8,” arXiv preprint arXiv:2305.09972, 2023.

Y. Bar-Shalom, X. R. Li, and T. Kirubarajan, Estimation with

Applications to Tracking and Navigation: Theory, Algorithms and Software. New York, NY, USA: John Wiley & Sons, 2001.

H. A. P. Blom and Y. Bar-Shalom, “The interacting multiple model algorithm for systems with Markovian switching coefficients,” IEEE Transactions on Automatic Control, vol. 33, no. 8, pp. 780–783, Aug. 1988.

R. A. Singer, “Estimating optimal tracking filter performance for manned maneuvering targets,” IEEE Transactions on Aerospace and Electronic Systems, vol. AES-6, no. 4, pp. 473–483, July 1970.

E. Mazor, A. Averbuch, Y. Bar-Shalom, and J. Dayan, “Interacting multiple model methods in target tracking: A survey,” IEEE Transactions on Aerospace and Electronic Systems, vol. 34, no. 1, pp. 103–123, Jan. 1998.

X. R. Li and V. P. Jilkov, “Survey of maneuvering target tracking. Part I: Dynamic models,” IEEE Transactions on Aerospace and Electronic Systems, vol. 39, no. 4, pp. 1333–1364, Oct. 2003.

S. J. Julier and J. K. Uhlmann, “Unscented filtering and nonlinear estimation,” Proceedings of the IEEE, vol. 92, no. 3, pp. 401–422, Mar. 2004.

E. A. Wan and R. Van Der Merwe, “The unscented Kalman filter for nonlinear estimation,” in Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium, Lake Louise, AB, Canada, 2000, pp. 153–158.

Y. Bar-Shalom, “Update with out-of-sequence measurements in tracking: Exact solution,” IEEE Transactions on Aerospace and Electronic Systems, vol. 38, no. 3, pp. 769–777, 2002.

Y. Bar-Shalom, H. Chen, and M. Mallick, “One-step solution for the multistep out-of-sequence-measurement problem in tracking,” IEEE Transactions on Aerospace and Electronic Systems, vol. 40, no. 1, pp. 27–37, 2004.

G. Galuppini, I. Schimperna, and L. Magni, “Offset-free output feedback tracking MPC for δ-ISS nonlinear systems subject to input and input rate constraints,” IEEE Transactions on Automatic Control, 2025.

I. A. Jimoh and H. Yue, “A velocity form model predictive control of an autonomous underwater vehicle,” IEEE Journal of Oceanic Engineering, 2025.

Y. Du et al., “A visual-based target tracking framework for UAV using model predictive contouring control,” IEEE Transactions on Industrial Electronics, 2025.

R. Verschueren et al., “acados—A modular open-source framework for fast embedded optimal control,” Mathematical Programming Computation, vol. 14, no. 1, pp. 147–183, 2022.

J. Andersson et al., “CasADi—A software framework for nonlinear optimization and optimal control,” Mathematical Programming Computation, vol. 11, no. 1, pp. 1–36, 2019.

A. Amer et al., “Visual tracking nonlinear model predictive control method for autonomous wind turbine inspection,” in Proceedings of the 21st International Conference on Advanced Robotics, IEEE, 2023.

W. Zhan et al., “Target size adaptive PID control method for gimbal visual servo,” in Proceedings of the 17th International Conference on Intelligent Human-Machine Systems and Cybernetics, IEEE, 2025.

P. Shi et al., “Intelligent vehicle path tracking control based on improved MPC and hybrid PID,” IEEE Access, vol. 10, pp. 94133–94144, 2022.

E. Rohmer, S. P. N. Singh, and M. Freese, “V-REP: A versatile and scalable robot simulation framework,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2013.


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность ИТ-КОНГРЕСС ВМК МГУ 2026 СНЭ

ISSN: 2307-8162