91640.pdf (4.05 MB)
Download file

Learning Modality-Consistency Feature Templates: A Robust RGB-Infrared Tracking System

Download (4.05 MB)
journal contribution
posted on 25.03.2019, 17:08 by Xiangyuan Lan, Mang Ye, Rui Shao, Bineng Zhong, Pong C. Yuen, Huiyu Zhou
With a large number of video surveillance systems installed for the requirement from industrial security, the task of object tracking, which aims to locate objects of interest in videos, is very important. Although numerous tracking algorithms for RGB videos have been developed in the decade, the tracking performance and robustness of these systems may be degraded dramatically when the information from RGB video is unreliable (e.g. poor illumination conditions or very low resolution). To address this issue, this paper presents a new tracking system which aims to combine the information from RGB and infrared modalities for object tracking. The proposed tracking systems is based on our proposed machine learning model. Particularly, the learning model can alleviate the modality discrepancy issue under the proposed modality consistency constraint from both representation patterns and discriminability, and generate discriminative feature templates for collaborative representations and discrimination in heterogeneous modalities. Experiments on a variety of challenging RGB-infrared videos demonstrate the effectiveness of the proposed algorithm.


This work was supported in part by Hong Kong Research Grants Council RGC/HKBU12254316 and Hong Kong Baptist University Tier 1 Start-up Grant. The work of H. Zhou was supported by UK EPSRC under Grant EP/N011074/1, Royal Society-Newton Advanced Fellowship under Grant NA160342, and European Union’s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No. 720325. The work of B. Zhong was supported by the National Natural Science Foundation of China under Grant 61572205.



IEEE Transactions on Industrial Electronics, 2019

Author affiliation

/Organisation/COLLEGE OF SCIENCE AND ENGINEERING/Department of Informatics


AM (Accepted Manuscript)

Published in

IEEE Transactions on Industrial Electronics


Institute of Electrical and Electronics Engineers (IEEE)



Acceptance date


Copyright date


Available date


Publisher version