Bir Bhanu, Center for Research in Intelligent Systems
Room 216, Winston Chung Hall, University of California at Riverside,
Riverside, CA 92521,
Tel. 951-827-3954, Fax. 951-827-2425
Mark Campbell (Cornell University)
Chinya Ravishankar (UCR)
Amit K Roy Chowdhury (UCR)
Le An, Graduate Student (research assistant)
Anirban Chakraborty, Graduate Student (research assistant)
Jessica Gregory, Graduate Student (research assistant)
Zhixin Jin, Graduate Student (research assistant)
Priyanka Khire, Graduate Student (research assistant)
Yingying Zhu, Graduate Student (research assistant)
- Major Goals of the Project:
- (a) To develop a synergistic framework and algorithms for a group of fixed and mobile
(ground and aerial) sensors to collaborate on scene understanding in disaster preparedness,
response, and recovery scenarios, which are characterized by highly dynamic and uncertain environments.
- (b) To perform a tight integration of perception and action in a probabilistic framework for
truly intelligent robotic systems, and to advance the field of cyberphysical systems by exploring
a new class of synergies across three areas: control, video understanding and perception,
and data management under uncertainty.
- (c) To experimentally validate the framework and algorithms by applying them to the domain of
surveillance, using a testbed incorporating autonomous agents, including mobile and fixed cameras,
robots, and unmanned ground or aerial vehicles.
- Accomplishments under these goals:
- 1) Research in multitarget multisensor tracking and reidentification, and online adaptation of the learned models.
2) Fundamental theory, algorithm research and experimental validation in the areas of: distributed estimation
and planning for spatiotemporal processes, distributed people tracking from mobile robots,and optimal planning
for uncertain processes such as exploration.
- Specific Objectives:
- 1) Develop methods for tracking multiple targets in a network of sensors by considering the physical constraints from the scene geometry.
2) Develop methods for online adaptation of the learned models as more data is available.
3) Develop distributed estimation and information planning algorithms to enable a wider
range of behaviors in cooperative robotic teams, which subsequently will enable a better
handling of uncertainty, and improved sensing, information collection and decision making.
- Significant Results:
A) Person reidentification aims at matching people in nonoverlapping cameras at different
time and locations. It is a difficult pattern matching task due to significant appearance
variations in pose, illumination, or occlusion in different camera views. We have developed two approaches:
- 1) In the first approach we first learn a subspace using canonical correlation analysis (CCA)
in which the goal is to maximize the correlation between data from different cameras but corresponding
to the same people. Given a probe from one camera view, we represent it using a sparse representation
from a jointly learned coupled dictionary in the CCA subspace. The L1 induced sparse representation are
regularized by an L2 regularization term which allows learning a sparse representation while maintaining
the stability of the sparse coefficients. To compute the matching scores between probe and gallery,
their L2 regularized sparse representations are matched using a modified cosine similarity measure.
Experimental results show that the proposed method outperforms the stateoftheart methods. (Lead: Bhanu)
- 2) In the second approach the matching is conducted in a reference space where the descriptor for a
person is translated from the original color or texture descriptors tosimilarity measures between this
person and the exemplars in the reference set. A subspace is first learned in which the correlationsof
the reference data from different cameras are maximized using regularized canonical correlation analysis
(RCCA). For reidentification, the gallery data and the probe data are projected onto this RCCA subspace
and the reference descriptors (RDs) of the gallery and probe are generated by computing the similarity
between them and the reference data. The identity of a probe is determined by comparing the RD of the
probe and the RDs of the gallery. A reranking step is added to further improve theresults using a saliencybased
matching scheme. Experiments on publicly available datasets show that the proposed method outperforms most
of the stateoftheart approaches. (Lead: Bhanu)
- Robust Canonical Correlation Analysis (ROCCA): Given a small training set direct application of canonical
correlation analysis (CCA) may lead to poor performance due to the inaccuracy in estimating the data
covariance matrices. We have proposed ROCCA with shrinkage estimation and smoothing technique is simple
to implement and can robustly estimate the data covariance matrices with limited training samples. Experimental
results on two publicly available datasets show that the proposed ROCCA outperforms regularized CCA (RCCA), and
achieves stateoftheart matching results for person reidentification as compared to the most recent methods. (Lead: Bhanu)
B) Multitarget Tracking and Data Association in a Sensor Network Considering Scene Constratins: Existing
data association techniques mostly focus on matching pairs of datapoint sets and then repeating this
process along spacetime to achieve long term correspondences. However, in many problems such as person
reidentification, a set of datapoints may be observed at multiple spatiotemporal locations and/or by multiple
agents in a network and simply combining the local pairwise association results between sets of datapoints
often leads to inconsistencies over the global spacetime horizons. In this work, we propose a novel Network
Consistent Data Association (NCDA) framework formulated as an optimization problem that not only maintains
consistency in association results across the network, but also improves the pairwise data association accuracies.
The proposed NCDA can be solved as a binary integer program leading to a globally optimal solution and is capable
of handling the challenging dataassociation scenario where the number of datapoints varies across different
sets of instances in the network. We also present an online implementation of NCDA method that can dynamically
associate new observations to already observed datapoints in an iterative fashion, while maintaining network consistency. (Lead: Roy Chowdhury)
- 1) Online Adaptation of Learned Models in Sensor Networks: Most traditional multi-camera person reidentification
systems rely on learning a static model on tediously labeled training data. Such a framework may not be suitable
for situations when new data arrives continuously or all the data is not available for labeling beforehand.
Inspired by the ‘value of information’ active learning framework, we propose a continuous learning person
reidentification system with a human in the loop. The human in the loop not only provides labels to the
incoming images but also improves the learned model by providing most appropriate attribute based explanations.
These attribute based explanations are used to learn attribute predictors along the way. The overall effect of
such a stratgey is that starting with a few annotated images, the system begins to improve via a symbiotic
relationship between the man and the machine. The machine assists the human to speed the annotation and the
human assists the machine to update itself with more annotation so that more and more distinct persons are
reidentified as more and more images come in. (Lead: Roy Chowdhury)
C) Planning and Discovering Assemblies of Moving Objects Using Partial Information: Consider objects moving in a
road network (e.g., vehicles, people), who may be free to choose routes, yet be required to arrive at certain
locations at certain times. Planning for such assemblies is hard when the network or the number of objects is
large. Conversely, discovering actual or potential assemblies of such objects is important in many security,
applications. This can be hard when object arrival observations are sparse due to inadequate sensor coverage or
object countermeasures. We have proposed the novel class of assembly queries to model these scenarios, and present
a unified scheme that addresses both of these complementary challenges. Given a set of objects and arrival constraints,
we show how to first obtain the set of all possible locations visited by each moving object, and then determine all
possible assemblies, including the participants, locations, and durations. We present a formal model for various
tracking strategies and several algorithms for using these strategies. We achieve excellent performance using
Contraction Hierarchies. (Lead: Ravishankar)
- 1) Inferential TimeDecaying Bloom FIlters: TimeDecaying Bloom Filters are probabilistic structures to answer
queries on recently inserted items. As new items are inserted, memory of older items decays. Wrong query
responses penalize the application using the filter. Current filters may only be tuned to static penalties,
and ignore Bayesian priors and much information latent in the filter. We address these issues by introducing
inferential filters, which integrate Bayesian priors and information latent in filters to make optimal
queryspecific decisions. Our methods are general, but here we focus on inferential timedecaying filters,
and show how to support novel query types and sliding window queries with varying error penalties. (Lead: Ravishankar)
D) Multisensor Data Fusion Methodology for Fusing Lidar and Vision on a Mobile Robot for Tracking People:
Both data association and tracking are considered uncertain. Empirically evaluated the work in a lengthy set
of tests outdoors, with multiple pedestrians, occlusions, and environmental conditions. An MS thesis was
ompleted on the subject. Also, developed an information based exploration planner with information goals,
and information constraints. Formal proofs were defined, and empirical evaluations were conducted. A receding
horizon approach to real time work was developed using a novel tail cost approximation. A conference paper
was presented on this subject, and a journal paper is nearing submital. (Lead: Mark Campbell, Cornell )
- Key Outcomes or other Achievements:
- Publications in major conferences and journals.
- Software has been released along with the papers.
- New dataset for multicamera person tracking and reidentification.
- New dataset for object recognition in unconstrained environments.
- Algorithms and software for lidar/vision data fusion for tracking people from moving robots, and an information optimal exploration planner.
- What opportunities for training and professional development has the project provided?
- Six graduate students have been partially supported by this grant. Four of the students have completed their PhDs and 2 more are PhD candidates.
- At Cornell university three graduate students have been partially supported by this grant, with one finishing with an MS and the other two still in the program.
- How have the results been disseminated to communities of interest?
- Publications, software and datasets.
- What do you plan to do during the next reporting period to accomplish the goals?
- We will focus on online and semisupervised learning approaches for adaptation of scene understanding models in unconstrained environments.
- Attribute-based learning methods with uncertainty associated with objects.
- Study the distributed version of the information exploration planner.
Journals or Juried Conference Papers
- A. Chakraborty, A. Das, A. RoyChowdhury (2016). Network Consistent Data Association. IEEE Trans. on Pattern Analysis and Machine Intelligence. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- A. Das, A. Chakraborty, A. RoyChowdhury (2014). Consistent Reidentification In A Camera Network. European Conf. on Computer Vision. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- A. Das, N. Martinel, C. Micheloni, A. RoyChowdhury (2015). ReIdentification in the Function Space of Feature Warps. IEEE Trans. on Pattern Analysis and Machine Intelligence. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- A. T. Kamal, J. H. Bappy, J. A. Farrell, A. RoyChowdhur (2016). Distributed MultiTarget Tracking and Data Association in Vision Networks. IEEE Trans. on Pattern Analysis and Machine Intelligence. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- Alex Ivanov, Mark Campbell (2016). An Efficient Robotic Exploration Planner with Probabilistic Guarantees. IEEE International Conference on Robotics and Automation. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- L. An, M. Kafai, S. Yang and B. Bhanu (2016). Person reidentification with reference descriptor. IEEE Transactions on Circuits and Systems for Video Technology. 26 (4), 776. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- L. An, S. Yang and B. Bhanu (2015). Person reidentification by robust canonical correlation analysis. IEEE Signal Processing Letters. 22 (8), 1103. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- L. An, X. Chen, S. Yang and B. Bhanu, (2016). Sparse representation matching for person reidentification. Information Sciences. 335356 74. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- Mark Campbell, Nisar Ahmed (2016). Distributed Data Fusion: Neighbors, Rumors, and the Art of Collective Knowledge. IEEE Control Systems. 36 (4), 83. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes ; DOI: 10.1109/MCS.2016.2558444
- R. Panda, A. Das, A. RoyChowdhury (2016). Video Summarization in a MultiView Camera Network. International Conf. on Pattern Recognition. . Status = ACCEPTED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- Reaz Uddin, Vassilis Tsotras, Chinya Ravishankar (). Indexing of Approximate Spatiotemporal Trajectories Using Hilbert Curves. Juried conference paper. . Status = OTHER; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
- X. Chen, L. An, and B. Bhanu (2015). Multitarget tracking in nonoverlapping cameras using a reference set. IEEE Sensors Journal. 15 (5), 2692. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
Other Conference Papers and Presentations
- A. Das, A. RoyChowdhury (2015). Active Image Pair Selection for Continuous Person Reidentification. IEEE Intl. Conf. on Image Processing. . Status = PUBLISHED; Acknowledgement of Federal Support = Yes
- R. Uddin, V. Tsotras and C. Ravishankar (2016). Assembly Queries: Planning and Discovering Assemblies of Moving Objects Using Partial Information. PVLDB Journal. . Status = UNDER_REVIEW; Acknowledgement of Federal Support = Yes
- Various Softwares Under Development.
- Lucas de la Garza. MultiObject, MultiSensor Detection and Tracking of Pedestrians on a Mobile Robot. (2016). Cornell University. Acknowledgement of Federal Support = Yes
This material is based upon work supported by the National Science Foundation Project ID No. CNS-1330110. Any opinions,
findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.