Accomplishments

• Major Goals of the Project:

(a) To develop a synergistic framework and algorithms for a group of fixed and mobile (ground and aerial) sensors to collaborate on scene understanding in disaster preparedness, response, and recovery scenarios, which are characterized by highly dynamic and uncertain environments.




(b) To perform a tight integration of perception and action in a probabilistic framework for truly intelligent robotic systems, and to advance the field of cyber­physical systems by exploring a new class of synergies across three areas: control, video understanding and perception, and data management under uncertainty.




(c) To experimentally validate the framework and algorithms by applying them to the domain of surveillance, using a testbed incorporating autonomous agents, including mobile and fixed cameras, robots, and unmanned ground or aerial vehicles.




• Accomplishments under these goals:

Major Activities:


1) Research in multi­target multi­sensor tracking and reidentification, and online adaptation of the learned models.

2) Fundamental theory, algorithm research and experimental validation in the areas of: distributed estimation and planning for spatio­temporal processes, distributed people tracking from mobile robots,and optimal planning for uncertain processes such as exploration.



• Specific Objectives:

1) Develop methods for tracking multiple targets in a network of sensors by considering the physical constraints from the scene geometry.

2) Develop methods for online adaptation of the learned models as more data is available.

3) Develop distributed estimation and information planning algorithms to enable a wider range of behaviors in cooperative robotic teams, which subsequently will enable a better handling of uncertainty, and improved sensing, information collection and decision making.

• Significant Results:

A) Person re­identification aims at matching people in non­overlapping cameras at different time and locations. It is a difficult pattern matching task due to significant appearance variations in pose, illumination, or occlusion in different camera views. We have developed two approaches:




1) In the first approach we first learn a subspace using canonical correlation analysis (CCA) in which the goal is to maximize the correlation between data from different cameras but corresponding to the same people. Given a probe from one camera view, we represent it using a sparse representation from a jointly learned coupled dictionary in the CCA subspace. The L1 induced sparse representation are regularized by an L2 regularization term which allows learning a sparse representation while maintaining the stability of the sparse coefficients. To compute the matching scores between probe and gallery, their L2 regularized sparse representations are matched using a modified cosine similarity measure. Experimental results show that the proposed method outperforms the state­of­the­art methods. (Lead: Bhanu)




2) In the second approach the matching is conducted in a reference space where the descriptor for a person is translated from the original color or texture descriptors tosimilarity measures between this person and the exemplars in the reference set. A subspace is first learned in which the correlationsof the reference data from different cameras are maximized using regularized canonical correlation analysis (RCCA). For reidentification, the gallery data and the probe data are projected onto this RCCA subspace and the reference descriptors (RDs) of the gallery and probe are generated by computing the similarity between them and the reference data. The identity of a probe is determined by comparing the RD of the probe and the RDs of the gallery. A reranking step is added to further improve theresults using a saliency­based matching scheme. Experiments on publicly available datasets show that the proposed method outperforms most of the state­of­the­art approaches. (Lead: Bhanu)




Robust Canonical Correlation Analysis (ROCCA): Given a small training set direct application of canonical correlation analysis (CCA) may lead to poor performance due to the inaccuracy in estimating the data covariance matrices. We have proposed ROCCA with shrinkage estimation and smoothing technique is simple to implement and can robustly estimate the data covariance matrices with limited training samples. Experimental results on two publicly available datasets show that the proposed ROCCA outperforms regularized CCA (RCCA), and achieves state­of­the­art matching results for person re­identification as compared to the most recent methods. (Lead: Bhanu)




B) Multi­target Tracking and Data Association in a Sensor Network Considering Scene Constratins: Existing data association techniques mostly focus on matching pairs of data­point sets and then repeating this process along space­time to achieve long term correspondences. However, in many problems such as person re­identification, a set of data­points may be observed at multiple spatio­temporal locations and/or by multiple agents in a network and simply combining the local pairwise association results between sets of data­points often leads to inconsistencies over the global space­time horizons. In this work, we propose a novel Network Consistent Data Association (NCDA) framework formulated as an optimization problem that not only maintains consistency in association results across the network, but also improves the pairwise data association accuracies. The proposed NCDA can be solved as a binary integer program leading to a globally optimal solution and is capable of handling the challenging data­association scenario where the number of data­points varies across different sets of instances in the network. We also present an online implementation of NCDA method that can dynamically associate new observations to already observed data­points in an iterative fashion, while maintaining network consistency. (Lead: Roy Chowdhury)




1) Online Adaptation of Learned Models in Sensor Networks: Most traditional multi-camera person re­identification systems rely on learning a static model on tediously labeled training data. Such a framework may not be suitable for situations when new data arrives continuously or all the data is not available for labeling beforehand. Inspired by the ‘value of information’ active learning framework, we propose a continuous learning person re­identification system with a human in the loop. The human in the loop not only provides labels to the incoming images but also improves the learned model by providing most appropriate attribute based explanations. These attribute based explanations are used to learn attribute predictors along the way. The overall effect of such a stratgey is that starting with a few annotated images, the system begins to improve via a symbiotic relationship between the man and the machine. The machine assists the human to speed the annotation and the human assists the machine to update itself with more annotation so that more and more distinct persons are re­identified as more and more images come in. (Lead: Roy Chowdhury)




C) Planning and Discovering Assemblies of Moving Objects Using Partial Information: Consider objects moving in a road network (e.g., vehicles, people), who may be free to choose routes, yet be required to arrive at certain locations at certain times. Planning for such assemblies is hard when the network or the number of objects is large. Conversely, discovering actual or potential assemblies of such objects is important in many security, applications. This can be hard when object arrival observations are sparse due to inadequate sensor coverage or object countermeasures. We have proposed the novel class of assembly queries to model these scenarios, and present a unified scheme that addresses both of these complementary challenges. Given a set of objects and arrival constraints, we show how to first obtain the set of all possible locations visited by each moving object, and then determine all possible assemblies, including the participants, locations, and durations. We present a formal model for various tracking strategies and several algorithms for using these strategies. We achieve excellent performance using Contraction Hierarchies. (Lead: Ravishankar)




1) Inferential Time­Decaying Bloom FIlters: Time­Decaying Bloom Filters are probabilistic structures to answer queries on recently inserted items. As new items are inserted, memory of older items decays. Wrong query responses penalize the application using the filter. Current filters may only be tuned to static penalties, and ignore Bayesian priors and much information latent in the filter. We address these issues by introducing inferential filters, which integrate Bayesian priors and information latent in filters to make optimal query­specific decisions. Our methods are general, but here we focus on inferential time­decaying filters, and show how to support novel query types and sliding window queries with varying error penalties. (Lead: Ravishankar)

D) Multi­sensor Data Fusion Methodology for Fusing Lidar and Vision on a Mobile Robot for Tracking People: Both data association and tracking are considered uncertain. Empirically evaluated the work in a lengthy set of tests outdoors, with multiple pedestrians, occlusions, and environmental conditions. An MS thesis was ompleted on the subject. Also, developed an information based exploration planner with information goals, and information constraints. Formal proofs were defined, and empirical evaluations were conducted. A receding horizon approach to real time work was developed using a novel tail cost approximation. A conference paper was presented on this subject, and a journal paper is nearing submital. (Lead: Mark Campbell, Cornell )

• Key Outcomes or other Achievements:
• Publications in major conferences and journals.
• Software has been released along with the papers.
• New dataset for multi­camera person tracking and reidentification.
• New dataset for object recognition in unconstrained environments.
• Algorithms and software for lidar/vision data fusion for tracking people from moving robots, and an information optimal exploration planner.


• What opportunities for training and professional development has the project provided?
• Six graduate students have been partially supported by this grant. Four of the students have completed their PhDs and 2 more are PhD candidates.
• At Cornell university three graduate students have been partially supported by this grant, with one finishing with an MS and the other two still in the program.


• How have the results been disseminated to communities of interest?
• Publications, software and datasets.


• What do you plan to do during the next reporting period to accomplish the goals?
• We will focus on online and semi­supervised learning approaches for adaptation of scene understanding models in unconstrained environments.
• Attribute-based learning methods with uncertainty associated with objects.
• Study the distributed version of the information exploration planner.

Journals or Juried Conference Papers

• A. Chakraborty, A. Das, A. Roy­Chowdhury (2016). Network Consistent Data Association. IEEE Trans. on Pattern Analysis and Machine Intelligence. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• A. Das, A. Chakraborty, A. Roy­Chowdhury (2014). Consistent Re­identification In A Camera Network. European Conf. on Computer Vision. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• A. Das, N. Martinel, C. Micheloni, A. Roy­Chowdhury (2015). Re­Identification in the Function Space of Feature Warps. IEEE Trans. on Pattern Analysis and Machine Intelligence. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• A. T. Kamal, J. H. Bappy, J. A. Farrell, A. Roy­Chowdhur (2016). Distributed Multi­Target Tracking and Data Association in Vision Networks. IEEE Trans. on Pattern Analysis and Machine Intelligence. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• Alex Ivanov, Mark Campbell (2016). An Efficient Robotic Exploration Planner with Probabilistic Guarantees. IEEE International Conference on Robotics and Automation. . Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• L. An, M. Kafai, S. Yang and B. Bhanu (2016). Person re­identification with reference descriptor. IEEE Transactions on Circuits and Systems for Video Technology. 26 (4), 776. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• L. An, S. Yang and B. Bhanu (2015). Person re­identification by robust canonical correlation analysis. IEEE Signal Processing Letters. 22 (8), 1103. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• L. An, X. Chen, S. Yang and B. Bhanu, (2016). Sparse representation matching for person re­identification. Information Sciences. 335­356 74. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• Mark Campbell, Nisar Ahmed (2016). Distributed Data Fusion: Neighbors, Rumors, and the Art of Collective Knowledge. IEEE Control Systems. 36 (4), 83. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes ; DOI: 10.1109/MCS.2016.2558444
• R. Panda, A. Das, A. Roy­Chowdhury (2016). Video Summarization in a Multi­View Camera Network. International Conf. on Pattern Recognition. . Status = ACCEPTED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• Reaz Uddin, Vassilis Tsotras, Chinya Ravishankar (). Indexing of Approximate Spatiotemporal Trajectories Using Hilbert Curves. Juried conference paper. . Status = OTHER; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes
• X. Chen, L. An, and B. Bhanu (2015). Multi­target tracking in non­overlapping cameras using a reference set. IEEE Sensors Journal. 15 (5), 2692. Status = PUBLISHED; Acknowledgment of Federal Support = Yes ; Peer Reviewed = Yes

• A. Das, A. Roy­Chowdhury (2015). Active Image Pair Selection for Continuous Person Re­identification. IEEE Intl. Conf. on Image Processing. . Status = PUBLISHED; Acknowledgement of Federal Support = Yes
• R. Uddin, V. Tsotras and C. Ravishankar (2016). Assembly Queries: Planning and Discovering Assemblies of Moving Objects Using Partial Information. PVLDB Journal. . Status = UNDER_REVIEW; Acknowledgement of Federal Support = Yes

• Various Softwares Under Development.

• Lucas de la Garza. Multi­Object, Multi­Sensor Detection and Tracking of Pedestrians on a Mobile Robot. (2016). Cornell University. Acknowledgement of Federal Support = Yes

• http://www.vislab.ucr.edu/PROJECTS/CPS/CPS.php