Visualization and Intelligent Systems Laboratory



Contact Information

Winston Chung Hall Room 216
University of California, Riverside
900 University Avenue
Riverside, CA 92521-0425

Tel: (951)-827-3954

Bourns College of Engineering
NSF IGERT on Video Bioinformatics

UCR Collaborators:

Other Collaborators:
Keio University

Other Activities:
IEEE Biometrics Workshop 2014
IEEE Biometrics Workshop 2013
Worshop on DVSN 2009
Multibiometrics Book

Webmaster Contact Information:
Michael Caputo

Last updated: June 15, 2016



Visual Semantic Concepts: Social Networks Based Concept Learning in Images

NSF Project ID: IIS - 1552454

Principal Investigator

Bir Bhanu
Center for Research in Intelligent Systems
Room 216, Winston Chung Hall, University of California at Riverside,
Riverside, CA 92521, Tel. 951-827-3954, Fax. 951-827-2425


Xiu Zhang, Graduate Student (research assistant)
Linan Feng, Graduate Student


  • Major Goals of the Project:
  • Motivated by the fact that multiple concepts that frequently co­occur across images form patterns which could provide contextual cues for individual concept inference, the objectives of the proposed EAGER project are:

    (a) Develop a social network inspired formal framework for finding hierarchical co­occurrence correlation among concepts, and use these patterns of co­occurrence as contextual cues to improve the detection of individual concepts in multimedia databases.

    (b) Develop algorithms to select visually­consistent­semantic concepts.

    (c) Develop an image content descriptor called concept signature that can record both the semantic concept and the corresponding confidence value inferred from low­level image features.

    (d) Evaluate the effectiveness of the proposed approach in two application domains: automatic image annotation and concept­based image/video retrieval. The validation of the proposed techniques will be carried out by performing experimentson multiple databases using a variety of quantitative measures.
  • Accomplishments under these goals:

  • Major Activities:
    Pi Bir Bhanu worked with his students Linan Feng and Xiu Zhang to perform the proposed research, carry out the experiments and publish the research work. Linan Feng and Bir Bhanu completed and revised a journal paper. Xiu Zhang worked on developing network­based hierarchical co­occurrence algorithms and exploiting correlation structure for available large image datasets for re­identification.

  • Specific Objectives:
  • 1) Discover and represent the co­occurrence patterns as hierarchical communities by graph modularity maximization in a network with nodes and edges representing concepts and co­occurrence relationships separately.

    2) Generate intermediate image descriptors by exploiting concept co­occurrence patterns in the pre­labeled training set that renders it possible to depict complex scene images semantically.

    3) Evaluate the effectiveness for automated image annotation and image retrieval
  • Significant Results:
    • We carried out experiments for automatic image annotation and semantic image retrieval on several challenging datasets. We use three datasets: 10,000 image and 2500 concepts from LabelMe dataset, 12,000 images and 5800 concepts from SUN09 dataset and 2682 images and 520 concepts from OSR dataset. We use a variety of features (color, histogram of oriented gradients, etc. We evaluate the results for automated image annotation using various measures, including F1 measure and precision measures and for retrieval using mean average precision. The key results are:
    • Co-occurrence pattern detection results - Our combined co-occurrence measure of normalized google distance, normalized tag distance, and automated local analysis is more effective than each of the individual measures in co-occurrence network construction as well as co-occurrence pattern detection. The combined measure gives the best performance in modularity measure.
    • Automated image Annotation: To analyze the scalability of our approach, we compare the results on the three datasets with increased complexity (OSR < SUN09 < LabelMe) evaluated by the total number of concepts in the datasets and the number of concepts per image. Our results show that generally when the images are complex the performance of the approaches drop. In particular, we observe that our approach achieves better maximum performance gain when the images have higher complexities. For example, LabelMe usually has more than 10 concepts in an image, the maximum performance gain reaches 20.59 percent when the training set contains 80 percent of the images. SUN09 contains on average 5-10 concepts per image, the maximum performance gain is between 11:29 and 14.00 percent. OSR has the least number of concepts in an image, and the maximum gain is the lowest as well which is approximately 10.00 percent only. This indicates that our approach is well suited for understanding images with complex scenes.
    • Image Retrieval: The proposed hierarchical concept co-occurrence patterns can boost the individual concept inference. In particular, we can observe that when using only a small fraction of the dataset for training, our method can still achieve comparatively good performance. Further, we observe that the returned images are more semantically related to the scene concept reflected in the query images rather than just visually related.

    • These detailed quantified experimental results (in the attached paper) demonstrate the following:
    • (a) The importance of the hierarchy of co-occurrence patterns and its representation as a network structure, and (b) The effectiveness of the approach for building individual concept inference models and the utilization of co-occurrence patterns for refinement of concept signature as a way to encode both visual and semantic information.
  • Key Outcomes or other Achievements:
    • Developed algorithms to represent the co­occurrence patterns as hierarchical communitiees by graph modularity maximization in a network with nodes and edges representing concepts and co­occurrence relationships separately.
    • Developed algorithms for a random walk process that works on the inferred concept probabilities with the discovered co­occurrence patterns to acquire the refined concept signature representation.

  • What opportunities for training and professional development has the project provided?
    • Project provided opportunity for research on large image databases, machine learning and data mining and the development of algorithms/tools. It provided many sitautions for the improvement and refinemement of oral/written communication skills.

  • How have the results been disseminated to communities of interest?
    • Publication in the top journal and workshop/conference.

  • What do you plan to do during the next reporting period to accomplish the goals?
    • Develop and refine the approach and its experimental validation on re­identification datasets.

Journals or Juried Conference Papers

  • L. Feng and B. Bhanu, “Semantic concept co-occurrence patterns for image annotation and retrieval,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, No. 4, April 2016. Link

Other Conference Papers and Presentations

  • L. Feng and B. Bhanu (2015). Co­occurrenece Patterns for Imgae Annotation and Retrieval. 3rd Workshop on Web­scale Vision and Social Media (VSM), International Conference on Computer Vision (ICCV). Santiago, Chile. Status = PUBLISHED; Acknowledgement of Federal Support = Yes



This material is based upon work supported by the National Science Foundation Project ID No. IIS-1552454. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.