Center for Research in Intelligent Systems
Room 216, Winston Chung Hall, University of California at Riverside,
Riverside, CA 92521,
Tel. 951-827-3954, Fax. 951-827-2425
Xiu Zhang, Graduate Student (research assistant)
Linan Feng, Graduate Student
- Major Goals of the Project:
Motivated by the fact that multiple concepts that frequently cooccur across images form patterns which could provide contextual cues for individual concept inference, the objectives of the proposed EAGER project are:
- (a) Develop a social network inspired formal framework for finding hierarchical cooccurrence correlation among concepts, and use these patterns of cooccurrence as contextual cues to improve the detection of individual concepts in multimedia databases.
- (b) Develop algorithms to select visuallyconsistentsemantic concepts.
- (c) Develop an image content descriptor called concept signature that can record both the semantic concept and the corresponding confidence value inferred from lowlevel image features.
- (d) Evaluate the effectiveness of the proposed approach in two application domains: automatic image annotation and conceptbased image/video retrieval. The validation of the proposed techniques will be carried out by performing experimentson multiple databases using a variety of quantitative measures.
- Accomplishments under these goals:
- Pi Bir Bhanu worked with his students Linan Feng and Xiu Zhang to perform the proposed research, carry out the experiments and publish the research work. Linan Feng and Bir Bhanu completed and revised a journal paper. Xiu Zhang worked on developing networkbased hierarchical cooccurrence algorithms and exploiting correlation structure for available large image datasets for reidentification.
- Specific Objectives:
- 1) Discover and represent the cooccurrence patterns as hierarchical communities by graph modularity maximization in a network with nodes and edges representing concepts and cooccurrence relationships separately.
2) Generate intermediate image descriptors by exploiting concept cooccurrence patterns in the prelabeled training set that renders it possible to depict complex scene images semantically.
3) Evaluate the effectiveness for automated image annotation and image retrieval
- Significant Results:
We carried out experiments for automatic image annotation and semantic image retrieval on several challenging datasets. We use three datasets: 10,000 image and 2500 concepts from LabelMe dataset, 12,000 images and 5800 concepts from SUN09 dataset and 2682 images and 520 concepts from OSR dataset. We use a variety of features (color, histogram of oriented gradients, etc. We evaluate the results for automated image annotation using various measures, including F1 measure and precision measures and for retrieval using mean average precision. The key results are:
- Co-occurrence pattern detection results - Our combined co-occurrence measure of normalized google distance, normalized tag distance, and automated local analysis is more effective than each of the individual measures in co-occurrence network construction as well as co-occurrence pattern detection. The combined measure gives the best performance in modularity measure.
- Automated image Annotation: To analyze the scalability of our approach, we compare the results on the three datasets with increased complexity (OSR < SUN09 < LabelMe) evaluated by the total number of concepts in the datasets and the number of concepts per image. Our results show that generally when the images are complex the performance of the approaches drop. In particular, we observe that our approach achieves better maximum performance gain when the images have higher complexities. For example, LabelMe usually has more than 10 concepts in an image, the maximum performance gain reaches 20.59 percent when the training set contains 80 percent of the images. SUN09 contains on average 5-10 concepts per image, the maximum performance gain is between 11:29 and 14.00 percent. OSR has the least number of concepts in an image, and the maximum gain is the lowest as well which is approximately 10.00 percent only. This indicates that our approach is well suited for understanding images with complex scenes.
- Image Retrieval: The proposed hierarchical concept co-occurrence patterns can boost the individual concept inference. In particular, we can observe that when using only a small fraction of the dataset for training, our method can still achieve comparatively good performance. Further, we observe that the returned images are more semantically related to the scene concept reﬂected in the query images rather than just visually related.
These detailed quantified experimental results (in the attached paper) demonstrate the following:
- (a) The importance of the hierarchy of co-occurrence patterns and its representation as a network structure, and (b) The effectiveness of the approach for building individual concept inference models and the utilization of co-occurrence patterns for reﬁnement of concept signature as a way to encode both visual and semantic information.
- Key Outcomes or other Achievements:
- Developed algorithms to represent the cooccurrence patterns as hierarchical communitiees by graph modularity maximization in a network with nodes and edges representing concepts and cooccurrence relationships separately.
- Developed algorithms for a random walk process that works on the inferred concept probabilities with the discovered cooccurrence patterns to acquire the refined concept signature representation.
- What opportunities for training and professional development has the project provided?
- Project provided opportunity for research on large image databases, machine learning and data mining and the development of algorithms/tools. It provided many sitautions for the improvement and refinemement of oral/written communication skills.
- How have the results been disseminated to communities of interest?
- Publication in the top journal and workshop/conference.
- What do you plan to do during the next reporting period to accomplish the goals?
- Develop and refine the approach and its experimental validation on reidentification datasets.
Journals or Juried Conference Papers
L. Feng and B. Bhanu, “Semantic concept co-occurrence patterns for image annotation and retrieval,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, No. 4, April 2016.
Other Conference Papers and Presentations
- L. Feng and B. Bhanu (2015). Cooccurrenece Patterns for Imgae Annotation and Retrieval. 3rd Workshop on Webscale Vision and Social Media (VSM), International Conference on Computer Vision (ICCV). Santiago, Chile. Status = PUBLISHED; Acknowledgement of Federal Support = Yes
This material is based upon work supported by the National Science Foundation Project ID No. IIS-1552454. Any opinions,
findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.