Giving Text More Imagination Space for Image-text Matching

摘要

Image-text matching is a hot topic in multi-modal analysis. The existing image-text matching algorithms focus on bridging the heterogeneity gap and mapping the feature into a common space under strong alignment assumption. However, these methods have unsatisfactory performance under the weak alignment scenario, which assumes that the text contains more abstract information, and the number of entities in the text is always fewer than objects in image. This is the first time, from our knowledge, to solve the image-text matching problem from the perspective of information difference with weak alignment. In order to both narrow the cross-modal heterogeneity gap and balance the information discrepancy, we proposed an imagination network to enrich the text modality based on pre-trained framework, which is helpful for image-text matching. The imagination network utilizes reinforcement learning to enhance the semantic information for text modality, and an action refinement strategy is designed to constrain the freedom and divergence of imagination. The experiment results show the superiority and generality of the proposed framework based on two pre-trained models, CLIP and BLIP on two most frequently-used datasets MSCOCO and Flickr30K.

主要结果

@inproceedings{dong2023imagination,
  author    = {Dong, Xinfeng and Han, Longfei and Zhang, Dingwen and Liu, Li and Han, Junwei and Zhang, Huaxiang},
  title     = {Giving Text More Imagination Space for Image-text Matching},
  booktitle = {Proceedings of the 31st ACM International Conference on Multimedia (MM)},
  year      = {2023},
  doi       = {10.1145/3581783.3612103}
}