Product embedding serves as a cornerstone for a wide range of applications in
eCommerce. The product embedding learned from multiple modalities shows
significant improvement over that from a single modality, since different
modalities provide complementary information. However, some modalities are more
informatively dominant than others. How to teach a model to learn embedding
from different modalities without neglecting information from the less dominant
modality is challenging. We present an image-text embedding model (ITEm), an
unsupervised learning method that is designed to better attend to image and
text modalities. We extend BERT by (1) learning an embedding from text and
image without knowing the regions of interest; (2) training a global
representation to predict masked words and to construct masked image patches
without their individual representations. We evaluate the pre-trained ITEm on
two tasks: the search for extremely similar products and the prediction of
product categories, showing substantial gains compared to strong baseline
models.

通过扩展 BERT 模型，提出了一种用于学习来自不同形态的嵌入的无监督学习方法，通过构建全局表示来预测被标记的词和构造标记的图像块，该方法在搜索极其相似产品和预测产品类别的任务中相较于强基线模型获得了显著的增益。