Abstract: This paper proposed a hierarchical semantic model which can destruct complex image semantic, and presented the method to destruct complex semantic and to construct model. Then the paper proposed the concept of semantic binding hierarchical visual vocabulary which is based on the hierarchical semantic model and also presented the method and steps to build the visual vocabulary, and the resolution to some detail problems about vocabulary as well. The paper applied the visual vocabulary into the content and semantic based image retrieval and proposed the steps to construct the image retrieving system and to retrieving images. The proposed model and method can support both the image retrieval based on image content and the image retrieval based on image semantic. The experiments prove the better performance of the model and method compared with the traditional SIFT(Scale Invariant Feature Transform) feature based image retrieval.