This paper proposes a novel system for fast and accurate content based retrieval of hyperspectral images. The proposed system aims at retrieving hyperspectral images that have both similar spectral characteristics associated with specific materials and fractional abundances to the query image. It consists of two modules. The first module characterizes the query and the target hyperspectral images in the archive by two descriptors: 1) a binary spectral descriptor representing spectral characteristics of distinct materials 2) an abundance descriptor that contains the normalized cumulative fractional abundance information of the corresponding materials. Both descriptors are obtained by a novel bag of endmembers based strategy. The second module aims at retrieving hyperspectral images from the archive that are most similar to query image based on a hierarchical strategy which evaluates the spectral and abundance descriptors similarity. Experiments carried out on a benchmark archive of hyperspectral images demonstrated the effectiveness of the proposed system in terms of retrieval accuracy and time.