Deep learning has proven to be an effective method for classifying images. Over the past years various network topologies have been trained using millions of images only to illustrate that capabilities of deep learning is to be expected only to grow in time. Some of the trained deep networks are made available to public and more such networks are expected to surface in time. When such deep networks are investigated, it is observed that, they are not only trained with high number of images (literally millions), but these images are coming from diverse categories where the images contain many dissimilar distinctive features. Therefore, our intuitions suggest that, if a deep network is trained over a rich variety of categories, the trained network may have the capability to detect sets of features that belong to images that are coming from unknown categories. This paper presents the initial findings of our ongoing research. The results strongly suggest that a trained deep network can potentially be used as a generic feature extractor to cluster images that the network cannot inherently identify.