Recently, an emerging class of methods, namely tracking by detection, achieved quite promising results on challenging tracking data sets. These techniques train a classifier in an online manner to separate the object from its background. These methods only take input location of the object and a random feature pool; then, a classifier bootstraps itself by using the current tracker state and extracted positive and negative samples. Following these approaches, a novel tracking system is proposed. A feature selection method is introduced to increase the discriminative power of the classifier. During tracking, a Hidden Markov Model (HMM) is utilized to filter the features that improve the performance. Moreover, a state of the proposed HMM is allocated to handle occlusions. The proposed tracker is tested on publicly available challenging video sequences and superior tracking results are achieved in real-time.