In this study, a semi-automatic target initialization algorithm is introduced based on a recently proposed visual saliency approach. First, a center-surround difference based initial window selection is utilized around the input point coordinate provided by the user, in order to select the window which is most likely to contain the actual target and background satisfying piecewise connectivity. Then, a recently proposed visual saliency algorithm is exploited in order to detect the bounding box encapsulating the most salient part of the object. The experiments support that the saliency based tracking window initialization is capable of handling marking errors, i.e. erroneous user inputs, and boosts the performance of several tracking algorithms in terms of the number of frames in which successful tracking is achieved, when compared with several fixed window size initializations.