Introduction: Localization of premature ventricular contraction (PVC) origin to guide the radiofrequency ablation (RFA) procedure is one of the prominent clinical goals of non-invasive electrocardiographic imaging. However, the results reported in the literature vary significantly depending on the source model and the level of complexity in the forward model. This study aims to compare the paced and spontaneous PVC localization performances of dipole-based and potential-based source models and corresponding inverse methods using the same clinical data and to evaluate the effects of torso inhomogeneities on these performances. Methods: The publicly available EP solution data from the EDGAR data repository (BSPs from a maximum of 240 electrodes) with known pacing locations and the Bratislava data (BSPs in 128 leads) with spontaneous PVCs from patients who underwent successful RFA procedures were used. Homogeneous and inhomogeneous torso models and corresponding forward problem solutions were used to relate sources on the closed epicardial and epicardial–endocardial surfaces. The localization error (LE) between the true and estimated pacing site/PVC origin was evaluated. Results: For paced data, the median LE values were 25.2 and 13.9 mm for the dipole-based and potential-based models, respectively. These median LE values were higher for the spontaneous PVC data: 30.2–33.0 mm for the dipole-based model and 28.9–39.2 mm for the potential-based model. The assumption of inhomogeneities in the torso model did not change the dipole-based solutions much, but using an inhomogeneous model improved the potential-based solutions on the epicardial–endocardial ventricular surface. Conclusion: For the specific task of localization of pacing site/PVC origin, the dipole-based source model is more stable and robust than the potential-based source model. The torso inhomogeneities affect the performances of PVC origin localization in each source model differently. Hence, care must be taken in generating patient-specific geometric and forward models depending on the source model representation used in electrocardiographic imaging (ECGI).