Face recognition is a key task of computer vision research that has been employed in various security and surveillance applications. Recently, the importance of this task has risen with the improvements in the quality of sensors of cameras, as well as with the increasing coverage of camera networks setup everywhere in the cities. Moreover, biometry-based technologies have been developed for the last three decades and have been available on many devices such as the mobile phones. The goal is to identify people based on specific physiological landmarks. Faces are one of the most commonly utilized landmarks, due to the fact that facial recognition systems do not require any voluntary actions such as placing hands or fingers on a sensor, unlike the other bio-metric methods. In order to inhibit cyber-crimes and identity theft, the development of effective methods is necessary. In this paper, we address the face recognition problem by matching any face image visually with previously captured ones. Firstly, considering the challenges due to optical artifacts and environmental factors such as illumination changes and low resolution, in this paper, we deal with these problems by using convolutional neural networks (CNN) with state-of-the-art architecture, ResNet. Secondly, we make use of a large amount of data consisting of face images and train these networks with the help of our proposed loss function. Application of CNNs was proven to be effective in visual recognition compared to the traditional methods based on hand-crafted features. In this work, we further improve the performance by introducing a novel training policy, which utilizes quadruplet pairs. In order to ameliorate the learning process, we exploit several methods for generating quadruplet pairs from the dataset and define a new loss function corresponding to the generation policy. With the help of the proposed selection methods, we obtain improvement in classification accuracy, recall, and normalized mutual information. Finally, we report results for the end-to-end system for face recognition, performing both detection and classification.