Among the human related factors, aggressive driving behavior is one of the major causes of traffic accidents . On the other hand, detection and characterization of driver aggressiveness is a challenging task since there exist different psychological causes behind it. However information about the driver behavior could be extracted from the data that is collected via different sensing devices. This paper presents a method to detect driver aggressiveness using only visual information provided by forward camera. The proposed method is based on detection of the road lines and the vehicles on the road and extracts information related with road lane departure rate, speed of the vehicle and possible forward collision time. Using these extracted features, a classifier is utilized in order to detect if driver shows an aggressive driving behavior. The proposed method is tested by a subjective testing method using 76 different driving sessions and achieved 90.4% success.