Detection of driver aggressiveness is a significant method in terms of safe driving. Every year, a vast number of traffic accidents occur due to aggressive driving behaviour. These traffic accidents cause fatalities, severe disorders and huge economical cost. Therefore, detection of driver aggressiveness could help in reducing the number of traffic accidents by warning related authorities to take necessary precautions. In this work, a novel method is introduced in order to detect driver aggressiveness on vehicle. The proposed method is based on the fusion of visual and sensor features to characterize related driving session and to decide whether the session involves aggressive driving behaviour. Visual information is used to detect road lines and vehicle images, whereas sensor information provides data such as vehicle speed and engine speed. Both information is used to obtain feature vectors which represent a driving session. These feature vectors are obtained by modelling time series data by Gaussian distributions. An SVM classifier is utilized to classify the feature vectors in order for aggressiveness decision. The proposed system is tested by real traffic data, and it achieved an aggressive driving detection rate of 93.1%.