Aggressive driving behavior is among the important causes of traffic accidents. Hence, detection of driver aggressiveness has an importance in terms of decreasing the number of traffic accidents. Collected driving data while the vehicle is in traffic can be used to make inferences about the aggressiveness of the driver. In this study, a multimodal method is proposed in order to detect driver aggressiveness. The proposed method is based on utilizing the visual data taken from the on vehicle camera and sensor data taken from the controller area network bus (CAN-bus) in order to decide whether the driving session involves aggressive driving behavior. Lane following pattern and vehicle following distance information is obtained from the data collected by camera while vehicle speed and engine speed information is obtained from CAN-bus. These information is fused to conceive feature vectors that represent the driving session and aggressiveness decision is made according to the classification of these feature vectors.