This article introduces a novel attentive driving framework in intelligent transportation systems (ITS) to investigate the influence of cognitive behavior on distracting driving activities that lead to inattention while driving. Therefore, this work proposes a holistic computational and communication framework that can monitor on-compartment real-time multimodal sensory observation such as physiological, camera, and environmental inputs while capable of distraction detection and emotion recognition for driver's mood stabilization. In particular, this work develops a capsule network for distraction detection, a 1-D convolutional neural network for emotion recognition, an a priori algorithm for sequential context fusion, and a Bayesian network for recommending auditory stimulus content for driver mood stabilization and audio-visual safety messages for road safety. Further, an asynchronous client control scheme has developed to overcome the challenges of multitime scale sensory observations and communicate among the multimodel sensory hubs. Finally, a prototype is developed and tested in a simulation environment. The quantitative analysis results show that the proposed framework can successfully detect around 89% and 87% of distractive activities and the affective state of a driver, respectively. Finally, based on experimental results, the proposed system demonstrates the capability to sustain a driver's attention for approximately 97% of the time, with a confidence level of 95%.