Small screen and wearable devices play a key role in most of our daily tasks and activities. However, depending on the context, users can easily experience situationally induced impairments and disabilities (SIIDs). Previous studies have defined SIIDs as a new type of impairment in which an able-bodied user's behaviour is impaired by the context including the characteristics of a device and the environment. This article systematically reviews the empirical studies on the effect of context on SIIDs. In particular, this review aims to answer the following two research questions: Which contextual factors have been examined in the literature that can cause SIIDs and how different contextual factors affect small screen and wearable device users' performance. This article systematically reviews 187 publications under a framework that has five factors for context analysis: physical, temporal, social, task, and technical contexts. This review shows that a significant amount of empirical studies have been conducted focusing on some factors such as mobility but there still are some factors such as social factors that need to be further considered for SIIDs. Finally, some factors have shown to have significant impact on users' performance such as multitasking but not all factors has been empirically demonstrated to have an effect on users' performance.