Neural networks (NNs) are widely used to create parametric models of S-parameters for various components in electronic systems. The focus of deriving these models has so far been numerical error reduction between the NN-generated S-parameters and the data source. However, this is not sufficient when creating such NNs since it does not guarantee predicted S-parameters to be physically consistent, i.e., passive and causal, which restricts their use cases. This article, therefore, proposes a causality enforcement layer (CEL) and passivity enforcement layer (PEL) that can be used in NNs, which ensures that NN-predicted S-parameters are of a passive and causal system. To achieve this, we utilize Kramers-Kronig relations and singular value properties of S-parameters during the training stage with the purpose of learning a physically consistent representation. This enables end-to-end training where no postprocessing is required to ensure physical consistency. We demonstrate the effectiveness of the presented approach for three different design applications, where the goal is to predict S-parameters from dc to 100 GHz. The results show that when NNs are trained using CEL and PEL, the predicted S-parameters are characterized as 100.0% causal and passive while having the same level of accuracy as NNs that solely focus on error minimization.