Monitoring the electrical activity inside the human brain using electrical and magnetic field measurements requires a mathematical head model. Using this model the potential distribution in the head and magnetic fields outside the head are computed for a given source distribution. This is called the forward problem of the electro-magnetic source imaging. Accurate representation of the source distribution requires a realistic geometry and an accurate conductivity model. Deviation from the actual head is one of the reasons for the localization errors. In this study, the mathematical basis for the sensitivity of voltage and magnetic field measurements to perturbations from the actual conductivity model is investigated. Two mathematical expressions are derived relating the changes in the potentials and magnetic fields to conductivity perturbations. These equations show that measurements change due to secondary sources at the perturbation points. A finite element method (FEM) based formulation is developed for computing the sensitivity of measurements to tissue conductivities efficiently. The sensitivity matrices are calculated for both a concentric spheres model of the head and a realistic head model. The rows of the sensitivity matrix show that the sensitivity of a voltage measurement is greater to conductivity perturbations on the brain tissue in the vicinity of the dipole, the skull and the scalp beneath the electrodes. The sensitivity values for perturbations in the skull and brain conductivity are comparable and they are, in general, greater than the sensitivity for the scalp conductivity. The effects of the perturbations on the skull are more pronounced for shallow dipoles, whereas, for deep dipoles, the measurements are more sensitive to the conductivity of the brain tissue near the dipole. The magnetic measurements are found to be more sensitive to perturbations near the dipole location. The sensitivity to perturbations in the brain tissue is much greater when the primary source is tangential and it decreases as the dipole depth increases. The resultant linear system of equations can be used to update the initially assumed conductivity distribution for the head. They may be further exploited to image the conductivity distribution of the head from EEG and/or MEG measurements. This may be a fast and promising new imaging modality.