We perform frequency-and time-resolved all-optical switching of a GaAs-AlAs micropillar cavity using an ultrafast pump-probe setup. The switching is achieved by two-photon excitation of free carriers. We track the cavity resonances in time with a high frequency resolution. The pillar modes exhibit simultaneous frequency shifts, albeit with markedly different maximum switching amplitudes and relaxation dynamics. These differences stem from the non-uniformity of the free carrier density in the micropillar, and are well understood by taking into account the spatial distribution of injected free carriers, their spatial diffusion and surface recombination at micropillar sidewalls. (C) 2014 AIP Publishing LLC.