Information retrieval from high resolution remotely sensed images is a challenging issue due to the inherent complexity and the curse of dimensionality of data under study. This paper presents an approach for building detection in high resolution remotely sensed images incorporating structural information of spatial data into spectral information. The proposed approach moves along eliminating irrelevant areas in a hierarchical manner. As a first step, pan-sharpened image is obtained from multi-spectral and panchromatic bands of Quickbird image. Vegetation and shadow regions are masked out by using Normalized Difference Vegetation Index (NDVI) and ratio of hue to intensity in YIQ model, respectively. Then, panchromatic band is filtered by mean shift filtering for smoothing structures while preserving the discontinuities near boundaries. Next, differential morphological profile (DMP) is calculated for each pixel and a relative measure of structure size is recorded as the first maximum value of DMP which generates a labeled image representing connected components according to sizes of structures. However, there appear some connected components which are irrelevant to buildings in shape. To eliminate those connected components, their skeletons are obtained via thinning to get a relative length measure along with measuring areas of connected components. These measures arc compared to a threshold individually, which provides a cue for a candidate building structure.