Segment-based classification is one of the popular approaches for object detection, where the performance of the classification task is sensitive to the accuracy of the output of the initial segmentation. Majority of the object detection systems directly use one of the generic segmentation algorithms, such as mean shift or k-means. However, depending on the problem domain, the properties of the regions such as size, color, texture, and shape, which are suitable for classification, may vary. Besides, fine tuning the segmentation parameters for a set of regions may not provide a globally acceptable solution in remote sensing domain, since the characteristic properties of a class in different regions may change due to the cultural and environmental factors. In this study, we propose a domain-specific segmentation method for building detection, which integrates information related to the building detection problem into the detection system during the segmentation step. Buildings in a remotely sensed image are distinguished from the highly cluttered background, mostly, by their rectangular shapes, roofing material and associated shadows. The proposed method fuses the information extracted from a set of unsupervised segmentation outputs together with this a priori information about the building object, called domain-specific information (DSI), during the segmentation process. Finally, the segmentation output is provided to a two-layer decision fusion algorithm for building detection. The advantage of domain-specific segmentation over the state-of-the-art methods is observed both quantitatively by measuring the segmentation and detection performances and qualitatively by visual inspection.