Hydrological models have been widely used for water resources management. Successful application of hydrological models depends on careful calibration and uncertainty analysis. Spatial unit of water balance calculations may differ widely in different models from grids to hydrological response units (HRU). The Soil and Water Assessment Tool (SWAT) software uses HRU as the spatial unit. SWAT simulates hydrological processes at sub-basin level by deriving HRUs by thresholding areas of soil type, land use, and slope combinations. This may ignore some important areas, which may have great impact on hydrological processes in the watershed. In this study, a hierarchical HRU approach was developed in order to increase model performance and reduce computational complexity simultaneously. For hierarchical optimization, HRUs are first divided into two-HRU types and are optimized with respect to some relevant influence parameters. Then, each HRU is further divided into two. Each child HRU inherits the optimum parameter values of the parent HRU as its initial value. This approach decreases the total calibration time while obtaining a better result. The performance of the hierarchical methodology is demonstrated on two basins, namely Sarisu-Eylikler and Namazgah Dam Lake Basins in Turkey. In Sarisu-Eylikler, we obtained good results by a combination of curve number (CN2), soil hydraulic conductivity, and slope for generating HRUs, while in Namazgah use of only CN2 gave better results.