In this work, a novel method is proposed to optimize the array geometry for DOA estimation. The method is based on minimization of fine error variances with the constraint that the gross error probability is below a certain threshold. For this purpose, a metric function that reflects the gross and fine error characteristics of the array is proposed. Theoretical analyses show that the minimization of this metric function leads to small DOA estimation error variance and small gross error probability. Analyses have been carried out under the assumptions of planar array geometry, isotropic array elements and AWGN. The genetic algorithm is used as an optimization tool and performance simulation is performed by comparing the DOA estimation errors of optimized array to uniform circular arrays (UCAs). Computer simulations support the theoretical analyses and show that the method proposed leads to a significant improvement in array geometry in terms of DOA estimation performance. Finally, the method is applied to an aircraft model and results are presented. (c) 2007 Elsevier B.V. All rights reserved.