We develop techniques of analyzing the unitarity of general Born-Infeld gravity actions in D-dimensional spacetimes. The determinantal form of the action allows us to find a compact expression quadratic in the metric fluctuations around constant curvature backgrounds. This is highly nontrivial since for the Born-Infeld actions, in principle, infinitely many terms in the curvature expansion should contribute to the quadratic action in the metric fluctuations around constant curvature backgrounds, which would render the unitarity analysis intractable. Moreover in even dimensions, unitarity of the theory depends only on finite number of terms built from the powers of the curvature tensor. We apply our techniques to some four-dimensional examples.