In this work, we focus on the problem of infrared (IR) object classification by dividing the object appearance space hierarchically with a binary decision tree structure. Specially designed features of the object appearances make the binary decisions at each node of the tree. These features are extracted using a fully connected deep neural network. At each node of the tree, we train individual deep CNNs such that each node specializes in its corresponding subspace. The proposed method is tested in our generated dataset, which consists of IR targets collected from different video records obtained from different IR sensors (both midwave and longwave) and taken from real world field. The generated dataset consists of four different class labels as shiplboat, tank, plane and helicopter containing a total of 16K samples. Using the proposed tree-based classifier, we observe a favourable performance increase in our dataset against a single deep CNN classifier.