© 2019 IEEE.The interactions between the cells, most notably due to their coupled interference and the large number of users, render the optimization of antenna parameters prohibitively complex. To cope with this problem, we propose a novel practical deep learning (DL) based reinforcement learning (RL) algorithm to jointly optimize antenna tilt angle and vertical and horizontal half-power beamwidths of the macrocells in a heterogeneous cellular network (HetNet). In the proposed algorithm, DL is used to extract the features by learning the locations of users, and mean field RL is used to learn the average interference values for different antenna settings. Our results illustrate that the proposed deep RL algorithm can approach the optimum weighted sum rate with hundreds of online trials, as opposed to millions of trials for standard Q-learning, assuming relatively low environmental dynamics. Furthermore, the proposed algorithm is compact and implementable, and empirically appears to provide a performance guarantee regardless of the amount of environmental dynamics.