The need for optically multi-functional micro- and nano-structures is growing in various fields. Designing such structures is impeded by the lack of computationally low-cost algorithms. In this study, we present a hybrid design scheme, which relies on a deep learning model and the local search optimization algorithm, to optimize a diffractive optical element that performs spectral splitting and spatial concentration of broadband light for solar cells. Using generated data set during optimization of a diffractive optical element, which is a one-time effort, we design topography of diffractive optical elements by using a deep learning-based inverse design scheme. We show that further iterative optimization of the reconstructed diffractive optical elements increases amount of spatially concentrated and spectrally split light. Our joint design approach both speeds up optimization of diffractive optical elements as well as providing better performance at least 57% excess light concentration with spectral splitting. The algorithm that we develop here will enable advanced and efficient design of multifunctional phase plates in various fields besides the application that we target in solar energy. The algorithm that we develop is openly available to contribute to other applications that rely on phase plates.