In the present study, the flutter performance of a composite thin-walled beam wing-engine system is optimized by implementing two different control approaches: 1) passive open-loop and 2) active closed-loop control. Sequential quadratic programming and genetic algorithm methods are applied in the optimization process. In the passive control method, variable stiffness is acquired by constructing laminates of thin-walled beam with curvilinear fibers having prescribed paths. The goal is to exploit the desirable fiber paths with improved flutter performance to determine an optimized wing-engine aeroelastic configuration. In the active control strategy, piezo-composite actuators and the linear quadratic Gaussian algorithm are used to improve the flutter characteristics. A novel optimization strategy based on the total energy of the aeroelastic system is introduced and applied in both passive and active control strategies. The minimum total aeroelastic energy is an indication of ideal optimization variables, which leads to optimum flutter performance. The governing equations are formulated based on Librescu's thin-walled beam theory and Hamilton's principle. An unsteady aerodynamic model based on incompressible indicial aerodynamics is applied. The governing equations of motion are solved using a Ritz-based solution methodology. Numerical results demonstrate a 16 and 46% improvement in the flutter speed of the wing-engine system using the proposed passive and active control approaches, respectively. The presented results provide valuable information concerning the design of advanced lightweight and high-aspect-ratio aircraft wings with mounted engines in terms of favorable aeroelastic performance characteristics.