Mycotoxins are toxic secondary metabolites produced by fungi. They have been demonstrated to cause various health problems in humans, including immunosuppression and cancer. A class of mycotoxins, aflatoxins, has been studied extensively because they have caused many deaths particularly in developing countries. Chili pepper is also prone to aflatoxin contamination during harvesting, production and storage periods. Chemical methods to detect aflatoxins are quite accurate but expensive and destructive in nature. Hyperspectral and multispectral imaging are becoming increasingly important for rapid and nondestructive testing for the presence of such contaminants. We propose a compact machine vision system based on hyperspectral imaging and machine learning for detection of aflatoxin contaminated chili peppers. We used the difference images of consecutive spectral bands along with individual band energies to classify chili peppers into aflatoxin contaminated and uncontaminated classes. Both UV and halogen illumination sources were used in the experiments. The significant bands that provide better discrimination were selected based on their neural network connection weights. Higher classification rates were achieved with fewer numbers of spectral bands. This selection scheme was compared with an information-theoretic approach and it demonstrated robust performance with higher classification accuracy.