BERT for Harmonic Time Series Modeling: A Multi-Stage Fine-Tuning Approach
33rd Conference on Signal Processing and Communications Applications-SIU-Annual, İstanbul, Türkiye, 25 - 28 Haziran 2025, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/siu66497.2025.11112046
- Basıldığı Şehir: İstanbul
- Basıldığı Ülke: Türkiye
- Orta Doğu Teknik Üniversitesi Adresli: Evet
Özet
This study demonstrates the potential of a BERT-based transformer model in harmonic signal modeling using synthetic sinusoidal data. The model was trained through a three-stage fine-tuning process (reconstruction, linear analysis, full tuning) with a masked language modeling approach. In the first stage, the model successfully filled in missing data and learned the basic features, while in subsequent stages, its ability to capture temporal dependencies and sequential patterns was enhanced. Additionally, patch, time, and station embedding strategies effectively represented the harmonic structure of the signal. The results indicate that pre-training with synthetic data can overcome the limited access to real-world data, allowing transformer models to be efficiently used in these types of problems.