Using ZipF frequencies as a representativeness measure in statistical active learning of natural language


Thesis Type: Postgraduate

Institution Of The Thesis: Orta Doğu Teknik Üniversitesi, Graduate School of Informatics, Cognitive Science, Turkey

Approval Date: 2008

Student: ONUR ÇOBANOĞLU

Supervisor: HÜSEYİN CEM BOZŞAHİN

Abstract:

Active learning has proven to be a successful strategy in quick development of corpora to be used in statistical induction of natural language. A vast majority of studies in this field has concentrated on finding and testing various informativeness measures for samples; however, representativeness measures for samples have not been thoroughly studied. In this thesis, we introduce a novel representativeness measure which is, being based on Zipf's law, model-independent and validated both theoretically and empirically. Experiments conducted on WSJ corpus with a wide-coverage parser show that our representativeness measure leads to better performance than previously introduced representativeness measures when used with most of the known informativeness measures.