Collocation-based retokenization methods for latent dirichlet allocation topic models

Abstract

Latent Dirichlet Allocation (LDA) discovers hidden themes in documents by using words as input. Past studies show that merging the words into collocation improves topic coherence in English. However, there are still questions about the best merging strategies, especially in the languages without clear word boundaries, such as Thai and Chinese. We compare chi-squared measure, t-statistics, and raw frequency strategies, and show that merging input tokens with appropriate strategies can improve the goodness of fit and topic coherence of the model.