Recurrent Neural Word Segmentation with Tag Inference

Qianrong Zhou, Long Ma, Zhenyu Zheng, Yue Wang, Xiaojie Wang
DOI: https://doi.org/10.1007/978-3-319-50496-4_66
2016-01-01
Abstract:In this paper, we present a Long Short-Term Memory (LSTM) based model for the task of Chinese Weibo word segmentation. The model adopts a LSTM layer to capture long-range dependencies in sentence and learn the underlying patterns. In order to infer the optimal tag path, we introduce a transition score matrix for jumping between tags of successive characters. Integrated with some unsupervised features, the performance of the model is further improved. Finally, our model achieves a weighted F1-score of 0.8044 on close track, 0.8298 on the semi-open track.
What problem does this paper attempt to address?