This talk was delivered during the 9th Brazilian Conference on Intelligent Systems (BRACIS) in 2020, and focuses on my research about word segmentation. This research has culminated in the development of a hashtag segmentation library called hashformers, which is publicly available at https://github.com/ruanchaves/hashformers.

Our research explores the impact of word segmentation on enhancing natural language processing performance across various domains, including social media sentiment analysis, source code summarization, and neural machine translation. We demonstrate that our proposed architecture outperforms previous approaches to word segmentation in Western languages, delivering superior cross-domain generalization and competitive performance.

Paper

The relevant paper for this talk was published as Domain adaptation of transformers for english word segmentation.