Abstract

The morphological structure of Semitic languages like Arabic and Hebrew is based on non-concatenative roots and templates. This complex word structure used by humans is obscured to neural models that use traditional tokenization algorithms. In this work, I present and evaluate Semitic Root Encoding (SRE), a tokenization method that represents Semitic words with distinct root and template stem tokens, and apply it to neural machine translation (NMT). I evaluate whether tokenization with the SRE method leads to increases in translation quality, whether it allows NMT models to hypothesize unseen word forms needed at inference, and how impervious resulting NMT models are to generating dubious word stems.

Degree

College and Department

Computational, Mathematical, and Physical Sciences; Computer Science

Rights

https://lib.byu.edu/about/copyright/