TAUS is delighted to present the latest DeMT™ Evaluation Report, a follow-up to our June 2022 publication. In today’s rapidly evolving landscape of online machine translation (MT) services, we revisit the state of customizable MT to provide you with a comprehensive update.

Over the past year, we’ve witnessed a considerable shift in the translation industry with the meteoric rise of large language models (LLMs). Through cutting-edge research and experimentation, these developments have ushered in a wave of transformative potential, revolutionising the way we approach translation. Is there still a need to build custom MT engines, one might ask.

Our report underscores the pivotal role of domain-specific machine translation, recognizing that different industries demand distinct tones and vocabularies. Whether it’s medical, e-commerce, or finance, tailoring translations to suit the nuances of each domain is paramount.

Data collection is at the heart of DeMT™ offering, encompassing public repositories, human data creation, crawling, and synthetic data generation. Quality now trumps quantity as MT engines grow more responsive to training, placing greater significance on precise data curation.

The DeMT™ Evaluation Report includes major MT engines Microsoft Custom Translator and Amazon Active Custom Translation, showcasing improvements in customization. Striking the right balance between domain relevance, training data quality, and quantity remains challenging, requiring ongoing trial and error efforts.

While BLEU scores have been a common metric for customization improvements, the report includes chrF and COMET scores to address BLEU’s limitations. The combination of these metrics provide a more comprehensive view of MT performance.

The evaluation included 30 language pair and domain combinations. The findings reveal that compared to last year’s results, customization contributes to substantial improvements in MT performance, offering more consistent results across various domain and language combinations. While the impact of customization remains somewhat unpredictable, this report underscores its growing maturity and its vital role in effective content translation strategies.

TAUS continues to push the boundaries of MT customization, and the DeMT™ Evaluation Report is your compass for navigating the dynamic world of machine translation.