Creating terminological correspondence recognition tests with GPT-4: a case study in English-to-Turkish translations in the engineering domain

As Machine Translation (MT) technologies become more advanced, the translation errors they generate are often increasingly subtle. When MT is integrated in ‘Human-in-the-Loop’ (HITL) translation workflows for specialized domains, successful Post-Editing (PE) hinges on the humans involved having in-depth subject competence, as knowledge of the specific terminology and conventions are essential to produce accurate translations. One way of assessing an individual’s expertise is through manual translation tests, a method traditionally used by Language Service Providers (LSPs) and translator educators alike. While manual evaluation can provide the most comprehensive overview of a translator’s abilities, they have the disadvantage of being time-consuming and costly, especially when large numbers of subjects and language pairs are involved. In this work, we report on the experience of creating automated tests with GPT-4 for assessing the ability to recognize domain-specific specialized terminology correspondence in the translation of English-to-Turkish engineering texts in HITL translation workflows. While there may be a level of usefulness in the resulting tests, they are not fit for direct implementation without further refinement.