Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 94 THE ACCURACY OF CHATGPT IN TRANSLATING COLLOQUIAL TERMS IN DAWAN LANGUAGE Iis Aprianti1, Yoni Rahayu2, Soraya Grabiella Dinamika3, Muhammad Muslim Nasution4 1Universitas Timor, Kefamenanu, Indonesia 2,3,4Universitas Medan Area, Medan, Indonesia Iisaprianti@unimor.ac.id ABSTRACT ChatGPT, a multilingual translation application has made intercultural conversation simpler including local language. However, ChatGPT's translation accuracy should be reviewed due to limits in recognizing domain-specific terms and cultural context. The aim of this study is to find out the translation accuracy of ChatGPT in translating colloquial terms in Dawan language. This study used qualitative method. The participants of this study were from the sixth semester English department students of University of Timor. The data source of this study was the translation result in Dawan into Indonesian language. While, the data of this study are ChatGPT’s translation result obtained from students’ word list from Dawan into Indonesian language. The result of the analysis showed showed that ChatGPT performs relatively well in translating basic vocabulary and common expressions, with 60% of the translations deemed accurate. However, the tool demonstrated significant limitations in handling culturally embedded and context-dependent terms. Keywords: Chat GPT, Dawan Language, Language Translation, Translation Accuracy, INTRODUCTION In recent years, technology and translation go hand in hand. Translation has been revolutionized by the rise of artificial intelligence (AI). The advancement of AI models in machine translation has expedited translation processes and aided millions of individuals in comprehending texts in a variety of languages. Among these AI models, ChatGPT has not only drawn considerable interest for its ability to generate human-like text and perform translations but has also emerged as a potential solution for language translation. ChatGPT is an artificial intelligence-based natural language model trained on extensive datasets and applied across various tasks, such as serving as a virtual assistant to help users work more efficiently, including in text translation (Kasneci et al., 2023; Mendoza et al., 2022). It can assist in generating content for various types of documents and provide feedback on grammar, style, and textual coherence (Sutanto et al., 2024). Interestingly, ChatGPT, as a machine translation tool, offers more than conventional translation; by enhancing prompts, it can be tailored to yield outputs aligned with professional translation standards (Aeni et al., 2024). http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 95 According to the author's review, multiple studies have explored ChatGPT and translation, such as analyses of its translation accuracy in scientific texts (Aeni et al., 2024), morphological and syntactic translation errors (Al-Salman & Haider, 2024), the use of ChatGPT in translation learning (Nurfaiza, 2024), and comparisons of ChatGPT's performance with other translation services (Hidayati & Nihayah, 2024). However, these studies have not yet addressed the acceptability and readability of ChatGPT translations in local languages. While ChatGPT performs effectively for instructional materials and achieves high translation quality in standardized contexts, it faces greater challenges in translating complex texts—particularly those in underrepresented local languages such as Dawan. The Dawan language, primarily spoken in East Nusa Tenggara, Indonesia, is used by over one million people, including the Dawan tribe and other ethnic groups in the Timor Island region. As with many regional languages, Dawan struggles with digital representation and automated translation due to its unique grammar, sentence structure, and culturally embedded expressions, which often lack direct equivalents in other languages. These linguistic and cultural intricacies present significant hurdles for ChatGPT. Moreover, the limited availability of training data for low-resource languages like Dawan further contributes to potential translation inaccuracies. Although ChatGPT demonstrates reliable performance on simpler texts, its accuracy declines when dealing with more complex content that demands cultural and contextual understanding. Given this phenomenon and the growing trend in AI-based translation, there is a compelling need for research to explore the accuracy of ChatGPT in translating colloquial terms in the Dawan language, particularly in terms of meaning conveyance, contextual appropriateness, and cultural sensitivity. Translation is more than a linguistic act; it is also a cultural and cognitive process that involves interpreting meanings between languages and contexts (Munday, 2016). In this framework, colloquial language refers to informal, everyday expressions deeply embedded in the speaker's cultural and regional context (Nida, 2015; House, 2016). Unlike standard language, colloquial expressions often carry idiomatic or metaphorical meanings, making them particularly challenging to translate accurately. Catford (2017) emphasized that the equivalence in translation, especially for colloquial or idiomatic expressions, is seldom direct. Instead, it requires contextual awareness, cultural familiarity, and semantic flexibility. Venuti (2017) argued that the translator must balance between domestication—making terms familiar to the target audience—and foreignization—preserving the unique features of the source language. The growing use of machine translation (MT), including tools like Google Translate and DeepL, has shown promising results for formal and standardized texts (Wu et al., 2016). However, colloquial expressions remain a major barrier for most MT systems due to their cultural specificity and non-literal meanings (Toral & Way, 2018). Garciarena et al. (2020) noted that MT tools typically rely on parallel corpora, which are scarce for low- resource languages and informal registers. Recent developments in neural machine translation (NMT) have improved MT accuracy. Bahdanau et al. (2015) introduced attention-based models that allow machines http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 96 to weigh parts of the input sequence differently, enhancing the interpretation of idiomatic language. Yet, as Koehn and Knowles (2017) observed, even advanced NMT systems struggle to retain the nuance of colloquial terms without domain-specific training. OpenAI's ChatGPT, based on the GPT-3 and GPT-4 architecture, represents a leap in natural language understanding and generation (Brown et al., 2020; OpenAI, 2023). Unlike earlier MT systems, ChatGPT can respond contextually, learn from prompts, and simulate human-like conversations. This capacity makes it a candidate for more nuanced translations, including colloquial speech (Kasneci et al., 2023). However, despite its fluency, several studies have noted limitations in ChatGPT's translation performance. Tang et al. (2023) pointed out that while GPT models can generate plausible translations, they sometimes prioritize grammatical fluency over semantic accuracy. Mendoza et al. (2022) further found that GPT-based models occasionally "hallucinate" translations that are contextually inappropriate, especially for languages with limited digital presence. Low-resource languages, such as Dawan, often lack large annotated corpora, making them difficult for both traditional MT and AI models to process accurately (Anastasopoulos & Neubig, 2019). According to Faten (2021), the cultural depth and oral nature of many indigenous languages contribute to translation complexity. Sutanto et al. (2024) also emphasized that the translation of local languages requires cultural intelligence and familiarity with linguistic variations at the regional level. Moreover, Ebrahimi et al. (2018) argued that without sufficient exposure, AI models tend to normalize low-resource languages into dominant linguistic patterns, leading to semantic loss or misinterpretation. For colloquial terms specifically, lack of training data limits ChatGPT's ability to understand the pragmatic and cultural implications embedded in those expressions. Dawan, spoken primarily in Timor Island, is rich in metaphor, idiomatic phrases, and cultural references (Schapper, 2017). Colloquialisms in Dawan often rely on metaphors drawn from nature, kinship, or traditional customs. As such, their meanings cannot be easily deduced without cultural immersion (Lole, 2020). For instance, a simple expression like “Naek ba tasa” (literally “go to the plate”) may imply readiness to face a challenge— something a machine might misinterpret literally. Studies on indigenous language preservation have highlighted the urgency of digitizing and documenting such expressions before they are lost (Grenoble & Whaley, 2016; Lewis & Simons, 2021). The integration of these languages into AI systems like ChatGPT could play a role in digital revitalization, but only if done carefully and ethically. Accuracy in translation is often measured through three main criteria: adequacy, fluency, and fidelity to the source meaning (Popović, 2015). Tools like BLEU (Bilingual Evaluation Understudy) and human evaluators are used to assess these aspects. However, as Graham et al. (2017) argue, standard metrics do not always account for cultural and contextual nuances—especially in colloquial and low-resource language settings. Manual evaluations, especially by native speakers and bilingual experts, remain the gold standard in assessing the translation of culturally embedded expressions. For http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 97 languages like Dawan, community participation is essential to ensure that translations are not only accurate but respectful and contextually grounded (Bird, 2020). In summary, the existing literature underscores several critical points. First, colloquial expressions represent one of the most difficult areas in translation due to their cultural and idiomatic nature. Second, while ChatGPT has advanced capabilities in natural language processing, its effectiveness for low-resource and colloquial language translation remains uncertain without tailored training and contextual knowledge. Third, Dawan language poses specific challenges due to its rich oral traditions and metaphorical expressions. Therefore, investigating ChatGPT's ability to translate Dawan colloquialisms not only contributes to the field of AI and linguistics but also supports broader efforts in indigenous language preservation. METHODS This study conducted Qualitative method. The results of this study are not formed in general conclusions or generalization but in a description. Thus, they are focused and detailed. The subject was the sixth semester students of academic year 2025 in English department of University of Timor. The data source of this study was the translation result in Dawan into Indonesian language. While, the data of this study are ChatGPT’s translation result obtained from students’ word list from Dawan into Indonesian language. Then, the translation data will be analyzed using content analysis with Nababan's translation quality assessment model which consists of accuracy, acceptability and readability. In the analysis stage, the translation data will be grouped and coded according to the theme, then analyzed and given an interpretation and ended with a conclusion. RESULTS This study analyzed the translation of 50 colloquial terms in the Dawan language into Indonesian using ChatGPT. The evaluation was guided by Nababan’s translation quality model, which assesses translations across three core dimensions: accuracy, acceptability, and readability (Nababan & Nuraeni, 2012). These dimensions collectively determine how well a translation conveys meaning, conforms to target language norms, and remains easily understandable to readers. Translation Accuracy The accuracy scores were categorized into three levels—accurate, less accurate, and inaccurate—based on the degree of semantic equivalence between the source (Dawan) and target (Indonesian) terms: • 30 terms (60%) were rated as accurate. • 12 terms (24%) were rated as less accurate. • 8 terms (16%) were rated as inaccurate. http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 98 Accurate Translations The highest accuracy was observed in terms where there is a direct semantic and syntactic equivalent in Indonesian. For example: • Ha’u → Saya (I): This translation is both contextually appropriate and linguistically accurate, as it reflects the standard first-person pronoun in both languages. • Mone → Anak laki-laki (boy): This translation precisely captures the intended meaning. Such terms are lexically simple, and their cultural connotations are minimal or universal enough to translate easily. Less Accurate Translations Translations were considered less accurate when the core meaning was preserved, but additional or inappropriate semantic elements were introduced. A good example is: Inan → Ibu kandung (biological mother): While technically correct, this translation is over-specified. In everyday use, inan generally means ibu (mother) in Dawan culture. The addition of kandung could mislead readers to think the context distinguishes biological from non-biological motherhood, which was not intended. Such cases reflect the model’s literal bias—a tendency to generate formal definitions or high-frequency mappings, rather than selecting the most pragmatically appropriate equivalents. Inaccurate Translations Some terms were significantly mistranslated, resulting in a distortion of cultural meaning. Notably: Tobe → Kepala desa (village head): This translation misses the ceremonial and tribal leadership function embedded in the term tobe, which refers to a traditional community leader in Dawan society. The bureaucratic notion of kepala desa fails to capture these indigenous values. These inaccuracies underscore ChatGPT’s inability to recognize sociocultural contexts, especially where terms involve customary law, traditional hierarchies, or oral metaphors. Acceptability and Readability Translation outputs were also evaluated for linguistic acceptability and ease of reading in Indonesian. The results showed: • 80% of the translations were acceptable and highly readable. • The remaining 20% included translations that, while grammatically sound, were stylistically awkward or unnatural in colloquial Indonesian. http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 99 ChatGPT’s tendency to use standard and neutral syntax contributed positively to readability. For example: Ha’u besi → Saya pergi (I go): The structure is clean, straightforward, and conforms to normative Indonesian grammar. However, problems arose when dealing with compound or idiomatic expressions. Consider: Ana-ana mone → Anak-anak laki-laki: While grammatically correct, this translation lacks pragmatic nuance. In Dawan, the phrase may imply “sons” or “young boys” depending on the social context. More culturally fitting translations might include para anak laki-laki or anak-anak lelaki. Such examples show that literal translation can hinder naturalness and cultural clarity, especially in socially embedded expressions. Interpretation and Implications These findings highlight both the capabilities and limitations of ChatGPT as a tool for translating low-resource and culturally rich languages. While the model performs well on standard vocabulary and syntax, it struggles with idiomatic, metaphorical, and cultural expressions—a trend that aligns with prior research (Al-Salman & Haider, 2024; Aeni et al., 2024). Several factors likely contribute to these limitations: 1. Data scarcity: Dawan is a low-resource language with limited presence in publicly available corpora. As noted by Anastasopoulos & Neubig (2019), MT systems perform poorly when training data is scarce or domain-specific. 2. Lack of domain adaptation: ChatGPT is not fine-tuned on indigenous languages or culturally grounded idioms. This often leads to overgeneralization or approximation using dominant language structures. 3. Contextual misalignment: The AI's understanding is based on probabilistic pattern recognition. Without cultural intelligence or regional linguistic insights, it cannot fully infer implicit meanings behind colloquial terms (Ebrahimi et al., 2018). This situation suggests the need for a human-in-the-loop approach, especially in tasks requiring cultural nuance. Native speakers or linguists must be involved in post-editing or fine-tuning AI output, particularly when translating for educational, ceremonial, or historical purposes. Furthermore, these results reinforce the urgency of digitally documenting indigenous languages (Grenoble & Whaley, 2016) and exploring ethical AI integration that does not erase the cultural uniqueness of minority communities. If well-trained, AI tools like ChatGPT could play a pivotal role in language preservation and revitalization. CONCLUSION This study examined the accuracy of ChatGPT in translating colloquial terms from the Dawan language into Indonesian. The analysis showed that ChatGPT performs relatively well in translating basic vocabulary and common expressions, with 60% of the translations http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 100 deemed accurate. However, the tool demonstrated significant limitations in handling culturally embedded and context-dependent terms. The main challenges stemmed from ChatGPT’s lack of sensitivity to cultural nuances and its reliance on general linguistic patterns. Terms such as tobe, which hold deep traditional meaning in the Dawan community, were often mistranslated due to the absence of direct equivalents in Indonesian and insufficient contextual understanding by the AI model. Despite these challenges, ChatGPT offers notable potential as a preliminary translation tool, especially for lesser-known languages. Its ability to provide readable and grammatically sound outputs makes it useful for first drafts or informal translation tasks. Nevertheless, for translations involving indigenous or culturally rich languages, human intervention remains essential to ensure the preservation of meaning and cultural integrity. In the future, integrating more localized data into AI training sets, enhancing prompt engineering strategies, and adopting hybrid human-AI translation workflows may help improve the accuracy and cultural sensitivity of machine translation tools like ChatGPT. In sum, this study demonstrates that: a. ChatGPT is reasonably effective at translating basic Dawan vocabulary into Indonesian with high readability and structural fluency. b. However, accuracy suffers when the input includes culturally loaded or context- sensitive terms. c. Colloquial and traditional expressions are often mistranslated or diluted, leading to loss of cultural integrity. Thus, ChatGPT can serve as a preliminary translation aid, but must be supplemented by human expertise, particularly for underrepresented and culturally rich languages like Dawan. Future work should explore the development of localized language models trained on regional corpora, and the inclusion of indigenous communities in AI tool design and evaluation. ACKNOWLEDGMENT The researcher would like to express their sincere gratitude to the sixth-semester students of the English Department at the University of Timor for their valuable contributions in providing the Dawan word lists used in this research. Special thanks are also extended to the lecturers and faculty members who supported the data collection process. This research would not have been possible without their active participation and collaboration. REFERENCES Aeni, R., Putera, L. J., & Melani, B. Z. (2024). The accuracy of ChatGPT in translating linguistics text in scientific journals. Didaktik: Jurnal Ilmiah PGSD STKIP Subang, 10(1), 59–68. http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 101 Al-Salman, S., & Haider, A. S. (2024). Assessing the accuracy of MT and AI tools in translating humanities or social sciences Arabic research titles into English: Evidence from Google Translate, Gemini, and ChatGPT. International Journal of Data and Network Science. https://api.semanticscholar.org/CorpusID:271237784 Anastasopoulos, A., & Neubig, G. (2019). Pushing the limits of low-resource morphological inflection. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 980–986. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations (ICLR). Bird, S. (2020). Decolonising speech and language technology. Proceedings of the 28th International Conference on Computational Linguistics (COLING 2020), 3504–3519. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. Ebrahimi, J., Rao, A., Lowd, D., & Dou, D. (2018). HotFlip: White-box adversarial examples for text classification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 31–36. Faten, S. (2021). Challenges in translating indigenous languages: A cultural perspective. Journal of Translation Studies, 18(2), 122–134. Garciarena, U., Gonzalez-Agirre, A., & Rigau, G. (2020). Evaluating multiword expression identification in machine translation. Computational Linguistics, 46(1), 1–36. Graham, Y., Baldwin, T., Moffat, A., & Zobel, J. (2017). Can machine translation systems be evaluated without human references? Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3769–3779. Grenoble, L. A., & Whaley, L. J. (2016). Saving languages: An introduction to language revitalization. Cambridge University Press. House, J. (2016). Translation as communication across languages and cultures. Routledge. Kasneci, E., Sessler, K., Yu, Y., Bannert, M., Dementieva, D., Fischer, F., & Schmidt, A. (2023). ChatGPT for good? Opportunities and challenges of large language models in education. Learning and Individual Differences, 103, 102274. Koehn, P., & Knowles, R. (2017). Six challenges for neural machine translation. Proceedings of the First Workshop on Neural Machine Translation, 28–39. Lewis, M. P., & Simons, G. F. (2021). Assessing endangerment: Expanding Fishman’s GIDS. Revue Roumaine de Linguistique, 66(1), 55–70. Lole, A. J. (2020). Cultural metaphors in Dawan language. Timor Linguistics Journal, 4(1), 23–35. http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 https://api.semanticscholar.org/CorpusID:271237784 Copyright © The Author(s) Vol.6, No. 2, April 2025 e-ISSN: 2723-4126 p-ISSN: 2776-8880 102 Mendoza, M., et al. (2022). Enzymatic transfer of acetate on histones from lysine reservoir sites to lysine activating sites. Science Advances, 8(3), eabj5688. Munday, J. (2016). Introducing translation studies: Theories and applications (4th ed.). Routledge. Nida, E. A. (2015). Language, culture, and translating. Shanghai Foreign Language Education Press. Nurfaiza. (2024). Pengaruh penggunaan ChatGPT dalam pembelajaran terjemah Indonesia- Arab. At-Turost: Journal of Islamic Studies, 99–113. OpenAI. (2023). GPT-4 technical report. https://openai.com/research/gpt-4 Popović, M. (2015). chrF: character n-gram F-score for automatic MT evaluation. Proceedings of the Tenth Workshop on Statistical Machine Translation, 392–395. Schapper, A. (2017). A grammar of Bunaq: A Papuan language of East Timor. De Gruyter Mouton. Sutanto, V. M., De Giacomo, G. G., Nakazawa, T., & Yamada, M. (2024). ChatGPT as a translation engine: A case study on Japanese-English. In Proceedings of the 30th Annual Conference of the Association for Natural Language Processing (pp. 2096– 1201). Tang, R., et al. (2023). Evaluating ChatGPT’s translation capabilities: Benchmarks and breakdowns. AI & Society. https://doi.org/10.1007/s00146-023-01596-4 Toral, A., & Way, A. (2018). What level of quality can neural machine translation attain on literary text? Translation Quality Assessment, 78–91. Venuti, L. (2017). The translator’s invisibility: A history of translation (3rd ed.). Routledge. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., ... & Dean, J. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. http://u.lipi.go.id/1593190689 http://u.lipi.go.id/1593190689 https://openai.com/research/gpt-4