Vol. 6, No.2, July 2025 | 56 P-ISSN: 2715-2448 | E-ISSN: 2715-7199 Vol.6 No.2 July 2025 Buana Information Technology and Computer Sciences (BIT and CS) Development of a Deep Learning-Based Text-To-Speech System for the Malang Walikan Language Using the Pre-Trained SpeechT5 and Hifi-GAN Models Aina Avrilia Imani1, Aviv Yuniar Rahman2, Firman Nurdiyansyah3 1 2 3 Department of Informatic Engineering, Universitas Widya Gama, Malang, Indonesia E-mail: avriliaaina@gmail.com1, aviv@widyagama.ac.id2, firmannurdiyansyah@widyagama.ac.id3 Received: 2025/06/07 | Revised: 2025/06/27 | Accepted: 2025/07/27 Abstract The Walikan language of Malang is a form of local cultural heritage that needs to be preserved in the digital era. This study aims to develop and evaluate a deep learning-based Text-to-Speech (TTS) system capable of generating speech in the Walikan language of Malang using pre-trained SpeechT5 and HiFi- GAN models without fine-tuning. In this system, SpeechT5 is used to convert text into mel-spectrograms, while HiFi-GAN acts as a vocoder to generate audio signals from the mel-spectrograms. The dataset used consists of 1,000 sentences in the Walikan language of Malang. The system evaluation was carried out using objective metrics of Word Error Rate (WER) and Character Error Rate (CER), by comparing the results of synthetic audio transcriptions against two types of reference audio, namely the original voices of female speakers and male speakers, using the Automatic Speech Recognition (ASR) system. The female voice was recorded with controlled articulation, while the male voice used natural intonation in everyday conversation. The results show that synthetic audio has the highest error rate with a WER of 0.9786 and a CER of 0.9024. Meanwhile, female audio has a WER of 0.5471 and a CER of 0.1822, while male audio shows a WER of 0.6311 and a CER of 0.2541. These findings indicate that the TTS model without fine-tuning is not yet capable of producing synthetic voices that can be recognized accurately by the ASR system, especially for regional languages that are not included in the initial training data. Therefore, the fine-tuning process and the preparation of a more representative dataset are important so that the TTS system can support the preservation of the Walikan Malang language more effectively in the digital era. Keywords: HiFi-GAN, Malang Walikan Language, SpeechT5, Text-to-Speech, and Word Error Rate. I. Introduction The Walikan language of Malang is a local cultural heritage that deserves preservation, especially in today's digital era, where regional languages are increasingly threatened with extinction. Deep learning-based Text-to-Speech (TTS) technology has developed rapidly and is one solution for language preservation by converting text into natural speech. However, most existing TTS models focus on resource-intensive languages, making them less than optimal for low-resource regional languages like Walikan. Research on TTS for regional languages is still very limited, especially in cases where there is little or no training data (zero-shot). Walikan also has unique characteristics not found in other languages, making it crucial to test how pre-trained models like SpeechT5 and HiFi-GAN perform under these conditions without fine-tuning. Several previous studies have used SpeechT5 and HiFi-GAN models for languages with more resources and have shown quite good results [1], [2]. However, the use of pre-trained models directly (zero-shot) in low-resource regional languages, particularly Malang Walikan, is still rarely researched. Vol. 6, No.2, July 2025 | 57 Other studies have compared objective evaluation methods such as Word Error Rate (WER) with subjective Mean Opinion Score (MOS) assessments to assess synthetic speech quality [3]. This study aims to develop and evaluate a deep learning-based TTS system using pre-trained models SpeechT5 and HiFi-GAN for Malang Walikan without fine-tuning. The evaluation was conducted using objective metrics WER and Character Error Rate (CER) by comparing the synthetic results to real human speech to assess the model's performance on low-resource languages. In the development of deep learning-based Text-to-Speech systems, various architectures have been developed to produce voices that increasingly resemble human speech. One recent model that has demonstrated superior performance is SpeechT5 [4], a transformer-based pre-trained model with an encoder-decoder approach designed for various speech processing tasks, including speech synthesis. This model is capable of understanding text structure and converting it into a mel-spectrogram representation, taking into account intonation and rhythm. To generate the final speech signal, HiFi- GAN [5] was used, a generative adversarial network (GAN)-based vocoder designed to convert mel- spectrograms into high-quality, natural audio. The combination of these two models was found to be effective in generating synthetic speech that approximates human voice quality. This research provides an important contribution to understanding how large pre-trained models perform on regional languages not yet represented in the training data, and provides a basis for further development through fine-tuning and the collection of a more comprehensive dataset, to support the digital preservation of the Walikan Malang language. II. Methods The steps in this research are shown in (Figure 1). . Figure 1. Steps a research. (Source: Personal Preparation) The dataset used in this study consists of 1,000 sentences in the Walikan language of Malang. Each sentence was recorded by two different speakers, one male and one female, resulting in a total of 2,000 audio files. This dataset was used to study the linguistic patterns and acoustic characteristics of Walikan and to evaluate the system's performance on voice variations based on speaker gender. The collected data was stored in spreadsheet format (Excel) for ease of management and processing. Before use, the data underwent pre-processing to remove irrelevant symbols or emojis and normalize the text. An example of the data used is shown in Table 1. EVALUATION Vol. 6, No.2, July 2025 | 58 Table 1. Dataset Language Walikan Language Walikan Language Indonesian etas maya sing cedek gang iku asaib sam rasane sate atam yang dekat gang itu biasa mas rasanya yang ayahab kuwi gluduk arudam yang bahaya itu petir madura tambah nade ae nawak kotis iki tambah gila aja teman satu ini wah umak lihai juga berbahasa arudam wah kamu lihai juga berbahasa madura rintep pol kakakku pintar banget kakakku agomes warga licek sukses halokes e semoga warga kecil sukses sekolah nya halokes sek jum sekolah dulu jum nde wendit akeh sedeb sam di wendit banyak monyet mas likis e adikku mari dicokot kucing kaki nya adikku habis digigit kucing bangga nggae soak arema bangga pakai kaos arema 1. Pre-Trained SpeechT5 Model SpeechT5 is a pre-trained model based on a transformer encoder-decoder architecture used to convert text into a mel-spectrogram, a visual representation of audio frequencies over time. This model was implemented without fine-tuning, using the default configuration from the Hugging Face library. The SpeechT5 model uses a Transformer-based encoder-decoder architecture, with six additional pre- net and post-net modules to process input and output in the form of text and speech. The encoder consists of two parts: the Speech Encoder Pre-net to convert the raw speech signal into speech features, and the Text Encoder Pre-net to convert the text into a textual representation. The outputs of these two parts are combined into a unified representation that is used by the decoder. Figure 2. SpeechT5 Architecture The decoder also has two paths: the Speech Decoder, which generates a mel-spectrogram from text input, and the Text Decoder, which generates text from voice input. The Pre-net and Post-net components in each section enhance the quality of the representation and output. The Speech Encoder pre-net utilizes features from wav2vec 2.0, while the decoder utilizes speaker embeddings (x-vectors) to support multi-speaker synthesis. The initial pre-training process is multi-task and cross-modal, including bidirectional masking prediction, sequence-to-sequence generation, and text pre-training using an infilling strategy. The encoder output representation is discretized using vector quantization, allowing the model to align the voice and text modalities. Vol. 6, No.2, July 2025 | 59 The implementation of the SpeechT5 model as the core of the Text-to-Speech system for the Malang Walikan language is carried out using the following steps: a. Data Preparation and Pre-trained Model The pre-trained SpeechT5 model is used in the Walikan language sentence dataset without fine- tuning. The audio output is evaluated using ASR and Word Error Rate (WER). b. Text Preprocessing 1) Normalization: Standardize the text (lowercase, remove irrelevant symbols/punctuation). 2) Tokenization: Break the text into tokens (words/subwords). 3) Conversion to Numeric ID: Map tokens to unique numbers based on the vocabulary. 4) Embedding: Convert the ID tokens into a numeric representation vector that the model understands. c. Encoding with a Transformer Encoder The embedding vector is processed by a Transformer encoder with self-attention to produce a contextual representation of the text (hidden states). d. Mel-spectrogram Prediction by the Decoder The Transformer decoder converts the contextual representation into a mel-spectrogram through self-attention and cross-attention with the encoder output. The mel-spectrogram depicts sound energy at various frequencies over time. e. Mel-spectrogram Conversion to Audio The decoded mel-spectrogram is converted to audio using HiFi-GAN. 2. HiFi-GAN HiFi-GAN is used as a vocoder to convert the mel-spectrogram output from SpeechT5 into a high- quality audio signal. This model plays a role in the final stage of the synthesis process, producing natural-sounding voices that approximate human voice quality. The output, in the form of a mel- spectrogram from SpeechT5, serves as input to HiFi-GAN, which then produces the final audio in WAV format. HiFi-GAN [6] is a Generative Adversarial Network (GAN)-based vocoder designed to convert mel-spectrograms into high-quality raw audio signals. Its architecture consists of a generator and two types of discriminators: the Multi-Scale Discriminator (MSD) and the Multi-Period Discriminator (MPD) [7]. Training is conducted in an adversarial manner, where the generator and discriminator compete to produce increasingly natural sounds. Figure 3. HiFi-GAN architecture [5] The HiFi-GAN generator is a fully convolutional network with an upsampling process based on transposed convolution [8]. To capture complex temporal patterns, a Multi-Receptive Field Fusion (MRF) module is used, which combines multiple residual blocks with different kernels and dilations. Vol. 6, No.2, July 2025 | 60 MPD focuses on periodic patterns in audio signals with sub-discriminators that process the input based on specific time periods (e.g., 2, 3, 5, 7, 11). Meanwhile, MSD captures patterns at multiple temporal scales, processing audio at different resolutions (raw, average pooled ×2, and ×4) to identify global characteristics of the signal. 3. System Output The system generates synthetic audio in Malang Walikan, reflecting the phonetic and linguistic characteristics of the input text. The resulting audio can be listened to as a spoken representation of the sentences in the dataset. 4. Evaluation The evaluation was conducted using the Word Error Rate (WER) and Character Error Rate (CER) metrics, which measure the error rate between the automatic transcription of synthetic audio and the original text. The lower the WER and CER values, the higher the system's accuracy and naturalness in producing speech that matches the text. Evaluation of Text-to-Speech (TTS) results is performed by measuring the accuracy of the synthesized audio transcription against the reference text using two main metrics: the Word Error Rate (WER) and the Character Error Rate (CER). a. Word Error Rate The Word Error Rate (WER) is the primary metric for evaluating the accuracy of a Text-to-Speech (TTS) system by comparing the audio transcription output from the model with the original text. The WER is calculated based on the number of word substitutions (S), deletions (D), and insertions (I) divided by the total number of words (N) in the original text, expressed as a percentage: WER = 𝑆+𝐷+𝐼 𝑁 𝑥 100 If WER = 0%, the system produces perfect output that is identical to the original text. The higher the WER value, the greater the errors in the speech synthesis results [9], [10]. b. Character Error Rate Character Error Rate (CER) is an evaluation metric similar to WER, but calculated at the character level. CER is more sensitive to small errors, especially in short sentences or languages with non- standard structures, such as Walikan. The CER formula is similar to WER, namely: CER = 𝑆+𝐷+𝐼 𝑁 𝑥 100 Where S, D, and I are the number of character substitutions, deletions, and insertions, and N is the total number of characters in the original text. CER complements WER by providing a more detailed error analysis at the phonetic level [11] III. Results and Discussions 1. Word Error Rate Synthetic audio quality was evaluated by calculating the Word Error Rate (WER) for nine representative test sentences with varying error levels. Three types of audios were analyzed: the SpeechT5 + HiFi-GAN synthesized audio, a female voice, and a male voice, to determine how well the ASR system recognized each type of voice. (1) (2) Vol. 6, No.2, July 2025 | 61 Table 2. Word Error Rate. Test Sentence WER SpeechT5 + HiFi-GAN WER Voice Female WER Voice Male info lokasi dong nawak 0.5 0.25 0.25 sesuai dengan gambar di bis halokes 0.5 0.17 0.33 wah umak lihai juga berbahasa arudam 0.67 0.5 0.83 rame ilakes area kayutangan di malam minggu 0.71 0.29 0.71 sekali nade tetep nade 0.75 0.75 0.75 agomes lancar rejeki hari ini 0.8 0.6 0.4 nakam lah mbah 1.0 0.33 1.0 umak ngalam 1.0 0.5 1.0 jenenge ae ongis nade yo nade temenan 1.0 0.57 0.71 Table 2 shows examples of WER values for the three audio types. The results indicate that all synthetic audio had WER values above 0.30, indicating a high level of pronunciation errors. Four sentences achieved a maximum WER of 1.0, meaning that all words were not recognized by ASR. The sentences with the lowest WER were "info lokasi dong nawak" and "sesuai dengan gambar di bis halokes," each with a value of 0.5. In contrast, human voices performed better and more consistently, with some sentences achieving WERs as low as 0.25. This indicates that natural audio is more capable of conveying phonetic information accurately. The high error rate in synthetic audio is likely due to the striking phonetic differences and the model's limitations in handling the non-standard Walikan language structure and its inversions. Sentences such as "wah umak lihai juga bahasa arudam" reflect linguistic challenges for the model that has not undergone fine-tuning. As a follow-up to this evaluation, the scatter plot in Figure 4 was used to visualize the distribution of WER values for each test sentence by voice type category. This visualization provides a more comprehensive picture of the distribution of recognition errors at the sentence level. Figure 4. Scatter plot comparing WER values by audio type Figure 4 shows a comparison of the Word Error Rate (WER) values for nine test sentences based on three audio types: the ones synthesized by the SpeechT5 + HiFi-GAN model (red/orange), the Vol. 6, No.2, July 2025 | 62 female voice (blue), and the male voice (green). In general, the synthetic audio produced the highest and most consistent WER values, with all sentences having values above 0.5, even reaching 1.0 in the last three sentences. This indicates that the Text-to-Speech system is not yet capable of producing pronunciations close to human accuracy, especially in the complex context of Walikan. Meanwhile, human audio, particularly female voices, demonstrated better and more stable performance, with WER values ranging from 0.2 to 0.6. Male voices also tended to be more accurate than synthetic audio, but experienced greater fluctuations between sentences. This difference indicates that synthetic audio still faces significant challenges in matching the natural phonetic characteristics of human speakers. Figure 5. Histogram of WER Comparison Figure 5 shows a comparison of the distribution of Word Error Rate (WER) values for three types of audios: female, male, and the SpeechT5 + HiFi-GAN synthesized voices. The histogram shows that the WER values of the synthesized voices are mostly close to 1.0, indicating ASR's failure to recognize the sentence content. In contrast, human voices show a lower and more even distribution of WER values, some even approaching 0.0, indicating near-perfect transcription. This finding confirms that without fine-tuning, the SpeechT5 + HiFi-GAN model is not yet capable of producing pronunciations accurate enough for ASR to recognize. Table 3. Average Word Error Rate Average Word Error Rate (WER) SpeechT5 + HiFi- GAN Real Voice (Female) Real (Male) 0.9786 0.5471 0.6311 Testing 1,000 test sentences showed an average WER of 0.9786 for the synthetic audio from SpeechT5 + HiFi-GAN (Table 3), close to 1.0, indicating that the majority of words were not recognized by ASR. This reflects the low quality of the synthesis, especially in the context of Walikan, which has unusual phonetics. In contrast, human voices performed better, with an average WER of 0.5471 (female) and 0.6311 (male), a difference of ±0.33–0.43 lower than the synthetic results. This confirms that ASR is better able to recognize natural voices. Vol. 6, No.2, July 2025 | 63 In addition to inaccurate pronunciation, the synthetic results also showed anomalies such as random capitalization and phonetic errors due to the model's bias toward English, for example, "sam" being recognized as "some". 2. Character Error Rate Evaluation using the Character Error Rate (CER) provides a more detailed picture of phonetic errors at the character level, especially for minor errors such as letter substitutions or spelling mistakes. Table 4 summarizes the minimum, maximum, median, and average CER values for the three audio source categories tested. Table 4. Character Error Rate Evaluation Voice Category Minimum CER Maximum CER Median CER Mean CER SpeechT5 + HiFi-GAN 0.3243 1.0000 1.0000 0.9024 Voice Female 0.0000 1.6154 0.1461 0.1822 Voice Male 0.0000 1.0000 0.2143 0.2541 Character Error Rate analysis (Table 4) shows that the synthetic audio from the SpeechT5 + HiFi- GAN model performed poorly compared to human audio. The average CER was 0.9024, with a median and maximum of 1.00, indicating that many sentences failed to be recognized by ASR. Even the minimum value of 0.3243 still indicated significant errors at the character level. In contrast, human audio was much more accurate. Female voices had an average CER of 0.1822 and male voices 0.2541, with a minimum value of 0.00 for both, indicating some sentences were perfectly recognized. These results align with previous WER findings, indicating that errors in synthetic audio occur not only at the word level but also at the character level. This reflects the model's limitations in representing Walikan phonology without fine-tuning. Based on the evaluation, the SpeechT5 and HiFi-GAN-based TTS systems performed poorly when used on Walikan without fine-tuning. High WER and CER values for synthetic audio indicate that ASR has difficulty recognizing the model's output, in contrast to human speech, which produces better accuracy. Table 5. Comparison of Regional Language TTS Studies Author Dataset Data Ground Truth Method Evaluation Validate WER CER MOS Agustina, C., 2024. [12] 250 Sentence Language Banjar - VITS - - 3.604 ALHUDA, M.Y., 2025. [13] 500 Sentence Language Palembang - VITS - - 4.58 Our proposed 1000 Sentence Language Walikan Malang Voice Artificial SpeechT5 & HiFi-GAN 0,9786 0,9024 - Voice Female SpeechT5 & HiFi-GAN 0,5471 0,1822 - Voice Male SpeechT5 & HiFi-GAN 0,6311 0,2541 - Vol. 6, No.2, July 2025 | 64 Table 5 compares this research with other TTS studies on regional languages. The studies by Agustina (2024) and Alhuda (2025) used VITS with a smaller dataset and only assessed voice quality through MOS (3.604 and 4.58), without objective metrics like WER or CER. Meanwhile, this study used 1,000 Walikan sentences and objectively evaluated them using WER and CER. The results showed that the synthetic audio had a WER of 0.9786 and a CER of 0.9024, indicating significant errors. However, the female and male voices recorded WERs of 0.5471 and 0.6311, and CERs of 0.1822 and 0.2541, indicating better ASR recognition of human voices. The advantage of this approach lies in the use of more measurable objective metrics and the hybrid SpeechT5 + HiFi-GAN model, which has potential, although it still requires further optimization and training to handle the complexity of regional languages and natural voice variations. IV. Conclusions This study developed a preliminary TTS system for Malang Walikan using SpeechT5 and HiFi- GAN without fine-tuning. Although the synthesis results were not optimal, the system demonstrated basic potential for improvement through further training with local data. Evaluation using ASR-based WER and CER showed that the synthetic audio (SpeechT5 + HiFi-GAN) produced a WER of 0.9786 and a CER of 0.9024, indicating a very high error rate. Audio from a female voice had a WER of 0.5471 and a CER of 0.1822, while audio from a male voice recorded a WER of 0.631 and a CER of 0.2541. These differences indicate that voice characteristics significantly impact system performance. The system was unable to replicate the speaker's voice characteristics, such as intonation and clarity of pronunciation, likely due to the lack of fine-tuning and a lack of adaptation to the unique structure of Walikan. However, the pronunciation of some words was quite accurate, indicating potential for further development.This research is an initial contribution to the development of TTS for undocumented regional languages, and emphasizes the importance of adapting models to local contexts for more accurate and natural results. Vol. 6, No.2, July 2025 | 65 References [1] J. Lehečka, Z. Hanzlíček, J. Matoušek, and D. Tihelka, “Zero-Shot vs. Few-Shot Multi-speaker TTS Using Pre-trained Czech SpeechT5 Model,” pp. 46–57, 2024, doi: 10.1007/978-3-031- 70566-3_5. [2] H. Wang, “Understanding Zero-shot Rare Word Recognition Improvements Through LLM Integration,” 2025, [Online]. Available: http://arxiv.org/abs/2502.16142 [3] A. Kirkland, S. Mehta, H. Lameris, G. E. Henter, E. Szekely, and J. Gustafson, “Stuck in the MOS pit: A critical analysis of MOS test methodology in TTS evaluation,” no. August, pp. 41– 47, 2023, doi: 10.21437/ssw.2023-7. [4] J. Ao et al., “SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing,” Oct. 2021, [Online]. Available: http://arxiv.org/abs/2110.07205 [5] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” Oct. 2020, [Online]. Available: http://arxiv.org/abs/2010.05646 [6] J. Su, Z. Jin, and A. Finkelstein, “HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks,” Jun. 2020, [Online]. Available: http://arxiv.org/abs/2006.05694 [7] Z. Qiu, J. Tang, Y. Zhang, J. Li, and X. Bai, “A Voice Cloning Method Based on the Improved HiFi-GAN Model,” Comput Intell Neurosci, vol. 2022, 2022, doi: 10.1155/2022/6707304. [8] D. Lim, S. Jung, and E. Kim, “JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech,” Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2022-Septe, pp. 21–25, 2022, doi: 10.21437/Interspeech.2022-10294. [9] A. Ali and S. Renals, “Word error rate estimation without asr output: E-WER2,” Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2020-Octob, pp. 616–620, 2020, doi: 10.21437/Interspeech.2020-2357. [10] A. Ali and S. Renals, “Word Error Rate Estimation for Speech Recognition: e-WER,” Jul. 2018. [Online]. Available: https://github.com/qcri/e-wer [11] I. Kottayam and J. James, “Advocating Character Error Rate for Multilingual ASR Evaluation,” 2023. [12] C. Agustina, “Implementasi Teknologi Text To Speech Bahasa Banjar Menggunakan Metode Vits,” (Doctoral dissertation, Universitas Islam Negeri Sultan Syarif Kasim Riau)., 2024. [13] M. Y. ALHUDA, “Text To Speech Bahasa Palembang Menggunakan Metode Vits,” (Doctoral dissertation, UNIVERSITAS ISLAM NEGERI SULTAN SYARIF KASIM RIAU)., 2025.