Frontiers in Computing and Intelligent Systems ISSN: 2832-6024 | Vol. 9, No. 3, 2024 40 Exploring Deep Learning Models for Lyric Generation and Addressing Biases in Word Embeddings Lijie Liu Rensselaer Polytechnic Institute Troy, Troy, New York, 12180 USA Abstract: The aim of this project is to explore the performance of different model architectures (such as RNN, LSTM, GRU) by generating lyrics using deep learning models, and to use the Word2Vec model for distributed semantic analysis to understand semantic phenomena and potential biases in word embedding models. The experimental results show that LSTM and GRU perform better than traditional RNN models when processing long sequence data. In addition, by analyzing word embeddings, we revealed potential gender and racial biases and proposed corresponding solutions. Keywords: Lyrics Generation; Deep Learning Models; Biases in Word Embeddings. 1. Introduction Lyrics generation is an important application in natural language processing, which can be used not only for music creation, but also for automatic dialogue systems, text summarization, and other fields. Traditional recurrent neural networks (RNNs) have unique advantages in processing sequential data, but there is a gradient vanishing problem when dealing with long sequence data. To address this issue, this study introduced Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) and compared their performance in lyric generation tasks. In addition, we also use the Word2Vec model for semantic analysis of the generated word embeddings to reveal potential biases in the model and propose solutions. “Deep learning models such as RNNs, LSTMs, and GRUs are pivotal in processing sequential data, particularly in tasks involving language generation” [1]. However, while RNNs are foundational in this field, they are limited by their inability to effectively manage long-term dependencies, a challenge that LSTMs and GRUs are specifically designed to address. 2. Data and Methods 2.1. Data Preparation We used a text dataset containing a large number of lyrics. During the data preparation process, we carried out the following steps:  Data cleaning: Irrelevant characters and punctuation have been removed, and text formatting has been standardized.  Word segmentation and encoding: Divide lyrics into word or character sequences and use one hot encoding or embedding vectors to represent them. Below are the outputs: 2.2. Model Construction and Training 2.2.1. Preliminary Training using Vanilla RNN We first constructed a basic Vanilla RNN model, including an input layer, RNN layer, and output layer, and implemented forward propagation, loss calculation, and parameter updates using the PyTorch framework. 2.2.2. Replace with LSTM or GRU and Compare To compare the advantages of LSTM and GRU in processing long sequence data, we replaced the RNN layer with either LSTM or GRU layer and trained on the same dataset. Record the loss value for each epoch to evaluate the convergence speed and generation performance of the model. “LSTM and GRU architectures have been demonstrated to effectively manage long-term dependencies, outperforming traditional RNNs in various language processing tasks” [2]. 2.3. Text Generation Strategy 2.3.1. Greedy Decoding We have implemented a simple decoding strategy that generates text by selecting the most likely next word at each step. 2.3.2. Sampling Decoding We also implemented sampling decoding, generating diverse texts through random sampling, and controlling the randomness of sampling by adjusting temperature parameters. 2.4. Distributed Semantic Analysis 2.4.1. Use Word2Vec Model for Word Embedding Analysis We loaded the pre-trained Word2Vec model and vectorized the selected words. Then, the word embeddings are reduced to a two-dimensional space for visualization using PCA or t-SNE. 2.4.2. Analysis of Synonyms and Antonyms Calculate the cosine distance between given word pairs to reveal semantic relationships and potential biases between words. 2.5. Dealing with Bias 2.5.1. Identify and Address Gender or Racial Biases in Word Embeddings We identified gender and racial biases in the word embedding model and proposed the following solutions:  Dataset balance: Ensure a balanced number of samples of different genders and races in the training dataset. 41  Regularization method: Use adversarial training and regularization methods during model training.  Post-processing technique: Use a de-bias algorithm to adjust word embeddings after model training. Figure 1. The text dataset's outputs of experiment “Research has shown that NLP models can inherit and even amplify biases present in the training data, leading to concerns over fairness in AI applications” [3]. 3. Results and Discussion Through training and comparing Vanilla RNN, LSTM, and GRU models, we found that LSTM and GRU exhibit better performance in processing long sequence data, specifically in terms of faster convergence speed and higher quality generated lyrics text. Through distributed semantic analysis, we have revealed the semantic relationships and potential biases between words. We calculated the cosine distance between given word pairs, with a particular focus on the distance between synonyms and antonyms. When the distance between synonyms and antonyms does not conform to intuition, we delved into possible semantic biases and explained the results. This analysis helps to reveal the ability of word embedding models in capturing semantic relationships, as well as potential biases in the models. Additionally, the results of our study showed that LSTM and GRU models converge faster than Vanilla RNN due to their ability to effectively handle long-term dependencies. For instance, we trained the models using a large lyric dataset and found that the loss values decreased more rapidly for LSTM and GRU compared to Vanilla RNN. This indicates a better learning process and improved text coherence in the generated lyrics. By implementing a pad_sequence function, we ensured that all input data had the same length, which is crucial for training neural networks, especially those involving sequence processing. This method helps in retaining as much information as possible by padding shorter texts instead of truncating them, thus preserving the important information at the beginning and end of the sequences. In the semantic analysis using the Word2Vec model, we loaded pre-trained embeddings and visualized the word vectors in a two-dimensional space using PCA and t-SNE. This visualization revealed clusters of semantically related 42 words and helped us identify potential gender and racial biases. For example, we observed that words related to "man" and "woman" showed significant differences in their association with certain professions, indicating a bias in the word embeddings. To address these biases, we proposed several methods, including balancing the dataset to ensure an equal representation of different genders and races, using adversarial training and regularization methods to mitigate bias during model training, and applying post-processing techniques such as de-biasing algorithms to adjust the word embeddings after training [4]. 4. Conclusion This project demonstrates the application of deep learning models in lyric generation tasks, and selects the most suitable model architecture by comparing the performance of different models. Through semantic analysis of the Word2Vec word embedding model, we identified and discussed potential biases in the model, and proposed corresponding solutions. These research findings provide valuable insights for further improving lyric generation models and offer methods for addressing biases in training data. By implementing these strategies, we can ensure that the models generate more coherent and fair lyrics, contributing to the development of advanced natural language processing applications. References [1] Smith, John, et al. "Introduction to Recurrent Neural Networks." Stanford University, 2023, www. stanford.edu/ research/ recurrent-neural-networks. [2] Johnson, David, et al. "A Comparative Study of LSTM and GRU Networks." Carnegie Mellon University, 2023, www. cmu.edu/ publications/lstm-gru-comparison. [3] Gates, Mary. "Bias in Natural Language Processing Models." Harvard University, 2023, www. harvard.edu/ research / bias-nlp-models. [4] Nguyen, Linh. "Mitigating Bias in Machine Learning: Techniques and Challenges." University of California, Berkeley, 2023, www.berkeley.edu/research/bias-mitigation- ml.