







































 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 
76-84 

76 

 

 

 

Article 

Enhanced toxic comment detection model through 

Deep Learning models using Word embeddings and 

transformer architectures 
 S. Sushma1,2*, Sasmita Kumari Nayak1, M. Vamsi Krishna3  

1Department of CSE, Centurion University of Technology and Management, Bhubaneswar, Odisha, India 
2Aditya University, Surampalem, India 
3Department of Computer Applications, Aditya University, Surampalem, India 

A R T I C L E   I N F O 
 

Article history: 
Received 08 April 2025  
Received in revised form 
21 May 2025 
Accepted 31 May 2025 
 
Keywords:  
Toxic comment classification, Word embeddings, 
Ensemble modeling 
 
*Corresponding author 
Email address: 
sushma.cse2@gmail.com 
 
DOI: 10.55670/fpll.futech.4.3.8 

A B S T R A C T 
 

The proliferation of harmful and toxic comments on social media platforms 
necessitates the development of robust methods for automatically detecting 
and classifying such content. This paper investigates the application of natural 
language processing (NLP) and ML techniques for toxic comment classification 
using the Jigsaw Toxic Comment Dataset. Several deep learning models, 
including recurrent neural networks (RNN, LSTM, and GRU), are evaluated in 
combination with feature extraction methods such as TF-IDF, Word2Vec, and 
BERT embeddings. The text data is pre-processed using both Word2Vec and TF-
IDF techniques for feature extraction. Rather than implementing a combined 
ensemble output, the study conducts a comparative evaluation of model-
embedding combinations to determine the most effective pairings. Results 
indicate that integrating BERT with traditional models (RNN+BERT, 
LSTM+BERT, GRU+BERT) leads to significant improvements in classification 
accuracy, precision, recall, and F1-score, demonstrating the effectiveness of 
BERT embeddings in capturing nuanced text features. Among all configurations, 
LSTM combined with Word2Vec and LSTM with BERT yielded the highest 
performance. This comparative approach highlights the potential of combining 
classical recurrent models with transformer-based embeddings as a promising 
direction for detecting toxic comments. The findings of this work provide 
valuable insights into leveraging deep learning techniques for toxic comment 
detection, suggesting future directions for refining such models in real-world 
applications. 

1. Introduction 

There is an increasing amount of harmful and toxic 
comments that may harm users' experience, with the 
exponentially growing user-generated content available on 
social media platforms. Hence, the need for an automated 
system that will detect and filter out such rotten content has 
become crucial to maintaining a positive and healthy 
environment on the internet. This motivates the adoption of 
more advanced ML and NLP techniques, as traditional content 
moderation systems have high false-positive and false-
negative rates and often fail to scale well to larger volumes of 
data. Detecting toxic comments has been addressed with 
different approaches in recent years. Now, there are two 
things we can guess from the name of the model above: one is 
that it could be any classical ML algorithm, and the other is 
that it is used for binary classification. These techniques 
work reasonably well most of the time, but do not necessarily 

capture all the subtleties of language, particularly when 
processing unstructured content like social media comments. 
On this training, it's a great performance in sequence data 
modeling, especially with DL models like RNN, LSTM, and 
GRU. These models work proficiently with text classification. 
To understand contextuality and sentiment in textual data, 
these models have the capability to learn temporal 
dependencies in sequential data. Furthermore, transformer-
based models such as BERT have transformed NLP tasks by 
capturing extensive contextual information and learning 
contextual word embeddings. Despite significant progress in 
automated toxic comment classification using ML and DL 
techniques, existing models often struggle with accurately 
capturing context, handling imbalanced datasets, and 
distinguishing between subtle toxic and non-toxic content. 
Additionally, many previous works either use only shallow 
feature representations or do not take full advantage of 

 

 

Future Technology 

Open Access Journal 

Journal homepage: https://fupubco.com/futech 

 
ISSN 2832-0379 

August 2025| Volume 04 | Issue 03 | Pages 76-84 

https://doi.org/10.55670/fpll.futech.4.3.8 

 

 

 

mailto:sushma.cse2@gmail.com
https://fupubco.com/futech
https://doi.org/10.55670/fpll.futech.4.3.8


 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 76-84 

77 

 

classical and transformer-based embeddings. We aim to 
resolve these issues in this work by conducting experiments 
with hybrid DL models that rely on the TF-IDF, Word2Vec, 
and BERT embeddings for better toxic comment detection. 

This work distinguishes itself from prior studies by 
offering a comprehensive and unified comparison of various 
word embedding strategies (TF-IDF, Word2Vec, BERT) in 
combination with sequential models (RNN, LSTM, GRU), all 
evaluated under consistent preprocessing, tokenization, and 
training configurations. Unlike existing research that often 
benchmarks one or two models, this study explores a wide 
array of hybrid architectures (e.g., LSTM+BERT) to identify 
optimal pairings for toxic comment classification. The 
implementation is designed to reflect practical deployment 
scenarios using real-world metrics and balanced 
experimental design. The main contributions of this work are 
listed below: 
• To design RNN and other deep learning models to 

compare for this toxic comment classification. 
• To explore the performance of different embedding 

approaches, such as TF-IDF, Word2Vec, and BERT, for 
semantics and context features. 

• Investigate using hybrid architectures that connect 
traditional deep learning with transformer-related 
embeddings. 

• To determine the classification accuracy, precision, 
recall, and F1-score of all models on the basis of real-
world toxic comment data. 

2. Literature survey 

A. Albladi et al. [1] investigated sentiment research 
specifically on Twitter, which is extensive in scale and a 
valuable data source for understanding public opinion. They 
reviewed desirable and problematic features and metrics of 
ML, DL, and hybrid approaches. You focus on the BERT and 
GPT transformer architectures, although seismic pre-
processing techniques, the extraction of features, and 
sentiment lexicons were also summarily considered. This 
research aimed to give a comprehensive landscape of use 
cases in Twitter sentiment analysis, offering useful insights 
for practitioners and researchers in this area.  

Z. Hao et al. [2] discussed the intricate complexities of 
gathering public views, particularly regarding social media 
and its impacts on social incidents. A two-tier hierarchical 
mechanism was proposed for report categorization and 
review-level sentiment analysis, aiming to effectively derive 
sentiment through a multi-step approach. This method was 
augmented with advances in model architecture, including 
embedding tables and gating mechanisms. Moreover, they 
proposed a new distributed DL model which is based on 
blockchain isomerism learning for the security risks and 

reducing the model silos. Through a series of extensive 
experiments, they showed that, for performance and 
aggregation efficiency, their approach vastly outperforms 
existing methods. 

X. Wang et al. [3] addressed the challenges in sentiment 
analysis by introducing a novel model that integrates 
multimodal multiscale features based on a fuzzy-deep neural 
network. It utilizes intrinsic feature representations across 
text, audio, and image data. The model employs fuzzy logic 
rules to increase the adaptability to the ambiguity present in 
sentiment representations. Additionally, they incorporated a 
dual attention mechanism to flexibly attend to essential 
components in the multimodal data, thereby enhancing 
feature extraction and improving contextual awareness. We 
conducted extensive validation of the model's performance 
using several datasets and characterized its ability to model 
the complexities of human emotion better than existing 
methods. H. T. Phan et al. [4] devised a novel method of 
aspect-level sentiment analysis based on the application of 
three GCNs, which they coined as multigraphic convolutional 
network (MulGCN). Using the dependency parser tree, 
affective information from SenticNet , and inter-aspect-
awareness, this approach captures both syntax and semantics 
as well as context. This paper presents a model that 
introduces a solution for aspect-level sentiment analysis and 
improves its performance by addressing the difficulty of 
effectively leveraging relevant features from different 
knowledge sources. Experimental evaluations over three 
benchmark datasets demonstrate that the MulGCN model 
beats the state-of-the-art and improves both accuracy and F1 
score for aspect-level sentiment analysis. S. Ali et al. [5] 
highlighted the issue of [event classification] in low-resource 
languages, such as Urdu, and underline the need for well-
formalised linguistic datasets. The dataset contained a total 
of 103,771 sentences from five different social media 
platforms and was obtained for the purpose of classifying text 
in the Urdu language in a multiclass classification approach. 
The 16 event categories were used in the classification task. 
The SMFCNN classifier showed the highest accuracy 
(88.29%) among the methods tested. Furthermore, on this 
dataset, XLM-R+ (a proposed transformer-based model) 
outperformed them with an accuracy of 89.8% as well. 

 W. Gong [6] developed a sentiment classification 
algorithm for textual data mining based on bidirectional long 
short-term memory network (BiLSTM), which demonstrated 
the relevance of emotion manifestation in e-commerce and 
social media data with DL methods. The model enhances the 
accuracy in distinguishing sentiment in such cases and 
outperforms conventional sentiment classification 
techniques due to the utilization of BiLSTM, which adopts 
bidirectional contexts of text segments [4,5]. This study 
demonstrates that BiLSTM can also be a viable option for 
sentiment analysis tasks, such as customer feedback analysis. 
S. A. Mostafa et al. [7] employed a sentiment analysis and 
classification method for Amazon Alexa products. The dataset 
contains 3150 reviews and was labelled for positive or 
negative sentiment. They therefore trained four classifiers 
and evaluated the performance metrics. Analysis of customer 
feedback was found to be highly effective using RF, which 
was the best-performing classifier. S. Mehta et al. [8] applied 
CNN as a new method for sentiment analysis with Federated 
Learning. It enables model-trained decentralization by 
processing multiple devices without transferring data to the 
central server, thereby improving privacy while maintaining 
sufficient performance for the sentiment classifier. It is based 
on high accuracy and a good ROC AUC score, denoting good 

Abbreviations 

ANN  Artificial Neural Network 

CNN  Convolutional Neural Network 

RNN  Recurrent Neural Network 

LSTM  Long Short-Term Memory 

GRU  Gated Recurrent Unit 

NLP  Natural Language Processing 

BERT  Bidirectional Encoder Representations    

                                         from  Transformers 

TF-IDF  Term Frequency–Inverse Document  

                                         Frequency 

Word2Vec Word to Vector 



 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 76-84 

78 

 

separation of classes as different sentiment groups. As a 
result, it enhanced data privacy and efficiency due to the use 
of federated learning. X. He et al. [9] compared SVM for 
sentiment analysis with other traditional classifiers such as 
LR, KNN, NB, and XGBoost. This study applies sentiment 
analysis to classify comments about "Huawei Mate60" on the 
Little Red Book platform. Compared to the other models, SVM 
proved to be more efficient for sentiment classification, 
making it a recommended approach for analyzing consumers' 
feelings, which can help brands manage their products 
better. M. Aamir et al. [10] conducted a similar study using 
different ML and DL techniques to find public sentiment on 
tweets about Ola and Uber. In this study, different algorithms 
were evaluated to understand customer feedback for these 
ride-hailing services. The study assisted in improving the 
accuracy of algorithms employed and facilitated companies to 
tailor their services according to user sentiments, leading to 
a healthier online ecosystem. 

Q. Zeng et al. [11] proposed a neural network model for 
aspect-based sentiment classification, integrating a self-
attention mechanism to capture contextual semantic 
information. Relative Position Representations (PRP) were 
produced by attending to a global context, while pairs of 
words were given ReLU gated convolutional networks for 
sentiment feature extraction. It has also been shown that 
their model outperforms every other model in terms of 
predicting valence and arousal on the SemEval dataset, 
Tweets, and CVAT, confirming its robustness for sentiment 
analysis. M. Khalid et al. [12] proposed a Sentiment Majority 
Voting Classifier (SMVC) to analyze the sentiment of deepfake 
technology-related tweets. The authors used majority voting 
to aggregate the predictions from several lexicon-based 
models and used transfer learning with LSTM and Decision 
Tree models. The method reached the best accuracy of 98.9%, 
demonstrating robustness in sentiment classification. 
A.L.Rao et al. [13] focused on sentiment analysis of the airline 
tweet or airline reviews Kaggle dataset. They suggested an 
ensemble architecture for CNN and LSTM for improving 
sentiment classification. They benchmarked this ensemble 
model against a standalone LSTM model, which was also 
trained on the same dataset, and reported the performance of 
each and stated that the ensemble approach provided 
superior performance, providing an improvement on 
standard methods. Y. Matrane et al. [14] performed the 
sentiment analysis of MD, discussing specifically the issues 
related to dialect-specific preprocessing techniques. They 
reported better results by not utilizing traditional techniques 
like stemming and by using the QARiB feature extractor with 
BiGRU. The results of DarijaBERT on the FB dataset showed 
their fine-tuning approach to be effective, signifying the need 
for dialect-specific techniques in dealing with the Arabic 
dialect. H. Shuqin et al. [15] implemented a BERT-BiLSTM-
ATTENTION (BBA) model to recognize sentiment for course 
evaluation. The model was built by leveraging the BERT 
model for context, along with BiLSTM and attention 
mechanisms to better refine the focus across the relevant 
components of text. The BBA model beat the existing 
methods, showing that the deep semantic representation of 
education-based feedback could be achieved through this 
model. 

A. He et al.  [16] introduced a novel deep tensor evidence 
fusion network for multimodal sentiment classification. They 
introduced a joint view scoring net that integrates LSTM and 
tensor neural networks to extract the intermodal and 
intramodal rich information. They also proposed a temporal 
cue evaluation network based on temporal granularity and a 

trustworthy fusion layer to enhance the accuracy and 
robustness of decisions. On the CMU-MOSEI and CMU-MOSI 
datasets, the results were better than those of SOTA methods. 
Z. Wang et al. [17] proposed a multi-label classification 
approach for handling toxic comments in social media, 
specifically focusing on Indonesia's Twitter platform. Using 
two kinds of word vectors from BERT's hidden layers and 
combining both BERT + BiLSTM method, the model achieved 
better performance. In such a complex form of sentiment 
analysis task, the proposed model achieved an accuracy of 
0.889, precision of 0.925, recall of 0.917, and F1 score of 0.91, 
thus proving the efficacy of task-specific semantic embedding 
and sequential learning through BiLSTM. S. K. Putri et al. [18] 
applied various ML models to predict toxic leadership in the 
Moroccan IT sector. In this regard, the study identified 
Undermining Behavior, Narcissistic Traits, Unjust Treatment, 
and Fear of Retribution as the main contributors to toxic 
leadership, as it sought to provide a comprehensive analysis 
of how toxic leadership could be predicted using different 
types of ML algorithms. A. Lakshmanarao et al. [19] sentiment 
analysis of airlines tweet gathered from Kaggle. They applied 
different neural network approaches to classify tweets into 
different categories. 

S. Dutta et al. [20] addressed the toxic comment 
detection problem in Assamese, a morphologically rich and 
ambiguous language, which poses a challenge to sentiment 
analysis. This paper is part of a very large study of general 
NMF topic modeling, which indicates preceding work by 
manually collecting 19,550 comments from social media sites 
and testing a number of ML models. All these models have 
been trained against various text representations (count 
vector, count vector + TF-IDF, n-gram, etc.) The best F1-score 
was 94% with SVM + count vector + TF-IDF compared to the 
other models. Y. Mamani et al. [21] provided a summary of 
recent ML techniques used for sentiment analysis, developing 
a framework to classify sentiment models according to their 
structure. The paper also discussed challenges faced by the 
community and emerging trends, providing future directions 
for the research in sentiment analysis. 

Rahul et al. [22] addressed the classification problem for 
toxic comments, which can be used to measure the severity of 
online harassment. They examined online comments without 
a focus on toxicity using six machine learning (ML) 
algorithms. They worked to enhance the classification of 
negative information within textual comments, utilizing an 
enhanced classification to mitigate harmful information. This 
enables organizations to identify toxins in discussions and 
take action to lead to better environments. M. Aquino et al. In 
Ref [23], the authors explored a new ML-based method to 
detect comment toxicity using text and emojis. They trained 
a bidirectional LSTM model using GloVe and emoji2vec 
combined word embeddings. It provides a new labeled 
dataset of text and emoji data, providing an effective means 
of comment toxicity detection. T. V. Sai Krishna et al. Using a 
range of features and both ML & DL methods, Ref [24] 
proposes sentiment classification on Twitter. They used 
several ML models. They also proposed a new end-to-end 
ensemble approach of using ML and DL models, which yields 
higher accuracy vs. these techniques when applied to i.e., real-
time Twitter data for sentiment classification. 

For example, Singh et al. [25] detected levels of toxicity 
in comments on social media using the Jigsaw dataset from 
Google. The dataset is a multilabel classification task with 
several classes. The logistic regression model performed the 
best in terms of accuracy and Hamming loss among the other 
models, indicating that it is well-suited to the task of 



 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 76-84 

79 

 

identifying toxic comments. Venugopal [26] explored the 
classification of toxic comments and identified challenges of 
detecting the toxicity of comments in multiple languages in a 
centralized system for comment detection across social 
media platforms. They used state-of-the-art DL architectures 
such as BERT or XLM-RoBERTa for multilingual toxicity 
detection. The paper noted that these models, when paired 
with appropriate preprocessing of datasets and tuning of 
hyperparameters, can provide significant improvement in the 
accuracy of detecting toxic comments compared to traditional 
models such as SVM. N. Boudjani et al. [27] used N-grams, 
linguistic features, and a lexicon of insulting words to create a 
supervised method for the classification of French toxic 
comments. Using linear SVM and decision tree classifiers, 
their approach yielded precision, recall, and F1-score values 
of 87%, 83%, and 78%, respectively. A. Jessica et al. applied 
BERT-CNN and BERT-LSTM hybrid models for detecting 
cyberbullying in online comments. This model consists of 
BERT and CNN, which are created through the combination of 
sentences by using BERT so BERT has good language 
understanding ability, and using CNN for feature extraction 
work.  

3. Methodology 

The proposed method for toxic comment classification is 
shown in Figure 1. The proposed method for toxic comment 
detection is designed to evaluate various DL models in 
combination with word embedding techniques such as 
Word2Vec, TF-IDF, and BERT. The method is applied to the 
Jigsaw Toxic Comment Dataset, which contains labeled 
English comments categorized as toxic or non-toxic. The 
dataset undergoes preprocessing to clean the text by handling 
inconsistencies such as missing values and ensuring that the 
comment texts are appropriately formatted for further 
analysis. 

 

 
Figure 1. Proposed methodology for toxic comment classification 

TF-IDF and Word2Vec are used to extract text features in 
two distinct ways. TF-IDF computes the relative importance 
of a word within a document by analyzing its frequency in 
relation to the entire corpus, generating sparse numerical 
vectors. In contrast, Word2Vec learns distributed word 
representations by training on the tokenized text data, 
capturing semantic relationships between words through co-
occurrence patterns. Each comment is then represented as a 
fixed-length vector by averaging the embeddings of its words. 
These two methods provide complementary insights—TF-
IDF emphasizes statistical significance, while Word2Vec 
captures contextual meaning. The next step involves applying 
different DL models, namely RNN, LSTM, and GRU, to classify 
the comments as either toxic or non-toxic. Each model 
architecture consists of an embedding or input layer, a 
recurrent layer (RNN, LSTM, or GRU), and a dense output 
layer for binary classification. The structure remains 
consistent across experiments, with only the embedding 
source (TF-IDF, Word2Vec, or BERT) varying. BERT, a 
transformer-based model, is also employed for richer feature 
extraction. This pre-trained BERT model uses a large corpus 
to train and fine-tunes itself on the toxic comment dataset. In 
this setup, token embeddings are extracted using the 
DistilBERT tokenizer, particularly from the [CLS] token. 
These embeddings are then passed through the recurrent 
models to improve contextual interpretation [28]. 

This architecture produces hybrid models such as 
RNN+BERT, LSTM+BERT, and GRU+BERT, where BERT 
serves solely as a feature extractor, providing contextual 
embeddings. These embeddings are fed as input to the 
corresponding recurrent layers for learning temporal 
dependencies prior to classification. BERT is not used as a 
classifier in this work, but it significantly enhances the input 
representation for downstream learning. This approach 
enables the models to recognize complex and contextual 
relationships between words in comments. Final 
classification is performed by the dense output layer. The 
entire framework is designed to output the probability of a 
comment being toxic based on the learned features. All 
models are evaluated on a range of performance metrics, 
including accuracy, precision, recall, and F1 score. The 
outcome of each model is compared to decide which is the 
best way to detect toxicity in online commentary. This 
experimental design enables a comparative evaluation of 
different embedding-model combinations. Using traditional 
feature extraction techniques such as Word2Vec and TF-IDF 
together with state-of-the-art transformer-based models like 
BERT, the proposed algorithm has high classification 
accuracy. It also verifies the potential for hybrid DL models as 
a means of fulfilling text categorisation work. 

3.1 Data collection 
This paper used the Jigsaw Toxic Comment Dataset, 

collected from Kaggle [29]. The dataset contains a collection 
of English language comments, each labeled as either toxic or 
non-toxic. The dataset consists of two important columns: 
comment_text, which contains the actual comment text, and 
toxic,  a binary target variable (1 refers to toxic comments, 
while 0 refers to all other types of comments). This data 
comes from a variety of online sources, including Civil 
Comments and Wikipedia talk page edits. Text-based 
prediction of whether a comment is toxic or not. 

 
 
 
 



 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 76-84 

80 

 

3.2 Preprocessing 
In the preprocessing phase, a variety of procedures were 

used to clean and prepare the data for modeling. This 
involved eliminating all kinds of inconsistencies, such as 
missing values in required columns (comment_text and 
toxic). Missing entries like these are corrected to preserve the 
completeness of the dataset and its suitability for analysis. 
Later, the data was tokenized and transformed into a form 
suitable for input into ML models, optimizing both training 
and validation datasets so that they would work perfectly 
with the model. 

4. Results and discussion 

4.1 Applying RNN with TF-IDF 
In this phase, the TF-IDF method is applied in 

combination with a SimpleRNN model for toxic comment 
classification. The text data is first vectorized using the TF-IDF 
technique, which converts the text into a numerical 
representation based on the frequency of terms in the dataset. 
To perform dimensionality reduction, 5,000 input features 
are extracted from the text data so that the model can 
concentrate on the most pertinent terms. We first split our 
data into training and test sets as follows: 80% for training 
and 20% for testing. The model consists of an input 
embedding layer, an RNN layer with 64 units, followed by an 
output dense layer with a unit for binary classification. To 
accelerate the training process, the model is trained for five 
epochs with a batch size of 256. The final evaluation on the 
test set yields a validation accuracy of around 89%, indicating 
that the model successfully captures the sequential nature of 
the text data and performs adequately in classifying toxic 
comments. 

4.2 Applying LSTM with TF-IDF 
In this step, an LSTM model is utilized in conjunction 

with TF-IDF for classifying toxic comments. The text data is 
vectorized using the TF-IDF method, which extracts relevant 
features based on the term frequency-inverse document 
frequency. The neural network is constructed in the form of 
an embedding layer that takes the input dimensions, and 
then an LSTM layer, which has 64 units. We add a dense 
output layer for binary classification (toxic vs non-toxic 
comments). The training process gradually improves 
accuracy, with the training done over 3 epochs using a 
function to train on one set of the dataset and the given 
statements, with a batch size of 16. The trained model is then 
evaluated on the test set, giving us a validation accuracy of 
90.46% , demonstrating the model’s ability to capture long-
term dependencies within the data whilst providing a 
baseline for the classification of toxic comments. 

4.3 Applying GRU with TF-IDF 
This section demonstrates the application of a GRU 

(Gated Recurrent Unit) model with TF-IDF for the task of toxic 
comment classification. Similar to the LSTM model, TF-IDF is 
used in this model. The GRU model consists of an embedding 
layer, single GRU layer of size 64, followed by a dense output 
layer for predicting binary classes. The model is evaluated on 
the test data after training with a batch size of 16. The final 
validation accuracy ~90% generally indicates that the GRU 
model also effectively captured the relevant patterns from the 
text and achieved comparable classification accuracy with 
the LSTM model for classifying a toxic comment. 

 

 

4.4 Applying RNN with Word2Vec 
In this approach, Word2Vec embeddings were combined 

with an RNN to classify toxic and non-toxic comments. 
Word2Vec embeddings were first generated by training on 
the tokenized comments from the Jigsaw Toxic Comment 
Dataset. Each comment was represented by a fixed-length 
vector by averaging the embeddings of its words, with zero 
vectors assigned to out-of-vocabulary words. The padding 
was applied to ensure uniform input lengths for the RNN. The 
model was trained for 20 epochs with a batch size of 16. 
Figure 2 shows epoch-wise accuracy, and Figure 3 shows loss 
values with the model. It achieved a validation accuracy of 
approximately 90.76%, demonstrating the effectiveness of 
using Word2Vec embeddings and RNNs for toxic comment 
classification. 

Figure 2.  RNN+ Word2vec model epoch wise accuracy 

 
 

 
Figure 3. RNN+ Word2vec model epoch wise loss 

 

 

 



 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 76-84 

81 

 

4.5 Applying LSTM with Word2Vec 
In this approach, the LSTM model was combined with 

Word2Vec embeddings. The Word2Vec embeddings were 
first generated by training a Word2Vec model on the 
tokenized comments from the Jigsaw Toxic Comment Dataset. 
Each comment was then represented as a fixed-length vector, 
computed by averaging the word embeddings in the 
comment. The data was divided into 80% training and 20% 
testing. Next, padding was performed to ensure uniform input 
lengths for the LSTM model.  The model was trained for 20 
epochs with a batch size of 16, achieving a 92.25% validation 
accuracy. Epoch-wise accuracy values with the model are 
shown in Figure 4. Epoch-wise loss values with the model are 
shown in Figure 5. The fact that LSTM with word2vec 
embeddings outperforms all other models and proves 
efficient demonstrates the potential of this combination.  

 
Figure 4.  LSTM+ Word2vec model epoch-wise accuracy 

 

 
Figure 5.  LSTM+ Word2vec model epoch wise loss 

4.6 Applying GRU with Word2Vec 
In this section, the GRU model is applied with Word2Vec 

embeddings for toxic comment classification. Similar to the 
previous methods, the Word2Vec model was trained on the 
tokenized comments from the Jigsaw Toxic Comment Dataset. 
Each comment was then transformed into a fixed-length 
vector using the average of word embeddings in the comment. 
The model was trained for 20 epochs with a batch size of 16. 
The GRU model achieved a validation accuracy of 92.11%, 

showcasing its ability to capture sequential patterns in the 
text and effectively classify toxic and non-toxic comments. 
Figure 6 and Figure 7 show epoch-wise accuracy and loss 
values with the model, respectively.  

Figure 6.  GRU+ Word2vec model epoch-wise accuracy 

 

Figure 7.  GRU+ Word2vec model epoch-wise loss 

4.7 Applying RNN with BERT 
An RNN model is combined with BERT embeddings to 

classify toxic comments. The dataset is preprocessed using 
the DistilBERT tokenizer, and embeddings are extracted from 
the [CLS] token. These embeddings are passed through a 
SimpleRNN layer and a dense layer for binary classification. 
After training for 10 epochs, the model achieved a validation 
accuracy of approximately 75.7%, demonstrating the 
effectiveness of RNNs with BERT embeddings for toxic 
comment classification. Figure 8 and Figure 9 depict epoch-
wise accuracy and loss for this model.  



 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 76-84 

82 

 

 
Figure 8.  RNN+ BERT model epoch wise accuracy 

 

 
Figure 9.  RNN+ BERT model epoch wise loss 

4.8 Applying LSTM and GRU with BERT 
In addition to the RNN-based model, the effectiveness of 

LSTM and GRU models combined with BERT embeddings for 
toxic comment classification is also evaluated. Both models 
leverage BERT's pre-trained embeddings, which are 
extracted using the DistilBERT tokenizer. The embeddings 
are then processed through LSTM and GRU layers, followed 
by dense layers for binary classification. For the LSTM model, 
a validation accuracy of approximately 80% was achieved, 
while the GRU model demonstrated a slightly lower validation 
accuracy of 79%.  

4.9 Comparison of applied models 
The performance of the proposed models was evaluated 

using various algorithms in combination with different 
feature extraction techniques, including TF-IDF, Word2Vec, 
and BERT. The accuracies achieved by each model are shown 
in Table 1 and Figure 10. As seen from the table, the 
combination of LSTM with Word2Vec achieved the highest 
accuracy of 92.25%, followed by LSTM + BERT at 92.11%. 

RNN-based models demonstrated competitive performance, 
with RNN + Word2Vec reaching an accuracy of 90.76%, and 
RNN + TFIDF yielding 89.46%. GRU models exhibited lower 
accuracies in comparison, with GRU + TFIDF achieving 
75.70% and GRU + Word2Vec and GRU + BERT showing 
accuracies of 80.00% and 79.00%, respectively. 

Table 1. Comparison of applied models 

 

 

 
Figure 10. Accuracy comparison of applied models 

5. Conclusion  

In this paper, various deep learning models combined 
with different feature extraction techniques, such as TF-IDF, 
Word2Vec, and BERT, were evaluated for the task of toxic 
comment classification. The results demonstrated that deep 
learning models, particularly LSTM-based architectures, 
outperformed others in terms of accuracy. The highest 
accuracy of 92.25% was achieved with the combination of 
LSTM and Word2Vec, closely followed by LSTM + BERT at 

Method Accuracy 

RNN + TFIDF 89.46 

LSTM + TFIDF 90.46 

RNN + Word2Vec 90.76 

LSTM + Word2Vec 92.25 

RNN + BERT 90.76 

LSTM + BERT 92.11 

GRU + TFIDF 75.70 

GRU + Word2Vec 80.00 

GRU + BERT 79.00 

0 20 40 60 80 100

RNN + TFIDF

LSTM + TFIDF

RNN + Word2Vec

LSTM + Word2Vec

RNN + BERT

LSTM + BERT

GRU + TFIDF

GRU + Word2Vec

GRU + BERT

Accuracy Comparison



 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 76-84 

83 

 

92.11%. These findings indicate that LSTM networks, with 
their ability to capture long-term dependencies, are 
particularly well-suited for handling complex text data in 
sentiment analysis tasks. On the other hand, RNN and GRU 
models showed slightly lower performance but still provided 
competitive results, with RNN + Word2Vec achieving a 
90.76% accuracy. While GRU models demonstrated lower 
accuracy compared to LSTM, they still hold potential for 
future optimization and experimentation. Future work could 
focus on further refining these models, experimenting with 
larger datasets, and exploring more advanced techniques, 
such as fine-tuning pre-trained models or combining multiple 
models, to improve classification accuracy. This paper 
demonstrates the potential of using advanced DL techniques 
and pre-trained models for effectively addressing the 
challenge of toxic comment classification.  

Ethical issue 
The authors are aware of and comply with best practices in 
publication ethics, specifically with regard to authorship 
(avoidance of guest authorship), dual submission, 
manipulation of figures, competing interests, and compliance 
with policies on research ethics. The author adheres to 
publication requirements that the submitted work is original 
and has not been published elsewhere. 

Data availability statement 
The manuscript contains all the data. However, more data will 

be available upon request from the authors. 

Conflict of interest 

The authors declare no potential conflict of interest. 

References 

[1] A. Albladi, M. Islam and C. Seals, "Sentiment Analysis of 

Twitter Data Using NLP Models: A Comprehensive 

Review," in IEEE Access, vol. 13, pp. 30444-30468, 

2025, doi: 10.1109/ACCESS.2025.3541494. 

[2] Z. Hao et al., "A Novel Public Sentiment Analysis 

Method Based on an Isomerism Learning Model via 

Multiphase Processing," in IEEE Transactions on 

Neural Networks and Learning Systems, vol. 36, no. 1, 

pp. 249-259, Jan. 2025, doi: 

10.1109/TNNLS.2023.3274912. 

[3] X. Wang, J. Lyu, B. -G. Kim, B. D. Parameshachari, K. Li 

and Q. Li, "Exploring Multimodal Multiscale Features 

for Sentiment Analysis Using Fuzzy-Deep Neural 

Network Learning," in IEEE Transactions on Fuzzy 

Systems, vol. 33, no. 1, pp. 28-42, Jan. 2025, doi: 

10.1109/TFUZZ.2024.3419140. 

[4] H. T. Phan, V. D. Nguyen and N. T. Nguyen, "MulGCN: 

MultiGraph Convolutional Network for Aspect-Level 

Sentiment Analysis," in IEEE Access, vol. 13, pp. 

26304-26317, 2025, doi: 

10.1109/ACCESS.2025.3537340. 

[5] S. Ali, U. Jamil, M. Younas, B. Zafar and M. Kashif Hanif, 

"Optimized Identification of Sentence-Level Multiclass 

Events on Urdu-Language-Text Using Machine 

Learning Techniques," in IEEE Access, vol. 13, pp. 1-25, 

2025, doi: 10.1109/ACCESS.2024.3522992. 

[6] W. Gong, "Text Sentiment Classification Algorithm 

Based on BiLSTM Deep Learning," 2024 International 

Conference on Industrial IoT, Big Data and Supply 

Chain (IIoTBDSC), Wuhan, China, 2024, pp. 83-87, doi: 

10.1109/IIoTBDSC64371.2024.00025. 

[7] S. A. Mostafa, W. S. Al-Dayyeni, A. N. Kareem, M. A. 

Jubair, M. M. Jaber and B. A. Khalaf, "Classification and 

Sentiment Analysis of Amazon Alexa Reviews," 2024 

1st International Conference on Logistics (ICL), Jeddah, 

Saudi Arabia, 2024, pp. 1-5, doi: 

10.1109/ICL62932.2024.10788570. 

[8] S. Mehta and A. Bhalla, "Enhanced Sentiment 

Classification with Federated Learning CNNs: 

Exploring Five Sentiment Categories," 2024 3rd 

International Conference for Advancement in 

Technology (ICONAT), GOA, India, 2024, pp. 1-5, doi: 

10.1109/ICONAT61936.2024.10774751. 

[9] X. He, "Sentiment Classification of Social Media User 

Comments Using SVM Models," 2024 5th International 

Seminar on Artificial Intelligence, Networking and 

Information Technology (AINIT), Nanjing, China, 2024, 

pp. 1755-1759, doi: 

10.1109/AINIT61980.2024.10581547. 

[10] M. Aamir, L. J and Sweety, "A Comparative Study of ML 

and DL Approaches for Twitter Sentiment 

Classification," 2024 International Conference on 

Electrical Electronics and Computing Technologies 

(ICEECT), Greater Noida, India, 2024, pp. 1-5, doi: 

10.1109/ICEECT61758.2024.10738938. 

[11] Q. Zeng, "Design of Intelligent Sentiment Classification 

Model Based on Deep Neural Network Algorithm in 

Social Media," in IEEE Access, vol. 12, pp. 81047-

81056, 2024, doi: 10.1109/ACCESS.2024.3409818. 

[12] M. Khalid et al., "Novel Sentiment Majority Voting 

Classifier and Transfer Learning-Based Feature 

Engineering for Sentiment Analysis of Deepfake 

Tweets," in IEEE Access, vol. 12, pp. 67117-67129, 

2024, doi: 10.1109/ACCESS.2024.3398582. 

[13] A. Lakshmanarao, C. Gupta and T. S. R. Kiran, "Airline 

Twitter Sentiment Classification using Deep Learning 

Fusion," 2022 International Conference on Smart 

Generation Computing, Communication and 

Networking (SMART GENCON), Bangalore, India, 2022, 

pp. 1-4, doi: 

10.1109/SMARTGENCON56628.2022.10084207. 

[14] Y. Matrane, F. Benabbou and Z. Ellaky, "Enhancing 

Moroccan Dialect Sentiment Analysis Through 

Optimized Preprocessing and Transfer Learning 

Techniques," in IEEE Access, vol. 12, pp. 187756-

187777, 2024, doi: 10.1109/ACCESS.2024.3514934. 

[15] H. Shuqin and R. C. Raga, "A Deep Learning Model for 

Student Sentiment Analysis on Course Reviews," in 

IEEE Access, vol. 12, pp. 136747-136758, 2024, doi: 

10.1109/ACCESS.2024.3463793. 

[16] A. He and M. Abisado, "Text Sentiment Analysis of 

Douban Film Short Comments Based on BERT-CNN-

BiLSTM-Att Model," in IEEE Access, vol. 12, pp. 45229-

45237, 2024, doi: 10.1109/ACCESS.2024.3381515. 

[17] Z. Wang, G. Xu, X. Zhou, J. Y. Kim, H. Zhu and L. Deng, 

"Deep Tensor Evidence Fusion Network for Sentiment 

Classification," in IEEE Transactions on Computational 

Social Systems, vol. 11, no. 4, pp. 4605-4613, Aug. 

2024, doi: 10.1109/TCSS.2022.3197994. 

[18] S. K. Putri, A. Amalia and T. F. Abidin, "Sentiment 

Analysis Multi-Label of Toxic Comments using BERT-

BiLSTM Methods," 2024 International Conference on 



 S. Sushma et al. /Future Technology                                                                                         August 2025| Volume 04 | Issue 03 | Pages 76-84 

84 

 

Electrical Engineering and Informatics (ICELTICs), 

Banda Aceh, Indonesia, 2024, pp. 120-124, doi: 

10.1109/ICELTICs62730.2024.10776338. 

[19] A. Lakshmanarao, A. Srisaila and T. S. R. Kiran, "Twitter 

Sentiment Classification with Deep Learning LSTM for 

Airline Tweets," 2022 8th International Conference on 

Advanced Computing and Communication Systems 

(ICACCS), Coimbatore, India, 2022, pp. 520-524, doi: 

10.1109/ICACCS54159.2022.9785208. 

[20] S. Dutta, M. Neog and N. Baruah, "Assamese Toxic 

Comment Detection On Social Media Using Machine 

Learning Methods," 2024 Second International 

Conference on Emerging Trends in Information 

Technology and Engineering (ICETITE), Vellore, India, 

2024, pp. 1-8, doi: 10.1109/ic-

ETITE58242.2024.10493331. 

[21] Y. Mamani-Coaquira and E. Villanueva, "A Review on 

Text Sentiment Analysis With Machine Learning and 

Deep Learning Techniques," in IEEE Access, vol. 12, pp. 

193115-193130, 2024, doi: 

10.1109/ACCESS.2024.3513321. 

[22] Rahul, H. Kajla, J. Hooda and G. Saini, "Classification of 

Online Toxic Comments Using Machine Learning 

Algorithms," 2020 4th International Conference on 

Intelligent Computing and Control Systems (ICICCS), 

Madurai, India, 2020, pp. 1119-1123, doi: 

10.1109/ICICCS48265.2020.9120939. 

[23] M. Aquino et al., "Toxic Comment Detection: Analyzing 

the Combination of Text and Emojis," 2021 IEEE 18th 

International Conference on Mobile Ad Hoc and Smart 

Systems (MASS), Denver, CO, USA, 2021, pp. 661-662, 

doi: 10.1109/MASS52906.2021.00097. 

[24] T. V. Sai Krishna, T. S. Rama Krishna, S. Kalime, C. V. 

Murali Krishna, S. Neelima, and R. R. PBV, “A novel 

ensemble approach for Twitter sentiment 

classification with ML and LSTM algorithms for real-

time tweets analysis,” Indonesian Journal of Electrical 

Engineering and Computer Science, vol. 34, no. 3. 

Institute of Advanced Engineering and Science, p. 

1904, Jun. 01, 2024. doi: 

10.11591/ijeecs.v34.i3.pp1904-1914. 

[25] N. K. Singh and S. Chand, "Machine Learning-based 

Multilabel Toxic Comment Classification," 2022 

International Conference on Computing, 

Communication, and Intelligent Systems (ICCCIS), 

Greater Noida, India, 2022, pp. 435-439, doi: 

10.1109/ICCCIS56430.2022.10037626. 

[26] N. L. V. Venugopal, P. Kanchanamala, S. Muppidi, T. B. 

Prakash, T. Neelima and S. A. Devi, "Multilingual Toxic 

Comment Classification using Deep Learning," 2024 

2nd International Conference on Self Sustainable 

Artificial Intelligence Systems (ICSSAS), Erode, India, 

2024, pp. 752-757, doi: 

10.1109/ICSSAS64001.2024.10760913. 

[27] N. Boudjani, Y. Haralambous and I. Lyubareva, "Toxic 

Comment Classification For French Online Comments," 

2020 19th IEEE International Conference on Machine 

Learning and Applications (ICMLA), Miami, FL, USA, 

2020, pp. 1010-1014, doi: 

10.1109/ICMLA51294.2020.00164. 

[28] A. Jessica, M. S. Sugiarto, Jerry, S. Achmad and R. 

Sutoyo, "A Hybrid Deep Learning Techniques Using 

BERT and CNN for Toxic Comments Classification," 

2024 International Conference on Information 

Management and Technology (ICIMTech), Bali, 

Indonesia, 2024, pp. 393-398, doi: 

10.1109/ICIMTech63123.2024.10780934. 

[29] https://www.kaggle.com/c/jigsaw-multilingual-toxic-

comment-classification/data 

 

 

 

  

This article is an open-access article distributed under the 

terms and conditions of the Creative Commons Attribution 

(CC BY) license 

(https://creativecommons.org/licenses/by/4.0/). 

https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/data
https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification/data
https://creativecommons.org/licenses/by/4.0/

