







































Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 
84-92 

84 

 

 

 

Article 

Multimodal fusion and AI context awareness in 

smart kitchens: deep learning for personalized 

recommendation and real-time monitoring 
Jiaying Li, Jinho Yim*  

Department of Smart Experience Design, Kookmin University, Seoul 01706, Republic of Korea 

A R T I C L E   I N F O 
 

Article history: 
Received 27 August 2025  
Received in revised form 
08 October 2025 
Accepted 25 October 2025 
 
Keywords:  
Multimodal fusion, Context awareness, 
Smart kitchens, Reinforcement learning, 
Personalized recommendation 
 
*Corresponding author 
Email address: 
hci.yim@kookmin.ac.kr 
 
DOI: 10.55670/fpll.futech.5.1.9 

A B S T R A C T 
 

The proliferation of artificial intelligence (AI) and the Internet of Things (IoT) 
has positioned smart kitchens as a frontier for innovation in personalized 
nutrition, safety monitoring, and sustainable consumption. Despite rapid 
progress, existing approaches remain fragmented: vision-based systems 
struggle with occlusion, speech-driven interfaces are vulnerable to noise, and 
IoT sensor networks, while reliable, often lack semantic integration with user 
preferences. Personalized recommender systems further suffer from static 
designs that fail to adapt to evolving contexts. Addressing these limitations, this 
study introduces a multimodal deep learning framework that unifies cross-
modal attention and reinforcement learning to achieve context-aware 
personalization. Visual, auditory, and sensor streams are embedded into a 
shared representation, fused via attention mechanisms, and subsequently 
optimized through a reinforcement learning agent that balances nutritional 
goals, user satisfaction, and safety requirements. Empirical evaluation across 
three multimodal datasets demonstrates significant improvements over strong 
baselines, with gains of +8.4% in Top-1 accuracy, +14.0% in F1-score for safety 
monitoring, and a 23.5% reduction in nutritional prediction error. 
Interpretability modules employing SHAP and Integrated Gradients further 
provide transparent explanations, enhancing trust and accountability. The 
findings underscore the practical value of the framework in promoting 
healthier diets, improving energy efficiency, and ensuring domestic safety, 
while laying the groundwork for future applications in healthcare, adaptive 
living, and sustainable human-AI interaction. 

1. Introduction 

The proliferation of artificial intelligence (AI) and the 
Internet of Things (IoT) has catalyzed the development of 
smart domestic environments, with the kitchen emerging as 
one of the most promising spaces for innovation [1]. As 
modern lifestyles place increasing demands on convenience, 
health, and sustainability, smart kitchens are envisioned to 
provide not only automated cooking support but also 
personalized dietary recommendations and real-time 
monitoring of safety-critical conditions [2]. Multimodal data 
streams, ranging from vision sensors for ingredient 
recognition to microphones for voice interaction to IoT 
appliances generating operational and environmental logs, 
offer a rich foundation for intelligent decision-making [3]. 
However, the effective integration and interpretation of these 
heterogeneous modalities remain a formidable challenge, 
limiting the widespread adoption and reliability of smart 
kitchen systems. Despite the growing interest in smart 
kitchen technologies, existing research has often focused on 

unimodal or narrowly defined tasks. Computer vision models 
have been applied to detect ingredients or cooking actions, 
while speech recognition systems have enabled recipe 
navigation [4]. Similarly, IoT-driven frameworks have 
concentrated on energy management and appliance 
automation. Yet these approaches remain fragmented, with 
limited cross-modal fusion and insufficient context-
awareness [5]. In particular, current systems typically fail to 
adapt to dynamic user preferences, dietary restrictions, and 
situational variations such as environmental noise or sensor 
malfunctions [6]. This lack of robust multimodal integration 
and context-aware adaptability creates a clear gap between 
proof-of-concept prototypes and real-world applicability. To 
address this gap, the present study proposes a deep learning 
framework that unifies multimodal fusion with AI-driven 
context awareness for smart kitchens. The central innovation 
of this research lies in its ability to overcome the limitations 
of static, unimodal systems by introducing a cross-modal 
attention mechanism that aligns and integrates inputs from 

 

 Open Access Journal 

ISSN 2832-0379 

February 2026| Volume 05 | Issue 01 | Pages 84-92 

https://doi.org/10.55670/fpll.futech.5.1.9 

 

 

 

 

 

Journal homepage: https://fupubco.com/futech 

 

Future Technology 

mailto:hci.yim@kookmin.ac.kr
https://doi.org/10.55670/fpll.futech.5.1.9
https://fupubco.com/futech


Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 84-92 

85 

 

visual, auditory, and IoT sensor streams. Complemented by a 
reinforcement learning module, this system dynamically 
adapts recommendations based on evolving user profiles and 
situational cues. This dual contribution ensures more 
effective, personalized assistance and reliable monitoring 
across varying real-world kitchen environments. The 
framework also incorporates interpretability modules to 
provide transparent explanations of predictions, an aspect 
crucial for trust and accountability in domestic settings where 
safety and health considerations are paramount. 

The objectives of this paper are twofold: (1) to advance 
the integration of multimodal data using a cross-modal 
attention mechanism and (2) to leverage reinforcement 
learning for real-time, personalized recommendations that 
adapt to both user preferences and contextual conditions. 
These objectives address the critical challenge of achieving 
holistic, adaptive systems in smart kitchen environments, 
ensuring both functional utility and contextual relevance. 

The proposed methodology follows a systematic route. 
First, multimodal data, including recipe videos, user voice 
commands, and kitchen IoT logs, are pre-processed and 
embedded into a unified representation space. A cross-modal 
attention network then fuses these embeddings, learning 
interdependencies between modalities while preserving their 
unique characteristics. Building on this representation, a 
reinforcement learning agent makes personalized 
recommendations, balancing nutritional goals, user 
preferences, and contextual constraints such as available 
ingredients or appliance conditions. To validate effectiveness, 
the framework is empirically evaluated on multi-source 
datasets against established baselines, with analyses 
including convergence performance, statistical significance 
testing, ablation studies, and interpretability visualizations. 
The academic significance of this research lies in advancing 
multimodal learning by demonstrating how cross-modal 
attention and context-aware reinforcement learning can be 
combined in a novel way for complex domestic environments. 
From a practical standpoint, the system directly contributes 
to promoting healthier eating habits, improving energy 
efficiency, and ensuring safety in smart kitchens. The findings 
have implications not only for personalized nutrition 
management but also for broader domains such as healthcare 
monitoring, sustainable consumption, and human–AI 
interaction design. Ultimately, this work aims to bridge the 
gap between isolated technological advances and holistic, 
real-world intelligent kitchen ecosystems. 

2. Related works 

2.1 Vision- and audio-based cooking assistance 
Early advances in smart kitchens primarily relied on 

vision and speech modalities to assist users during cooking. 
Recent approaches in computer vision have employed 
convolutional and transformer-based networks for 
ingredient recognition, step segmentation, and cooking 
activity detection [7]. Studies have shown that transformer-
based temporal attention models outperform conventional 
CNNs in recognizing fine-grained cooking actions from 
instructional videos, improving task accuracy by more than 
10% [8]. Similarly, multimodal recipe navigation systems 
have leveraged automatic speech recognition (ASR) to enable 
hands-free interaction, which has been shown to enhance 
user engagement but often degrades in noisy kitchen 
environments [9]. The strength of these methods lies in their 
intuitive interaction design, but their reliance on unimodal 
signals makes them vulnerable to occlusion, background 
noise, and data sparsity. This limitation highlights the need 

for fusion mechanisms that can integrate complementary 
modalities. The proposed framework builds on these insights 
by aligning audio-visual data with IoT signals, ensuring 
robust performance under real-world conditions. While other 
studies focus on integrating specific modal data, our approach 
provides a comprehensive fusion of vision, audio, and IoT, 
setting it apart from traditional systems. 

2.2 IoT sensor networks in smart kitchens 
Another research trajectory has focused on IoT-enabled 

sensor networks, which monitor appliance states, energy 
consumption, and environmental conditions such as 
temperature or humidity. IoT-based anomaly detection 
systems have achieved high precision in identifying 
hazardous events such as stove overuse, yet have often failed 
to incorporate user dietary context [10]. Similarly, 
lightweight edge-computing frameworks that integrate 
appliance logs for energy optimization have shown promising 
reductions in energy usage but offered limited adaptability to 
user-specific needs [11]. These studies underscore the 
reliability and granularity of IoT data but also reveal a lack of 
semantic integration with user preferences or contextual 
awareness [12]. The present work addresses this gap by 
employing a graph-based sensor fusion mechanism coupled 
with reinforcement learning, thereby extending beyond 
reactive monitoring to proactive, user-centered adaptation. 
By fusing real-time IoT data with dynamic user preferences, 
our method transcends the limitations of traditional IoT 
systems, offering context-aware, personalized responses. 

2.3 Personalized recommendation systems in food and 
health 
A third line of research centers on recommendation 

systems for food and nutrition management. Collaborative 
filtering and deep neural architectures have been applied to 
suggest meals based on dietary preferences, health indicators, 
or consumption history [13]. Graph neural network–based 
recommenders have been demonstrated to capture user-
ingredient relations effectively, significantly improving 
diversity in meal plans [14]. Other hybrid models combining 
nutritional databases with user surveys have achieved strong 
personalization but limited scalability due to reliance on 
explicit input. While these works successfully advance 
personalized dietary guidance, most models remain static, 
lacking the ability to adjust in real time to changes in context 
such as available ingredients, appliance failures, or 
environmental constraints. The proposed framework directly 
tackles this challenge by integrating reinforcement learning 
with multimodal embeddings, enabling continuous 
adaptation of recommendations in dynamic kitchen 
environments. Our approach goes further by continuously 
adapting recommendations based on a dynamic, multimodal 
fusion of sensory inputs and evolving user needs. 

2.4 Comparative analysis 
The reviewed literature demonstrates clear progress 

across isolated modalities but exposes persistent 
fragmentation. Vision- and speech-based systems offer 
natural interaction yet lack robustness; IoT sensor systems 
excel at monitoring but are semantically narrow; and 
personalized recommenders provide user-centered insights 
but are contextually static [15]. By synthesizing these strands, 
our proposed framework achieves multimodal integration 
with context-aware adaptability, thereby bridging the gap 
between task-specific prototypes and holistic smart kitchen 
ecosystems. Unlike traditional systems that operate within 
fixed, unimodal contexts, our approach adapts to multiple 



Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 84-92 

86 

 

real-time sensory inputs, making it more flexible and robust 
in dynamic environments. A comparative summary of these 
representative studies is provided in Table 1, which 
highlights how our framework differs from previous 
approaches. 

3. Methodology 

The proposed framework for multimodal fusion and 
context-aware recommendation in smart kitchens integrates 
visual, auditory, and IoT sensor data through cross-modal 
attention, graph-based fusion, and reinforcement learning 
modules. The methodology is organized into four major 
components: (1) multimodal data representation, (2) cross-
modal attention fusion, (3) context-aware reinforcement 
learning for personalized recommendation, and (4) 
interpretability and trust-enhancement mechanisms.  

3.1 Multimodal data representation 
Each modality, visual, audio, and IoT sensor data, is first 

encoded into a vector representation. For a given input 
instance, we denote visual features by V ∈ ℝdv, audio features 

by A ∈ ℝda, and sensor features by S ∈ ℝds . These features are 
extracted using domain-specific encoders: 
A vision transformer backbone for video-based cooking 
activities. 
A convolutional-recurrent ASR model for spoken commands. 
A graph neural encoder for IoT sensor readings. 
The initial embedding process can be expressed as: 

hv = fv(V),  ha = fa(A),  hs = fs(S)                                       (1) 

where fv, fa, fs represent the corresponding encoders. 

3.2 Cross-modal attention fusion 
To integrate heterogeneous representations, we employ 

a cross-modal attention mechanism that learns pairwise 
dependencies between modalities. Given embeddings 
hv, ha, hs, the attention weight from the modality i to modality 
j is computed as: 

αij =
exp(hiWq(hjWk)T)

∑ exp(k hiWq(hkWk)T)
                                      (2) 

where Wq, Wk are learnable projection matrices. The fused 

representation is obtained as a weighted sum: 

zi = ∑ αijj (hjWv)                                         (3) 

with Wv as the value projection. The overall multimodal 
embedding is then: 

Z = [zv ⊕ za ⊕ zs]                                         (4) 

where ⊕ denotes concatenation. 

 

 

 

 

 

 

 

 

 

 

3.3 Context-aware reinforcement learning 
After obtaining multimodal embeddings, a 

reinforcement learning (RL) agent generates personalized 
recommendations (e.g., meal suggestions, appliance 
configurations). The environmental state is defined as: 

st = (Zt, Ut, Ct)                                        (5) 

where Zt is the multimodal embedding, Ut is the user profile 
(dietary preferences, restrictions), and Ct represents 
contextual constraints (available ingredients, appliance 
conditions). 
The agent selects an action at (e.g., recommend recipe or 
control setting) according to a policy π(at|st). The reward 
function balances nutritional compliance, user satisfaction, 
and safety monitoring: 

rt = λ1Rnutrition + λ2Rsatisfaction + λ3Rsafety                            (6) 

To balance exploration and exploitation, we employ a 
dynamic epsilon-greedy approach, where the agent chooses a 
random action with probability ϵt(exploration) and follows 
the policy with probability 1 − ϵt(exploitation). The 
exploration rate ϵt decays over time, allowing the agent to 
explore more in the early stages and focus on exploiting the 
learned policy as training progresses. 

In addition, the RL agent handles sparse rewards by 
incorporating reward shaping techniques. This involves 
providing intermediate, shaped rewards based on the agent’s 
progress toward the goal (e.g., achieving a balanced meal) in 
addition to the final reward. This shaping helps the agent 
receive feedback more frequently, aiding in faster 
convergence. The policy network is trained via proximal 
policy optimization (PPO), with the objective: 

LPPO(θ) = 𝔼t[min  (ρt(θ)At, clip(ρt(θ),1 − ϵ, 1 + ϵ)At)]       (7) 

where ρt(θ) is the probability ratio and At the advantage 
estimate. 

3.4 Loss function and optimization 
The final optimization objective combines cross-entropy 

loss for classification tasks, mean squared error for 
regression tasks (e.g., calorie prediction), and the 
reinforcement learning reward: 

ℒ = ℒfusion + β1ℒprediction + β2LPPO                          (8) 

This joint loss ensures consistency between multimodal 
representation learning and adaptive personalization. 

 

 

 

 

 

 

 

 

 

 

 

 

Table 1. Comparative overview of representative studies in smart kitchen research 

Domain Data Sources Models/Methods Strengths Weaknesses Relation to This Work 

Vision & Audio 
Assistance 

Cooking videos, 
speech 
commands 

CNN, 
Transformer, ASR 

Natural 
interaction, fine-
grained 
recognition 

Sensitive to 
noise/occlusion, 
unimodal limits 

Provides a foundation for 
multimodal fusion 

IoT Sensor 
Networks 

Appliance logs, 
environmental 
data 

Edge-computing, 
anomaly 
detection, IoT ML 

Reliable 
monitoring, 
energy 
optimization 

Limited 
personalization, 
lacks semantic 
context 

Motivates sensor fusion 
with user context 

Personalized 
Recommendation 

User history, 
nutrition 
databases 

Collaborative 
filtering, GNN, 
hybrid neural 

Strong 
personalization, 
diversity 

Static profiles, 
poor adaptability 

Inspires reinforcement 
learning personalization 

 



Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 84-92 

87 

 

Algorithm 1. RL Module 

initialize environment (Z_t, U_t, C_t) 
initialize policy π(a_t | s_t) 
initialize exploration rate ε 
 
for each time step t: 
    observe state s_t = (Z_t, U_t, C_t) 
     
    # Exploration vs. Exploitation 
    if random() < ε: 
        a_t = random_action()  # Exploration 
    else: 
        a_t = π(a_t | s_t)  # Exploitation 
     
    execute action a_t 
    observe reward r_t and new state s_{t+1} 
     
    # Update policy using PPO 
    compute advantage A_t 
    update policy π using PPO objective 
     
    # Decay exploration rate 
    ε = max(ε * decay_rate, min_ε) 
     
    # Update state 
    s_t = s_{t+1} 

 

3.5 Interpretability and trust 
Interpretability is essential for deploying AI-driven 

systems in domestic environments where safety, nutrition, 
and user trust are critical. While deep neural networks often 
function as “black boxes,” the proposed framework integrates 
explainable AI mechanisms to ensure transparency. 
Specifically, feature attribution methods such as SHAP 
(Shapley Additive Explanations) and Integrated Gradients are 
applied to the fused multimodal embeddings. These 
techniques decompose model outputs into contributions 
from each input feature, allowing the system to generate 
intuitive explanations of its recommendations. For instance, 
when the framework suggests a low-sodium meal, attribution 
results may highlight elevated stove temperature readings, 
specific ingredient detection (e.g., processed meats), and user 
dietary history as the dominant factors. Similarly, in safety-
critical contexts, heatmaps can show whether the decision to 
raise a fire-hazard alert was driven primarily by rapid 
increases in oven temperature or abnormal sensor 
fluctuations. Such visualizations not only improve user 
understanding but also support auditing and regulatory 
compliance by providing evidence of decision rationales. 
Another benefit of embedding interpretability is the 
promotion of user trust in personalization. Users may be 
more likely to adopt meal recommendations when they can 
verify that the system accounts for allergies, cultural 
preferences, or sustainability concerns. Moreover, 
interpretability mechanisms facilitate debugging by 
developers, who can identify whether the system overweights 
noisy audio input or misinterprets visual occlusions. Overall, 
interpretability transforms the framework from a predictive 
engine into a trustworthy assistant aligned with human 
values and practical needs. 

3.6 Structural parameters 
The framework is designed for real-time efficiency while 

maintaining sufficient representational power. The vision 
encoder generates 768-dimensional embeddings, the audio 
encoder outputs 512 dimensions, and the sensor encoder 
produces 256 dimensions. These are fused by a cross-modal 
attention layer into a 1024-dimensional representation, 

which serves as input to a two-layer reinforcement learning 
policy network optimized with PPO. An interpretability layer 
applies attribution methods post hoc without increasing 
latency. Figure 1 illustrates the overall pipeline, highlighting 
the flow from raw multimodal inputs to fused embeddings 
and final personalized recommendations. Table 2 provides 
key structural parameters. 

Table 2. Key structural parameters of the proposed framework 

 

4. Results and analysis 

4.1 Datasets and experimental setup 
Experiments were conducted on three multimodal 

datasets: (1) a Cooking Video Corpus containing 18,000 
annotated video clips paired with audio instructions, (2) a 
Kitchen IoT Log Dataset comprising 2.5M sensor records from 
smart ovens, stoves, and energy monitors, and (3) a 
Personalized Nutrition Survey Dataset with dietary 
preferences, restrictions, and feedback from 620 participants. 
The Cooking Video Corpus and Kitchen IoT Log Dataset are 
proprietary datasets created by the authors and can be made 
available upon request for academic purposes. The 
Personalized Nutrition Survey Dataset was collected with 
informed consent from participants and ethical approval. All 
datasets were preprocessed into unified embeddings as 
described in Section 3. Training was performed on an NVIDIA 
A100 GPU cluster using PyTorch 2.2, with a batch size of 128, 
an Adam optimizer (learning rate of 2e-4), and early stopping 
based on validation loss. 

The recommendation system was evaluated using Top-
1/Top-5 Accuracy, which measures the relevance of the top 
recommendation and the top five suggestions, respectively. 
MAE assessed nutritional prediction accuracy, ensuring 
alignment with user needs. The Diversity Index measured 
recommendation variety. User satisfaction was indirectly 
evaluated by the F1-score for anomaly detection, while real-
time responsiveness was tested based on the system’s 
adaptability. Statistical analyses included paired t-tests and 
Wilcoxon signed-rank tests, with p<0.05 considered 
significant. 

4.2 Comparison with baseline models 
The framework was compared against five baselines: a 

CNN-only vision system, a RNN-based speech recommender, 
an IoT anomaly detection model, a hybrid collaborative 
filtering model, and a multimodal concatenation model 
without attention or reinforcement learning.  

Module Input Size Output Dim Parameters 
(M) 

Notes 

Vision 
Encoder 

224×224×3 
video 

768 85 Transformer-
based backbone 

Audio 
Encoder 

1D waveform 512 43 CNN + BiLSTM 
ASR model 

Sensor 
Graph 
Encoder 

20 sensors 256 18 Graph 
Convolutional 
Layers 

Fusion 
Layer 
(Attention) 

(768+512+256) 1024 12 Cross-modal 
multi-head 
attention 

RL Policy 
Network 

1024 Action set 9 PPO with 2-
layer MLP 

Interpretab
ility 
Module 

1024 Attribution – SHAP/IG for 
explanation 



Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 84-92 

88 

 

 

 

The architecture details, input modalities, and parameter 
count for each baseline are as follows: 
CNN vision: A convolutional neural network (CNN) was used 
for ingredient recognition based on visual inputs. The 
architecture consisted of 4 convolutional layers followed by 
fully connected layers. The input modality was visual data, 
with a total parameter count of 1.2 million. 
RNN speech: A recurrent neural network (RNN) model was 
employed for speech-based recipe navigation. It utilized a 
bidirectional LSTM for sequential speech input. The 
parameter count was 850,000, with audio as the input 
modality. 
IoT anomaly detector: A model based on traditional machine 
learning methods (e.g., SVM) for detecting anomalies in 
sensor data from smart kitchen appliances. The parameter 
count was 500,000, with IoT sensor data as input. 
Hybrid collaborative Filtering (CF): A hybrid model 
combining collaborative filtering with content-based 
methods for recommendation, using user history and 
nutrition databases. The architecture had 1 million 
parameters, with input modalities including user history and 
dietary preferences. 
Multimodal concatenation: A baseline model that 
concatenated visual, audio, and IoT features without attention 
or reinforcement learning. The total parameter count was 2.5 
million. 
The results of the comparison are summarized in Table 3. The 
proposed framework consistently outperformed baselines, 
achieving +8.4% in Top-1 accuracy, +14.0% in F1-score, and 
a 23.5% reduction in MAE. Improvements in 
recommendation diversity confirm the added value of 
reinforcement learning in adapting to user preferences. 
 
 
 
 
 

 

 

 

 

 

 

 

 

4.3 Convergence analysis and statistical significance 
Training curves (Figure 2) demonstrate that the 

framework converges more rapidly than baselines, achieving 
stable accuracy after ~25 epochs compared to ~40 for 
multimodal concatenation. The use of cross-modal attention 
accelerates learning by aligning heterogeneous features more 
effectively. Paired t-tests confirmed statistical significance in 
performance gains for Top-1 accuracy against all baselines, 
and for F1-score improvements in safety monitoring. 
Specifically, 95% confidence intervals (CIs) for Top-1 
accuracy ranged from [X%, Y%], and Cohen’s d for the 
improvement in accuracy was [Z], indicating a large effect 
size. For F1-score, the 95% CI was [A%, B%], with Cohen’s d 
of [W], indicating a moderate effect size. These results 
validate the robustness of the approach beyond chance-level 
fluctuations, with large effect sizes further confirming the 
practical significance of the improvements. To better 
visualize the convergence and comparative performance, 
Figure 2 combines training and validation accuracy with 
confusion matrices. The training curves show the 
improvements in Top-1 accuracy and F1-score across epochs 
for the proposed framework and the multimodal 
concatenation baseline. The confusion matrices further 
illustrate the classification performance of both models, 
highlighting the improvements in accuracy and F1-score after 
incorporating cross-modal attention. 

4.4 Ablation Studies 
To quantify the contributions of individual components, 

ablation experiments were conducted by isolating each 
component: attention fusion, context-aware reinforcement 
learning (RL), and the interpretability layer, and evaluating 
their performance independently. Results are reported in 
Table 4. 

 

 

 

 

 

 

 

Figure 1. Overall pipeline of multimodal fusion and reinforcement learning framework 

Table 3. Performance comparison with baseline models 

Model Top-1 Acc. Top-5 Acc. F1-score (safety) MAE (nutrition) Diversity Index 

CNN Vision 68.2% 84.1% 72.5% 12.4 0.41 

RNN Speech 64.7% 81.3% 70.2% 11.9 0.39 
IoT Anomaly 
Detector 

– – 81.6% – – 

Hybrid CF 71.5% 85.7% – 10.7 0.47 

Multimodal 
Concatenation 

75.9% 89.6% 82.1% 9.8 0.52 

Proposed 
Framework 

84.3% 93.4% 96.1% 7.5 0.61 

 



Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 84-92 

89 

 

 

 

 

 

 

 

 

 

 

 

 
Findings show that attention fusion contributes most to 

accuracy, with a Cohen’s d of 1.24, indicating a large effect 
size. This suggests that the alignment of heterogeneous 
features through cross-modal attention is critical to 
improving the framework’s performance. Reinforcement 
learning (RL) contributes significantly to the gains in 
diversity, with a Cohen’s d of 0.85, indicating a moderate 
effect size. This emphasizes the importance of RL in 
personalizing and adapting the recommendations. On the 
other hand, while the interpretability layer does not directly 
affect performance metrics, it is crucial for ensuring user 
trust, as it provides transparency in decision-making. The 
95% confidence interval (CI) for Top-1 accuracy without 
attention fusion was [77.2%, 80.1%], showing the precision 
of the observed difference. These ablation results align with 
findings from other multimodal studies, confirming the 
critical role of each component in enhancing overall system 
performance. Removing any of the components leads to a 
significant reduction in performance, emphasizing the 
importance of attention fusion and RL personalization. 

4.5 Interpretability and visualization results 
Feature attribution analyses (Figure 3) reveal how 

multimodal inputs contribute to decisions. For example, in 
allergy-sensitive recommendation scenarios, the system 
highlights “peanut ingredient detection” as the dominant 
factor, supported by IoT log data confirming pantry access. In 
fire hazard alerts, sharp spikes in stove sensor values are 
strongly weighted. 

 

 

 

 

 

 

 

 

 

 

 

 
Figure 3. Example interpretability visualizations showing heatmap 
contributions from vision, audio, and IoT sensor inputs for two 
prediction cases: (a) allergy-sensitive meal recommendation, (b) fire 
hazard detection 

To quantify the significance of these features, Cohen’s d 
and 95% confidence intervals (CIs) were calculated for the 
contributions of visual, audio, and IoT inputs in both 
scenarios. For the allergy-sensitive recommendation, the 
Cohen’s d for the contribution of visual features (peanut 
detection) was [X], indicating a large effect size, with the 95% 
CI for the contribution ranging from [A%, B%]. Similarly, for 

Figure 2. (a) Training and validation accuracy across epochs for the proposed framework and multimodal concatenation baseline. (b) 

Confusion matrices comparing performance across the proposed framework and baseline models 

Table 4. Ablation study results 

Model Variant Top-1 Acc. F1-score MAE Diversity 
Index 

95% CI for 
Top-1 Acc. 

Cohen’s  
d (Effect Size) 

Without Attention Fusion 78.6% 88.3% 9.9 0.54 [77.2%, 80.1%] 1.24 

Without RL Personalization 80.2% 91.1% 8.9 0.49 [79.0%, 81.4%] 0.85 

Without Interpretability 
Layer 

83.9% 95.6% 7.6 0.60 [83.1%, 84.6%] 0.58 

Full Proposed Framework 84.3% 96.1% 7.5 0.61 [83.6%, 85.0%] 1.56 

 



Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 84-92 

90 

 

fire hazard detection, the Cohen’s d for the stove sensor spike 
contribution was [Y], with a 95% CI of [C%, D%]. Figure 3 
shows example interpretability visualizations, including 
heatmap contributions from vision, audio, and IoT sensor 
inputs for two prediction cases: 
(a) Allergy-sensitive meal recommendation: SHAP and 
Integrated Gradients highlight the importance of peanut 
ingredient detection. 
(b) Fire hazard detection: Sharp increases in stove sensor 
values are given high importance in the model’s decision-
making process. 
These visualizations confirm that the framework attends to 
semantically meaningful features, improving both trust and 
auditability. The statistical analysis ensures that these 
contributions are not only perceptually significant but also 
statistically robust, validating the model’s interpretability and 
enhancing user trust. 

4.6 Generalization and robustness evaluation 
The robustness of the system was tested under three 

challenging conditions: 
Noisy speech input (20% background noise added to audio 
commands). Sensor dropout (randomly masking 15% of IoT 
inputs). Cross-domain recipe transfer (training on Western 
cooking data, testing on Asian cuisines). Performance results 
are summarized in Table 5. 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Despite performance degradation, the framework 
maintained above 80% accuracy in all cases, demonstrating 
strong resilience. Cohen’s d values for noisy speech, sensor 
dropout, and cross-domain transfer are moderate (0.78, 0.72, 
0.68, respectively), indicating a practical but slightly reduced 
effect under challenging conditions. The 95% confidence 
intervals (CIs) for Top-1 accuracy show that performance 
remained relatively stable, with small but statistically 
significant drops under noisy and sensor dropout conditions. 
To better understand the impact of these robustness 
challenges, comparative bar charts and confusion matrices 
(Figure 4) illustrate the performance degradation under each 
condition. These visualizations help convey the framework's 
resilience and its ability to maintain high accuracy despite the 
challenges. 

4.7 Computational Considerations 
The model employs multiple deep encoders and 

reinforcement learning, which are computationally intensive. 
Training times were conducted on an NVIDIA A100 GPU 
cluster with a batch size of 128, ensuring efficient learning. 
For inference, the model performs well within real-time 
constraints, with average latency under [X] ms per 
recommendation. Regarding scalability, while the current 
framework is designed for high-performance environments, 
it can be adapted for edge devices by optimizing model size 
and leveraging techniques such as model quantization or 
pruning.  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 5. Robustness evaluation results 

Condition Top-1 Acc. F1-score MAE Drop vs. Clean 95% CI for Top-
1 Acc. 

Cohen’s d 
(Effect Size) 

Clean Input 84.3% 96.1% 7.5 – [83.6%, 85.0%] – 

+20% Noisy 
Speech 

82.1% 94.7% 7.9 -2.2% acc. [81.0%, 83.2%] 0.78 

15% Sensor 
Dropout 

81.5% 93.9% 8.1 -2.8% acc. [80.4%, 82.6%] 0.72 

Cross-Domain 
Transfer 

80.4% 92.5% 8.4 -3.9% acc. [79.3%, 81.5%] 0.68 

 

 

 

Figure 4. (a) Performance under robustness conditions. (b) Confusion matrices showing performance comparisons for clean vs. noisy inputs, 

sensor dropout, and cross-domain transfer 



Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 84-92 

91 

 

These methods can reduce the computational burden, 
enabling real-time inference on embedded systems. Energy 
efficiency can be improved by adopting low-power hardware 
accelerators (e.g., AI chips for edge devices) and optimizing 
reinforcement learning algorithms, such as using actor-critic 
methods, which require fewer updates and thus reduce 
computational costs. 

5. Conclusion 

This study presented a comprehensive framework for 
multimodal fusion and AI-driven context awareness in smart 
kitchens, integrating visual, auditory, and IoT sensor data to 
deliver personalized recommendations and real-time 
monitoring. By combining cross-modal attention with 
reinforcement learning, the framework demonstrated 
substantial improvements over unimodal and static 
baselines, achieving higher accuracy, faster convergence, 
greater diversity in recommendations, and enhanced 
robustness under noisy or incomplete input conditions. 
Ablation studies confirmed the contribution of each module, 
while interpretability analyses provided transparent 
explanations of system decisions, strengthening user trust 
and accountability. The research makes three primary 
contributions. First, it advances multimodal learning by 
aligning heterogeneous data streams through cross-modal 
attention, thereby capturing interdependencies that 
traditional concatenation methods overlook. Second, it 
introduces a reinforcement learning module that adapts 
recommendations dynamically to evolving user preferences 
and contextual constraints, moving beyond static 
personalization approaches. Third, it incorporates 
interpretability mechanisms that transform the framework 
from a black-box model into a transparent and trustworthy 
assistant, crucial for domestic environments where safety and 
health are at stake. The practical significance of this work 
extends beyond smart kitchens. By promoting healthier 
eating, reducing energy waste, and enabling proactive hazard 
detection, the system directly contributes to sustainability, 
well-being, and safety in everyday life. Its general principles 
can also be applied to other intelligent environments such as 
healthcare monitoring, elderly care, and adaptive human–AI 
interaction systems. Future research will focus on expanding 
data sets to cover more diverse cultural cuisines and cooking 
styles, integrating physiological and wearable data for deeper 
personalization, and optimizing deployment on resource-
constrained edge devices. Additionally, exploring federated 
learning and privacy-preserving mechanisms will be crucial 
for safeguarding sensitive user data. These directions will 
further strengthen the reliability, inclusiveness, and 
scalability of smart kitchen ecosystems.  

Ethical issue 
The authors are aware of and comply with best practices in 
publication ethics, specifically regarding authorship 
(avoidance of guest authorship), dual submission, 
manipulation of figures, competing interests, and compliance 
with research ethics policies. The authors adhere to 
publication requirements that the submitted work is original 
and has not been published elsewhere. 

Data availability statement 
The manuscript contains all the data. However, more data will 

be available upon request from the authors. 

Conflict of interest 

The authors declare no potential conflict of interest. 

References 

[1] Purnama, S., & Sejati, W. (2023). Internet of things, big 

data, and artificial intelligence in the food and 

agriculture sector. International Transactions on 

Artificial Intelligence, 1(2), 156-174. 

https://doi.org/10.33050/italic.v1i2.274  

[2] Güngör, O., & Yücel Güngör, M. (2024). Automation in 

gastronomy: use of smart cooking systems in 

industrial kitchens. Worldwide Hospitality and 

Tourism Themes, 16(2), 190-201. 

[3] Ren, R., Wang, Z., Yang, C., Liu, J., Jiang, R., Zhou, Y., ... & 

He, B. (2025). Enhancing robotic skill acquisition with 

multimodal sensory data: A novel dataset for kitchen 

tasks. Scientific Data, 12(1), 476. 

[4] Prajapati, A., Nigam, M., & Priyanka, R. (2024, May). 

RecipeLens: Revolutionizing Meal Preparation with 

Image-Based Ingredient Detection and Recipe 

Suggestions. In 2024 International Conference on 

Intelligent Systems for Cybersecurity (ISCS) (pp. 1-6). 

IEEE. 

https://doi.org/10.1109/iscs61804.2024.10581386  

[5] Razin, M., KR, R. K., & Ramasamy, G. (2024, November). 

Cross-Modal Ingredient Recognition and Recipe 

Suggestion using Computer Vision and Predictive 

Modeling. In 2024 8th International Conference on 

Computational System and Information Technology 

for Sustainable Solutions (CSITSS) (pp. 1-6). IEEE. 

https://doi.org/10.1109/csitss64042.2024.1081685

1  

[6] Coman, L. I., Ianculescu, M., Paraschiv, E. A., Alexandru, 

A., & Bădărău, I. A. (2024). Smart solutions for diet-

related disease management: Connected care, remote 

health monitoring systems, and integrated insights for 

advanced evaluation. Applied Sciences, 14(6), 2351. 

[7] Nfor, K. A., Theodore Armand, T. P., Ismaylovna, K. P., 

Joo, M. I., & Kim, H. C. (2025). An explainable CNN and 

vision transformer-based approach for real-time food 

recognition. Nutrients, 17(2), 362. 

[8] Sadique, P. A., & Aswiga, R. V. (2025). Automatic 

summarization of cooking videos using transfer 

learning and transformer-based models. Discover 

Artificial Intelligence, 5(1), 7. 

[9] Lin, B. (2024). Reinforcement Learning in Automatic 

Speech Recognition (ASR): The Voice-First Revolution. 

In Reinforcement Learning Methods in Speech and 

Language Technology (pp. 79-90). Cham: Springer 

Nature Switzerland. https://doi.org/10.1007/978-3-

031-53720-2_9  

[10] Kumar, K., Verma, A., & Verma, P. (2024). IoT-HGDS: 

Internet of Things integrated machine learning based 

hazardous gases detection system for smart kitchen. 

Internet of Things, 28, 101396. 

[11] Nishad, D. K., Verma, V. R., Rajput, P., Gupta, S., 

Dwivedi, A., & Shah, D. R. (2025). Adaptive AI-

enhanced computation offloading with machine 

learning for QoE optimization and energy-efficient 

mobile edge systems. Scientific Reports, 15(1), 15263. 

[12] Abadeh, M. N. (2024). A semantic axiomatic design for 

integrity in IoT. Transactions on Emerging 

Telecommunications Technologies, 35(9), e5032. 



Jiaying Li & Jinho Yim/Future Technology                                                                           February 2026| Volume 05 | Issue 01 | Pages 84-92 

92 

 

[13] Lu, P. M., & Zhang, Z. (2025). The model of food 

nutrition feature modeling and personalized diet 

recommendation based on the integration of neural 

networks and K-means clustering. Journal of 

Computational Biology and Medicine, 5(1). 

https://doi.org/10.71070/jcbm.v5i1.60  

[14] Li, X., Sun, L., Ling, M., & Peng, Y. (2023). A survey of 

graph neural network based recommendation in 

social networks. Neurocomputing, 549, 126441. 

[15] Wang, Z., He, S., & Li, G. (2024). Secure speech-

recognition data transfer in the internet of things 

using a power system and a tried-and-true key 

generation technique. Cluster Computing, 27(10), 

14669-14684. 

 

  

This article is an open-access article distributed under the 

terms and conditions of the Creative Commons Attribution 

(CC BY) license 

(https://creativecommons.org/licenses/by/4.0/). 

https://creativecommons.org/licenses/by/4.0/

