Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

696 
https://internationalpubls.com 

Quantum Resistant Cryptosystem-Based Security Protocol for 5G 

Network 

 

Hitesh T Loriya1, Divyesh R Keraliya2, Rahul D Mehta3 
1Electronics and Communication Engineering Department, L E College - Morbi, hitesh_loriya@yahoo.co.in   

2Electronics and Communication Engineering Department, GEC - Rajkot, drkeraliya@gmail.com                     
3Electronics and Communication Engineering Department, GEC - Rajkot, rdmehta@hotmail.com 

 

Article History: 

Received: 03-05-2024 

Revised: 05-07-2024 

Accepted: 25-07-2024 

Abstract 

Security is crucial for wireless communication networks, especially as quantum 

computing advances rapidly. It won’t be long before quantum attacks become feasible, 

potentially crippling large wireless networks within minutes. Current methods for 

securing connections and transactions—such as keys, certificates, and data—could be 

compromised by quantum computers. One concern is the use of fake base stations with 

stronger signal strengths to lure users into connecting with them in wireless 

communication network. A quantum-powered attacker could easily break traditional 

encryption algorithms and launch various attacks almost instantly. An example of this is 

the "bidding down" attack, where an attacker convinces both the user and network entities 

that security features are not upheld, despite their actual presence. This type of attack 

exploits the security weaknesses of older mobile networks. This paper explores security 

advancements and challenges related to contemporary public key cryptography, including 

RSA algorithms based on factorization and discrete logarithm problems with Diffie-

Hellman and Elliptic-Curve Cryptography (ECC). While these methods provide adequate 

protection today, quantum computers could potentially break these algorithms or weaken 

cryptographic keys and hashes within minutes. Quantum cryptography, or quantum-

resistant cryptography, objectives to develop algorithms and protocols resilient to 

quantum computing threats. This paper examines security vulnerabilities in wireless 

communication networks, focusing on key confirmation and authentication mechanisms. 

We propose a robust authentication and key agreement protocol for 5G networks using 

quantum-resistant cryptography. Our proposed protocol, verified with a verification tool, 

enhances the security of the authentication and key agreement procedures in wireless 

communication networks.  

Keywords: Security, Quantum attack, Cryptosystem, NTRU 

1. Introduction 

Cryptography involves the study of systems designed to securely transfer information among users, 

combining principles from engineering and mathematics. Its primary applications include ensuring 

authentication, confidentiality, key distribution and integrity. Two main types of cryptosystems used 

for secure information transfer are private key cryptosystems and public key cryptosystems. In the 

context of 5G networks, the 3GPP defines a security architecture [1] that employs cryptosystems to 

protect subscriber information exchanged between network entities. Quantum computing, which 

utilizes quantum bits or qubits, represents a significant departure from traditional computing. Unlike 

classical computers, which process information sequentially, quantum computers can handle multiple 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

697 
https://internationalpubls.com 

pieces of information simultaneously, producing numerous possible solutions at once. There are five 

prominent public key cryptosystems: Diffie-Hellman, Elliptic Curve Cryptography (ECC), Elgamal 

Cryptographic System (ECS), RSA, and NTRU. Alese et al. [2] conducted a relative analysis of ECC, 

RSA, Elgamal, and Menezes-Vanstone Elliptic Curve Encryption algorithms, implemented in Java, 

evaluating their performance based on key generation, encryption, and decryption times. Giripunje et 

al. [3] discussed asymmetric key cryptography solutions, emphasizing the importance of secure 

authentication in mobile communications. This paper highlighted that ECC, requiring smaller key sizes 

and lower computational power compared to RSA, is more suitable for constrained devices such as 

mobile phones. Pallipamu et al. [4] performed a security examination of digital signature schemes, 

including DSA, and Elgamal, and RSA, assessing the mathematical complexity of key generation, 

signature verification, and overall security strengths of these algorithms. 

2. Objectives 

The security of information is a critical concern in wireless communication networks. Implementing 

public key algorithms on highly constrained devices, such as mobile phones and PDAs, necessitates 

the use of fast and efficient cryptosystems. It is essential to analyze various public key cryptosystems 

to identify the most efficient option. Among these, the NTRU public key cryptosystem stands out as 

the fastest, offering different levels of security with high performance even on limited resources. 

NTRU employs lattice-based cryptography for data encryption and decryption. Unlike many other 

public key cryptosystems, NTRU is resistant to quantum attacks using Shor's algorithm. 

3. NTRU Cryptosystem 

Here is a brief overview of the well-known RSA cryptosystem. The RSA cryptosystem, developed by 

Ron Rivest, Adi Shamir, and Leonard Adleman, is widely used and incorporated into web browsers 

from Microsoft and Netscape. The process for generating RSA public and private keys is as follows: 

Key Generation Algorithm: 

1. Select Two Large Prime Numbers p and q. 

2. Calculate n=p×q. 

3. Compute ϕ(n)=(p−1)×(q−1). 

4. Select a public key exponent e such that gcd(ϕ(n),e)=1 and 1<e<ϕ(n). 

5. Determine the Private Key d such that d×e mod  ϕ(n)=1 

6. Public key is (n,e) and Private key is (n,d). 

Encryption and Decryption: 

Encryption: To encrypt a message M, compute C = Me mod n 

Decryption: To decrypt the ciphertext C, compute M = Cd mod n = (Me)d mod n = (M)ed mod n which 

retrieves the original message M. This process ensures secure communication by using the public key 

for encryption and the private key for decryption. 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

698 
https://internationalpubls.com 

The NTRU Cryptosystem, developed by J H. Silverman, J Hoffstein, and J Pipher, is named after the 

N-th degree TRUncated polynomial ring [5]. NTRU’s operation defines in the ring ℤ [𝑥] (𝑥𝑛 −  1)⁄  

which is identified as the ring of convolution polynomials of rank “n”, where “n" is a prime. Addition 

can be performed in O(n) and multiplication requires only O(n2) operations in the ring of convolution 

polynomials. NTRU utilizes three primary rings: 

1. R = ℤ [𝑥] (𝑥𝑛 −  1)⁄   

2. Rp = ℤ/𝑝ℤ[𝑥] (𝑥𝑛 −  1)⁄  

3. Rq = ℤ/𝑞ℤ[𝑥] (𝑥𝑛 −  1)⁄  

Here, p (specifying a ring  ℤ 𝑝ℤ⁄ ) and q (specifying a ring ℤ 𝑞ℤ⁄ ) are positive integers used to reduce 

the coefficients of polynomials during encryption and decryption. Polynomials in these rings can be 

represented either by their coefficients or directly in polynomial form. Operations such as addition and 

multiplication in Rp or Rq are similar to those in R, with the result's coefficients reduced modulo p or 

q. 

An element f (A polynomial in ℤ [𝑥] (𝑥𝑛 −  1).⁄ ) of the rings R, Rp and Rq can be represented as 

polynomial or vector coefficients.  

 

NTRU operate in the ring R = ℤ [𝑥] (𝑥𝑛 −  1).⁄  An element  f  ∈ R represented as a polynomial, 

𝑓 = ∑ 𝑓𝑖𝑥𝑖

𝑛−1

𝑖=0

= [𝑓0, 𝑓1,  … , 𝑓𝑛−1]  

f  ∗ 𝑔 = ℎ 

"𝑔" is a polynomial in ℤ [𝑥] (𝑞, 𝑥𝑛 −  1)⁄  used with 𝑓𝑞 (A polynomial in ℤ [𝑥] (𝑞, 𝑥𝑛 −  1)⁄  ). This 

polynomial is obtained by reducing the coefficients of f modulo q. to construct the public key 

“h” is the public key, a polynomial in ℤ [𝑥] (𝑞, 𝑥𝑛 −  1).⁄  

ℎ𝑘  =  ∑ 𝑓𝑖𝑔𝑘−𝑖 +  ∑ 𝑓𝑖𝑔𝑛+𝑘−𝑖 =  ∑ 𝑓𝑖𝑔𝑖.

𝑖+𝑗≡𝑘 mod 𝑛

𝑛−1

𝑖=𝑘+1

𝑘

𝑖=0

 

The variable "k" represents a security parameter that determines the level of protection against specific 

types of attacks, such as plaintext awareness. The public key (h) is computed by performing a cyclic 

convolution between f and g. The output is in the R. To minimize the coefficient, it is necessary to 

calculate the multiplication modulo q. This will ensure that the result is of the form ℤ [X] (𝑞, 𝑥𝑛 −  1).⁄  

The computation of the product f * g necessitates n^2 multiplications. In NTRU, either f or g has 

coefficients that are predominantly 0's and ±1's, allowing for a very efficient computation of f * g. In 

addition, when N is a large amount, one can select n to have a high degree of divisibility by 2. In this 

scenario, the convolution product can be calculated using Fast Fourier Transforms, requiring O (N log 

N) operations. Operations involving generating functions (G) and hashing functions (H) must be 

defined on rings of polynomials. The G and H elements are utilized in the construction of a digital 

envelope within the NTRU protocol. 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

699 
https://internationalpubls.com 

𝑝𝑝(n) = {Polynomials with coefficients modulo p and degree no greater than n−1}, and 

[𝑔]𝑝= {
𝑔 with its cofficients reduced

modulo 𝑝 to the range (− 𝑝 2, 𝑝 2).⁄⁄
 

G function and a H function are exactly defining as  

𝐺 = 𝑝𝑝 (𝑁)  →  𝑝𝑝 (𝑁) and 𝐻 = 𝑝𝑝 (𝑁) ×  𝑝𝑝 (𝑁) →  𝑝𝑝 (𝐾) 

They should be computationally efficient, highly non-linear, and difficult to predict. The digital 

envelope in the NTRU public key cryptosystem relies on the selection of functions G and H, as well 

as an integer k. The likelihood of successfully forging a valid ciphertext is 𝑝−𝑘. 

Key Creation: 

Two small polynomials f  ∈  ℒ𝑓 (Polynomials in ℤ [𝑥] (𝑥𝑛 −  1)⁄  whose coefficients satisfy 𝑑𝑓 ( 

Distribution of the coefficients of the polynomial f (Part of the private key)) and 

𝑔 (Part of public key)  ∈  ℒ𝑔 (The set of polynomials in ℤ [𝑥] (𝑥𝑛 −  1)⁄  whose coefficients satisfy 

𝑑𝑔 (Distribution of the coefficients of the polynomial g)) are randomly generated to produce a pair of 

public and private keys. The polynomial f must fulfil the extra condition of having modular inverses 

both modulo q and modulo p. fp represents a polynomial in the quotient ring ℤ [𝑥] (𝑝, 𝑥𝑛 −  1)⁄ , which 

is a crucial component of the private key. This polynomial is derived by decreasing the coefficients of 

f modulo p. For appropriate parameter selections, this statement holds true for the majority of f choices, 

and the computation of these inverses can be easily performed by modifying the Euclidean method. 

The inverses of 𝑓𝑞
−1 and 𝑓𝑝

−1 are denoted as mentioned above   

𝑓𝑞
−1 ∗ 𝑓 ≡ 1 mod q and 𝑓𝑝

−1 ∗ 𝑓 ≡ 1 mod p. 

f, g, 𝑓𝑞
−1 and 𝑓𝑝

−1are kept secret and public key is calculated and issued as follow 

   ℎ ≡ 𝑝𝑓𝑞
−1 ∗ 𝑔 mod 𝑞. 

Polynomial of f is the private key and Polynomial of h is the public key  

Encryption: 

NTRU cryptosystem chooses plaintext m (polynomial in ℤ [𝑥] (𝑝, 𝑥𝑛 −  1))⁄ . from the set  

𝑚 ∈  𝑝𝑝 (𝑛 − 𝑘). 

and a polynomial in ℤ [𝑥] (𝑞, 𝑥𝑛 −  1)⁄  (used with h to encode a message) which is random polynomial 

𝑟 ∈  ℒ𝑟 (Polynomials in ℤ [𝑥] (𝑥𝑛 −  1)⁄  whose coefficients satisfy 𝑑𝑟 (Number of 1s and -1s used in 

a certain random polynomial r)) Cipher text (The encrypted message, a polynomial in 

ℤ [𝑥] (𝑞, 𝑥𝑛 −  1)⁄ ) is computed as follow 

𝑒 ≡ 𝑟 ∗ ℎ + [𝑚 + 𝐻 (𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘 +  𝐺([𝑟 ∗ ℎ]𝑝)]
𝑝
  (mod 𝑞). 

Decryption: 

The decryption process begins with multiplying (convolving) the received polynomial e by the private 

key f 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

700 
https://internationalpubls.com 

𝑎 ≡ 𝑓 ∗ 𝑒 mod 𝑞, 

Choose the coefficients of a in the interval from −𝑞 2⁄  to 𝑞 2.⁄  Now treating a as a polynomial with 

integer coefficients, computes the temporary polynomial 𝑡 ∈  ℤ[𝑥]/(𝑝, 𝑥𝑛 − 1) by 

𝑡 = 𝑓𝑝
−1  ∗ 𝑎 ( mod 𝑝), 

Next, it calculates the two intermediate quantities 

𝑏 ≡ 𝑒 − 𝑡 (mod 𝑝) and  ≡ 𝑡 − 𝐺(𝑏) ( mod 𝑝), 

And then writes c in the form 

𝑐 = 𝑐′ +  𝑐"𝑥𝑛−𝑘 with deg (𝑐′)  < 𝑛 − 𝑘 and deg(c")  < 𝑘. 

Finally, compares the quantities 

𝑐" and 𝐻(𝑐′, 𝑏). 

If the decrypted message matches the original, it is accepted  𝑐′ as valid. Otherwise, the message is 

rejected as invalid.  

NTRU decryption method is given below.  

 𝑎 ≡ 𝑓 ∗ 𝑒  

≡ 𝑓 ∗ 𝑟 ∗ ℎ + 𝑓 ∗  [𝑚 + 𝐻(𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘 +  𝐺([𝑟 ∗ ℎ]𝑝)]
𝑝
 (mod 𝑞)  

≡ 𝑓 ∗ 𝑝𝑟 ∗ 𝑓𝑞
−1  ∗  𝑔 +  𝑓 ∗  [𝑚 + 𝐻(𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘 +  𝐺([𝑟 ∗ ℎ]𝑝)]

𝑝
 mod 𝑞, 

≡ 𝑝𝑟 ∗  𝑔 +  𝑓 ∗  [𝑚 + 𝐻(𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘 +  𝐺([𝑟 ∗ ℎ]𝑝)]
𝑝
 mod 𝑞, 

By selecting a suitable parameter, it is ensured that all coefficients of the parameter lie between the 

range of (-q)⁄2 and (q)⁄2. This guarantees that the parameter remains unchanged even if its coefficients 

are lowered modulo q. This implies that by reducing the coefficients of the polynomial f * e modulo q 

within the range of (-q)/2 to q/2, it becomes feasible to precisely recover the polynomial. 

a = 𝑝𝑟 ∗  𝑔 +  𝑓 ∗  [𝑚 + 𝐻(𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘 +  𝐺([𝑟 ∗ ℎ]𝑝)]
𝑝
 in R 

Reducing a modulo p then gives the polynomial 

𝑓 ∗  [𝑚 + 𝐻(𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘 +  𝐺([𝑟 ∗ ℎ]𝑝)]
𝑝
in R 

And then multiplying by 𝑓𝑝
−1  produces 

t = 𝑚 + 𝐻(𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘 +  𝐺([𝑟 ∗ ℎ]𝑝) in ℤ[𝑋]/(𝑝, 𝑋𝑛 − 1) 

Compute 𝑏 = 𝑒 − 𝑡  and recover 𝑏 = 𝑟 ∗ ℎ. 

Therefore computation of c yields 

𝑐 =  𝑚 + 𝐻(𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘. 

Accordingly, 𝑐′ is the original message m, and 𝑐" should match up with the hash 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

701 
https://internationalpubls.com 

𝐻(𝑚, [𝑟 ∗ ℎ]𝑝) = 𝐻(𝑚, 𝑏), as noted above. 

Parameter Selection: 

With appropriately chosen parameters, the decryption process has a very high probability of 

successfully recovering the original message. However, occasional decryption failures can occur, often 

due to improper centering of the message. To address this, it is advisable to include a few check bits 

in each message block. If decryption fails due to improper centering, you can attempt to recover the 

message by adjusting the coefficients of 𝑎 ≡ 𝑓 ∗ 𝑒 mod 𝑞 within a slightly different range, for 

example, from −𝑞 2 + 𝑥⁄  to 𝑞 2 + 𝑥.⁄  where x is a small positive or negative value. If no such 

adjustment yields a valid decryption, a gap failure occurs, making the message difficult to decrypt. For 

well-chosen parameters, such failures are infrequent and can generally be considered negligible in 

practice. 

(a) Sample spaces. The space of messages ℒ𝑚 consists of all polynomials modulo p.  

ℒ𝑚 = {𝑚 ∈ 𝑅: 𝑚 has coefficients lying between 

−
1

2
(p − 1) and 

1

2
 (p − 1) and has degree at most 𝑁 − 𝑘 − 1 }. 

It is most suitable to take assume p is odd.  Sets of the form are used to describe the other sample 

spaces. 

ℒ(𝑑1, 𝑑2) = {𝑓 ∈ 𝑅: 𝑓 has 𝑑1cofficients equal 1, 𝑑2 cofficients equal − 1, the rest 0}. 

With this notation, choose three positive integers 𝑑𝑓 , 𝑑𝑔, 𝑑𝑟and set 

ℒ𝑓 = ℒ(𝑑𝑓 , 𝑑𝑓 − 1),    ℒ𝑔 = ℒ(𝑑𝑔, 𝑑𝑔), 𝑎𝑛𝑑    ℒ𝑟 =  ℒ(𝑑𝑟 , 𝑑𝑟). 

(Don't set ℒ𝑓 = ℒ(𝑑𝑓 , 𝑑𝑓) and a polynomial satisfying 𝑓(1) = 0 can never be invertible.) Notice that 

𝑓 ∈ ℒ𝑓 , 𝑔 ∈  ℒ𝑔and r ∈ ℒ𝑟  have 𝐿2norms  

|𝑓|2 = √2𝑑𝑓 − 1 − 𝑛−1,           |𝑔|2 = √2𝑑𝑔,          |𝑟|2 = √2𝑑𝑟 .      

Give values for 𝑑𝑓 , 𝑑𝑔, 𝑑𝑟 that allow decryption while maintaining various security levels. 

(b) Define the width of an element 𝑓 ∈ 𝑅  

|𝑓|∞ =  
max

0 ≤ 𝑖 ≤ 𝑁 − 1
 {𝑓𝑖} −

min
0 ≤ 𝑖 ≤ 𝑁 − 1

{𝑓𝑖}. 

This is a sort of 𝐿∞ norm on R. Similarly, describe a cantered 𝐿2norm on R by 

|𝑓|2 =  (∑(𝑓𝑖 −  𝑓)̅
2

𝑛−1

𝑖=0

)

1 2⁄

, 𝑤ℎ𝑒𝑟𝑒 𝑓̅ =  
1

𝑛
∑ 𝑓𝑖.

𝑛−1

𝑖=0

  

(Equivalently, |f|2 √n⁄  is the standard deviation of the coefficients of F.)  

The following proposition was suggested by Don Coppersmith. 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

702 
https://internationalpubls.com 

Proposition: For any ϵ > 0 there are constants γ1, γ2 > 0, depending on ϵ and N, such that for 

randomly chosen polynomials f, g ∈ R, the probability is greater than 1 − ε that they satisfy 

𝛾1|𝑓|2|𝑔|2 ≤ |𝑓 ∗ 𝑔|∞ ≤ 𝛾2|𝑓|2|𝑔|2. 

Of course, this proposition would be useless from a practical viewpoint if the ratio 𝛾2 𝛾1⁄ were very 

large for small 𝜀′s. However; it turns out that even for moderately large values of N and very small 

values of 𝜖, the constants 𝛾1, 𝛾2 are not at all extreme. 

(c) A Decryption Criterion.  Let 

𝑚′ = [𝑚 + 𝐻(𝑚, [𝑟 ∗ ℎ]𝑝)𝑋𝑛−𝑘 + 𝐺([𝑟 ∗ ℎ]𝑝)]
𝑝
 

Polynomial used (𝑒 ≡ 𝑟 ∗ ℎ + 𝑚′ mod 𝑞. ) for encryption.  In order for the decryption process to 

operate, it is necessary that 

|𝑓 ∗ 𝑚′ + 𝑝𝑟 ∗ 𝑔|∞ < 𝑞. 

This will almost always be true if parameters are selected so that 

|𝑓 ∗ 𝑚′|∞ ≤ 𝑞 4⁄  and |𝑝𝑟 ∗ 𝑔|∞ ≤ 𝑞 4⁄ ; 

This proposes that take 

|𝑓|2|𝑚|2 ≈ 𝑞 4𝛾2 and |𝑟|2|𝑔|2 ≈ 𝑞 4𝑝𝛾2⁄⁄  

For a 𝛾2 matching to a small value for 𝜖. For example, experimental evidence recommends that for   N 

= 167 and N = 503, suitable values for 𝛾2 are 0.27 and 0.17 respectively.    

NTRU is a lattice-based public-key cryptosystem recognized for its high performance and strong 

resistance to quantum computing attacks, making it a leading alternative to RSA and Elliptic Curve 

Cryptography (ECC). It is built on the “Approximate Close Lattice Vector Problem,” a mathematical 

challenge that underpins its security. One of NTRU’s defining features is its efficient polynomial 

multiplication, which is the most complex operation during encryption and decryption. This operation 

is notably faster compared to the corresponding processes in other asymmetric cryptosystems such as 

RSA, ElGamal, and ECC. Performance comparisons between the NTRU cryptosystem and RSA 

focusing on key-size, key-generation, encryption, and decryption—are illustrated in Figures 1, 2, and 

3, respectively [6]. 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

703 
https://internationalpubls.com 

  

Figure-1 Key-size V/s Key-generation               Figure-2 Key-size V/s Encryption (blocks/sec) 

 

Figure-3 Key-size V/s Decryption (blocks/sec) 

The performance analysis from the figures clearly indicates that NTRU outperforms RSA in key 

generation, encryption, and decryption. NTRU has been endorsed by two prominent standards 

organizations: the IEEE and the Financial-Services Industry Standards Committee-X9, specifically 

through the X9.98 Lattice-Based Polynomial Public-Key Establishment Algorithm. This endorsement 

highlights NTRU's role in securing communications within the financial sector. The X9.98 standard 

represents a significant advancement by enhancing the resilience of systems against both quantum and 

classical attacks, thereby strengthening the overall security of financial services systems. 

4. Proposed AKA security protocol 

The 3GPP committee has established the security architecture for the 5G network [2]. The proposed 

AKA protocol involves four main entities: UE (User-Equipment), SEAF (Subscription-Encryption and 

Authentication Function), AUSF (Authentication-Server Function), and UDM/ARPF (Unified Data 

Management/Authentication-Repository and Processing Function), as illustrated in Figure 4. The 

NTRU cryptosystem is employed to secure message exchanges between these entities. 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

704 
https://internationalpubls.com 

 

Figure 4 Proposed AKA security protocol. 

In the protocol, the UE sends its subscriber identity in message-M1 to SEAF, encrypted with the NTRU 

public key of UDM/ARPF. SEAF then forwards a request containing the subscriber identity to AUSF 

and sends SNN in message-M2, also encrypted with UDM/ARPF's NTRU public key. AUSF passes 

the received message to UDM/ARPF as AV_REQ message-M3. UDM/ARPF decrypts the 

authentication request using its NTRU private key and generates an authentication response, 

UDM_AV_RES. This response, which contains validation information, is sent to AUSF in message-

M4 (AVs), encoded with AUSF's NTRU public key. This allows AUSF to confirm the UE's identity. 

AUSF stores the validation vectors (AVs) and uses the NTRU public key of SEAF to send RAND and 

AUTN from the selected AV to SEAF in message-M5. SEAF then decrypts these messages using its 

NTRU private key and forwards RAND and AUTN to UE in message-M6, encrypted with UE's NTRU 

public key. UE verifies the MAC in AUTN by decoding it with its NTRU private key, XMAC. If the 

MACs match, UE calculates and returns the response RES to SEAF in message-M7. UE also computes 

CK and IK, similar to the calculations in UDM/ARPF. SEAF then sends the received message to AUSF 

as AUTH_UE_AUTH_REQ in message-M8. Upon receiving RES from UE, AUSF compares it with 

XRES from the AV. If they match, the authentication is deemed successful, and AUSF computes the 

session key KAUSF from CK and IK to secure the wireless communication with UE. AUSF also derives 

KSEAF from KAUSF and sends it to SEAF in message-M9. UE independently calculates KAUSF and KSEAF. 

This process ensures that (i) UE and AUSF share the same session key KAUSF, and (ii) UE and SEAF 

share the same session key KSEAF. An optional EAP (Extensible Authentication Protocol) procedure 

follows message-M8, where SEAF sends an N1 Message (EAP Success) to UE with KgNB in message-

M10. The security of the protocol has been verified using ProVerif, a cryptographic verifier based on 

the Dolev-Yao model. ProVerif, which uses a modified form of pi-calculus, assesses secrecy and 

authentication properties. The absence of detected attacks indicates that the subscriber identity can be 

securely transferred across insecure channels between network entities. Even if an attacker controls 

the channel, the subscriber identity and other messages remain secure. 



Communications on Applied Nonlinear Analysis 

ISSN: 1074-133X 

Vol 31 No. 7s (2024) 

 

705 
https://internationalpubls.com 

5. Discussion 

This paper demonstrates that the NTRU cryptosystem is one of the fastest public key cryptosystems, 

offering various levels of security at high speeds. Unlike RSA and similar algorithms, which are 

vulnerable to quantum computing techniques capable of factoring integers and computing discrete 

logarithms in polynomial time, NTRU remains robust. The security of NTRU is based on the 

challenging problem of lattice reduction, making it resistant to quantum computing attacks. Approved 

for standardization by IEEE in 2009, NTRU offers strong security even on devices with constrained 

resources, where computing power bandwidth and storage are limited. It is known for its low CPU and 

battery consumption, reducing server utilization significantly. NTRU's efficiency surpasses that of 

other public key cryptosystems in both hardware and software implementations. The proposed AKA 

protocol, which leverages the NTRU cryptosystem, ensures the confidentiality of subscriber identities 

and other messages within wireless communication networks. We used the ProVerif tool to validate 

this approach. While the NTRU-based solution may introduce more delay compared to existing 

methods, it provides enhanced security. This highlights a trade-off between security and computational 

overhead, but ultimately, the improved security justifies the added computational cost. 

Acknowledgement 

We would like to convey our sincere thanks to Dr. K R Parmar and Dr. A. Kulshreshta for their 

guidance and support. 

Conflicts of interest 

“The authors declare that there is no conflict of interest regarding the publication of this paper.” 

References 

[1] 3GPP, “Security Architecture and Procedures for the 5G System”, 3GPP TS 33.501 version 15.2.0. Technical Report, 

The 3rd Generation Partnership Project.  

[2] Alese, B. K., Philemon E. D., Falaki, S. O., “Comparative Analysis of Public-Key Encryption Schemes”, International 

Journal of Engineering and Technology, Volume 2, No. 9, September 2012. 

[3] Giripunje Lokesh, Nimbhorkar Sonali, “Comprehensive Security System for Mobile Network Using Elliptic Curve 

Cryptography over GF (p)”, International Journal of Advanced Research in Computer Science and Software 

Engineering, Volume 3, Issue 5, May 2013.  

[4] Pallipamu Venkateswara Rao, K Thammi Reddy, P Suresh Varma “A Survey On Digital Signatures”, International 

Journal of Advanced Research in Computer and Communication Engineering, Volume 3, Issue 6, June 2014 

[5] Hoffstein, J., Lieman, D., Pipher, J. and Silverman, J.H., “NTRU: A Ring-Based Public Key Cryptosystem” Available 

online on www.ntru.org. 

[6] Hoffstein, J., Lieman, D., Pipher, J. and Silverman, J.H., “NTRU: A Public Key Cryptosystem”. 

http://grouper.ieee.org/ groups/1363/lattPK/submissions.html#NTRU1 

[7] Hien Ba Nguyen, Thesis on “An Overview of the NTRU Cryptographic system”, San Diego State University, 2014. 

[8] Jover R P, Marojevic V, “Security and protocol exploit analysis of the 5G specifications”, IEEE Access 2019, 7, 

24956–24963.  

[9] Ahmad I, Shahabuddin S, Kumar T, Okwuibe, J, Ylianttila M, “Security for 5G and beyond”, IEEE Commun. Surv. 

Tutor. 2019, 21, 3682–3722. 

[10] Hu X., Liu C., Liu S., Li J., Cheng X., “A vulnerability in 5G authentication protocols and its Countermeasure”, 

IEICE Trans. Inf. Syst. 2020, 103, 1806–1809. 

[11] Khan H., Martin K.M., “A survey of subscription privacy on the 5G radio interface-the past, present and future”, J. 

Inf. Secure. Appl. 2020, 53, 102537. 

[12] Blanchet, B.; Smyth, B.; Cheval, V.; Sylvestre, M. “ProVerif 2.05: Automatic Cryptographic Protocol Verifier, User 

Manual and Tutorial”. Proverif User Manual, 2023.  https://bblanche.gitlabpages.inria.fr/proverif/