FACTA UNIVERSITATIS 

Series: Electronics and Energetics Vol. 33, N
o
 2, June 2020, pp. 217-226 

https://doi.org/10.2298/FUEE2002217V 

© 2020 by University of Niš, Serbia | Creative Commons License: CC BY-NC-ND 

COMPARATIVE ANALYSIS OF ML AND MAP DETECTORS 

FOR PAM CONSTELLATIONS IN AWGN CHANNEL


  

Slobodan A. Vlajkov, Aleksandra Ž. Jovanović, Zoran H. Perić
 

University of Niš, Faculty of Electronic Engineering, Department of Telecommunications, 

Niš, Republic of Serbia 

Abstract. In this paper we perform a comparative performances analysis of “maximum 

a posteriori” (MAP) and “maximum likelihood” (ML) detectors for one-dimensional 

constellation in the adaptive white Gaussian noise (AWGN) channel. More precisely, 

error probabilities per symbol for the aforementioned detectors are compared for the 

case when the pulse amplitude modulation (PAM) constellation with the equidistant 

and non-equiprobable constellation points is used as one-dimensional constellation. 

We perform analysis for different distributions of the constellation point probabilities 

and different values of the signal-to-noise ratio (SNR). The analysis indicates which 

detector can be adequate choice for the certain distribution of constellation point 

probabilities and the SNR. Besides this, for the straightforward performance assessment 

of the MAP detector we derive a formula for the symbol error probability. Our analysis 

also points out that the nonuniform distribution of the constellation points probabilities 

does not necessarily improve the symbol error probability. With the aim to decrease the 

symbol error probability we propose a method for defining constellation point 

probabilities. The presented results show that PAM constellation designed by utilizing 

the method we propose significantly outperforms the conventional PAM constellation 

in terms to the symbol error probability. 

Key words: PAM constellation, AWGN channel, ML detector, MAP detector, symbol error 

probability 

1. INTRODUCTION 

One classical issue in digital communications is estimation of the symbol error 

probability after transmission of digital signal through the additive white Gaussian noise 

(AWGN) channel 1-17. This error probability mainly depends on the decision rule, 

that is, on the type of the detector. The decision rule that minimizes the probability of 

decision error is the “maximum a posteriori” (MAP) decision rule. However, the detector 

based on MAP decision rule requires the knowledge of a priori probabilities of symbols 

                                                           
Received April 14, 2019; received in revised form July 1, 2019 

Corresponding author: Slobodan A. Vlajkov 

Faculty of Electronic Engineering, Aleksandra Medvedeva 14, 18000 Niš, Republic of Serbia 

E-mail: vlajkov.slobodan@gmail.com 

  



218 S.A. VLAJKOV, A.Ž. JOVANOVIĆ, Z.H. PERIĆ 
 

and also it has considerable complexity. From these reasons, the simpler detectors are 

desirable. One such simpler detector is the “maximum likelihood” (ML) detector that does 

not take into consideration the a priori probabilities of symbols 1, 2. Although the topic 

on MAP and ML detection has been studied for a long time, it has not still completely 

investigated largely because of the complexity of the MAP detection. Thus, the comparison 

of ML and MAP detectors for multidimensional constellations was recently performed in 

3 pointing that the topic is still actual. Besides this, as there have been an increasing 

number of studies on the constellations with the non-equiprobable symbols 3-17, the 

research we perform here can be meaningful in detection of these constellations. 

In this paper, for PAM constellations with the equidistant and non-equiprobable symbols 

we estimate and compare the symbol error probabilities of the MAP and ML detectors in 

order to obtain answer on question which detector is suitable for certain scenario. Also, we 

derive a formula for the symbol error probability of MAP detector. The formula we propose 

is useful for the performances estimation when signal is modulated with the non-

equiprobable and equidistant PAM constellation and is transmitted through the AWGN 

channel. 

The choice of the constellation point probabilities considerably influences the symbol 

error probabilities whereby one should notice that the the unadequate choice of the 

constellation point probabilities can considerably increase the symbol error probability. 

This issue motivates us to develop a method for determining constellation points 

probabilities, that lead to the significantly decreased symbol error probability. We will 

demonstrate that the PAM constellation based on method we propose outperforms some 

known PAM constellations in terms of the symbol error probability. 

In order to present our research in a distinct and concise manner, in follows we firstly 

define the one-dimensional constellation. Then we focus on the signal reception by 

considering in detail the ML and MAP detectors. Finally, we propose the method for 

designing PAM constellation with the equidistant and non-equiprobable constellation points. 

2. SYMBOL ERROR PROBABILITY FOR PAM CONSTELLATION IN AWGN CHANNEL  

As already mentioned in the Introduction we investigate the performances of ML and 

MAP detectors for PAM constellations in AWGN channel. Before we define constellation, 

we briefly explain the considered scenario. Namely, the useful signal sent into the channel 

is s(t). Due to transmission through the communication channel the signal is corrupted by 

the additive white Gaussian noise n(t). Then, on the channel output, that is, at the detector 

input there is the signal r(t) which represents the sum of the useful signal and the noise. 

Based on the sample of the received signal r = r(kT) = s(kT) + n(kT) = ai + n, k  0, the 
detector makes decision about the transmitted signal. 

2.1. PAM constellation parameters 

We consider M-ary PAM constellation with constant distance d between the amplitudes 

of adjacent constellation points ai, i = 1, 2,..., M. We also assume that constellation is 

symmetric, which enables us to focus on the positive part of the constellation. By following 

these assumptions, we formulate the expression for the constellation point amplitude as 

follows: 



 Comparative Analysis of ML and MAP Detectors for PAM Constellations in AWGN Channel 219 

 
2

,...,1
2

1 M
idia

i









 . (1) 

The parameter we also define is the probability of constellation point marked with 

Pi = P(ai). We assume PAM constellation with non-equiprobable symbols, so that it holds 

 constP
i
 , (2) 

under the constraint that the sum of probabilities of all constellation points is equal to 1: 

 



2

1

12
M

i
i

P . (3) 

Finally, we define the average energy per symbol Es in the following manner 1: 

 



2/

1

2
2

M

i
iis

PaE . (4) 

After that we formulate expressions for the average energy per bit Eb 1: 

 
M

E
E s

b

2
log

  (5) 

and the signal-to-noise ratio per bit SNR 1: 

 
0

log10
N

E
SNR b , (6) 

where N0 is the spectral power density of the AWGN. 

2.2. MAP and ML decision rules 

The sample of the received signal is expressed analytically in the following manner: 

 nar
i
 , (7) 

where ai is the sample of the useful signal s(t) and n is the sample of the AWGN noise n(t) 

having Gaussian probability density function with zero-mean value and variance σn
2
 = N02. 

From (7) follows that for given ai the sample r has also the Gaussian probability density 

function with the same variance as the noise variance, but with the mean value ai 1: 

 
2

00

( )1
( ) exp ,i

i

r a
p r a r

NN

 
      

 
. (8) 

The MAP rule that minimizes the error probability reads 1, 2: 

 ( ) , if ( ) ( ) for
i i m

D r a p a r p a r m   . (9) 



220 S.A. VLAJKOV, A.Ž. JOVANOVIĆ, Z.H. PERIĆ 
 

By applying Bayesian rule on (9) we obtain a simpler form for the decision rule: 

 ( ) ( ) for
i i m m

p r a P p r a P m  . (10) 

By using (10), (8) and (1) we derive the decision thresholds of MAP detector: 

 

0 1

1 2 1

( 1) ln , 2,...,
2 2

0,

i
i

i

M

N P M
m i d i

d P

m m





   

  

. (11) 

Let us now define the ML decision rule 1, 2: 

 ( ) , if ( ) ( ) for
i i m

D r a p r a p r a m   . (12) 

By substituting (8) and (1) into (12) we derive the decision thresholds of ML detector: 

 

1 2 1

( 1) , 2,...,
2

0,

i

M

M
m i d i

m m


  

  

. (13) 

By comparing equations (10) and (11) with equations (12) and (13) for Pi = 1M one can 

notice the well-known fact that in the case of equiprobable symbols the MAP and ML 

decision rules are equivalent. After determining decision thresholds the following general 

formulation for decision rules can be assumed 

 
1

( ) , if
i i i

D r a m r m


   . (14) 

To stress the difference between the ML and MAP decision rules, we graphically 

illustrate the decision thresholds in Fig. 1. 

 

 
Fig. 1 The decision thresholds for: 1) MAP detector mi = a for Pi  1 < Pi,  mi = b for Pi  1 = Pi,  

mi = c for Pi  1 > Pi 
; 2) ML detector bm

i
 . 

2.3. Formulas for symbol error probability  

During transmission through the channel the channel noise affects the digital signal. 

Due to that, there is a probability that the transmitted symbol is not accurately detected. 

The probability that the wrong symbol is detected is called the error probability per 

symbol (Pe). The general form for the symbol error probability of a symmetric PAM 

constellation is 1, 2: 



 Comparative Analysis of ML and MAP Detectors for PAM Constellations in AWGN Channel 221 

 
/ 2

1

2 ( )
M

e i i

i

P P P e a


  , (15) 

where P(eai) represents conditional error probability per symbol: 

 
1

1 1
( ) [ , ) 1 [ , ) 1 ( )

i

i

m

i i i i i i i i

m

P e a P r m m a P r m m a p r a dr


 
               . (16) 

By substituting (16), (8), (11) and (1) into (15) we derive an analytical expression for the 

symbol error probability: 

 

0 1

1

20 0

/ 2 1
0 01

2 10 0

2 10

2

20

erfc erfc ln
22 2

erfc ln erfc ln
2 22 2

erfc ln .
22

e

M

i i

i

i i i

M

M

M

N Pd d
P P

d PN N

N NP Pd d
P

d P d PN N

PNd
P

d PN





 



    
       

        

       
          

           

 
 
 
 

  (17) 

where erfc() is the complementary error function: 

 
2

z

2
erfc(z) exp{ }t dt





  . (18) 

One can notice that in the case of the ML detection, the expression for the symbol error 

probability gets much simpler form: 

 
2

0

(1 )erfc
2

e M

d
P P

N

 
   

 
 

. (19) 

Assuming in (19) that symbols are equiprobable, we derive the known expression for the 

symbol error probability of the conventional PAM constellation 1:  

 















0
2

erfc
1

N

d

M

M
P

e
. (20) 

3. NUMERICAL RESULTS AND DISCUSSION 

In order to be able to quantify and compare the performances of the ML and MAP 

detectors for the previously defined scenario, for a given number of constellation points 

M we change other parameters of PAM constellation and calculate SNR and Pe. We assume 

the number of constellation points M = 8. We define the probabilities of the constellation 

points on several manners. Actually, we observe four cases illustrated in Fig. 2. In the first 

case in the Fig. 2 the probabilities of the constellation points are equal and amount 

Pi = 0.125 (the uniform distribution). In the case given in the Fig. 2.b the probability of the 



222 S.A. VLAJKOV, A.Ž. JOVANOVIĆ, Z.H. PERIĆ 
 

constellation point decreases with the increase of the amplitude of the constellation point 

(decreasing distribution). In the third case we observe the PAM constellation whose 

probability of constellation point increases with the constellation amplitude, so we name it 

the increasing distribution. Graphical representation of the third case is shown in Fig. 2.c. 

The last case is characterized with the probabilities of constellation points arranged in 

„zigzag‟ form, that is, the distribution of constellation points does not change monotonically 

with the constellation amplitude. The last case can be seen in Fig. 2.d. 

 
a.                                                               b. 

 
c.                                                                  d. 

Fig. 2 Distribution of the constellation point probabilities for 8-PAM constellation: 

a) uniform distribution Pi = [0.125, 0.125, 0.125, 0.125]; 

b) decreasing distribution Pi = [0.21, 0.15, 0.1, 0.04]; 

c) increasing distribution Pi = [0.04, 0.1, 0.15, 0.21]; 

d)„zigzag‟ distribution Pi = [0.18, 0.07, 0.18, 0.07], where Pi = [P1, P2, P3, P4]. 

For all presented distributions of the probabilities of constellation points and for 

different values of the SNR, the symbol error probabilities for ML and MAP detectors 

are calculated by using (17) and (19) and listed in the Table 1. 

In order to better distinct the difference in the performances of the ML and MAP 

detectors, we introduce the relative difference between the symbol error probabilities δ: 

   100100% 






MAP

e

MAP

e

ML

e

MAP

e

e

P

PP

P

P
, (21) 



 Comparative Analysis of ML and MAP Detectors for PAM Constellations in AWGN Channel 223 

where Pe
MAP 

and Pe
ML 

are the error probabilities for MAP and ML detector, respectively. 

In Fig. 3 the  in function of the SNR is graphically presented. 
By analyzing the results presented in Fig. 3 and Table 1, we can derive several 

conclusions. In every case the error probability per symbol of MAP detector is lower than 

the corresponding one of ML detector (the known fact). For PAM constellation with 

equiprobable symbols (see the first case) we confirm that there is no difference between the 

symbol error probabilities of ML and MAP detectors. Here it should be also noticed that the 

first case defines the conventional PAM constellation 1. For the second case, that is, when 

the distribution of constellation probabilities is decreasing, we observe that the relative 

difference between the symbol error probabilities of ML and MAP detectors amounts about 

4.6 % for SNR = 0 dB and 2.5 % for SNR = 20 dB. Besides that, the symbol error 

probability for both types of detectors is for three orders lower than the corresponding one 

in the first case. In the third case, when the distribution of probabilities is increasing, 

relative difference in symbol error probabilities is somewhat greater and ranges from 7.5 % 

to 3.2 % for the signal – to – noise ratio per bit ranging from 0 dB to 20dB. However, for 

this kind of probability distribution one should observe that the error probability for both 

detectors is for three orders higher than the error probability of a conventional PAM. 

Finally, the fourth case with the „zigzag‟ distribution of the constellation points probabilities 

the relative difference in symbol error probabilities has a significantly greater value than it is 

in other cases. This difference reaches 19.1% for SNR = 0dB and 9.2% for SNR= 20dB. 

Thereby, the symbol error probability for both detectors is of the same order as the error 

probability of the conventional PAM constellation. From Fig. 3 we can also observe that 

curves in all cases approach the curve corresponding to the case where there is no difference 

between MAP and ML detection. This actually means that with the increase of SNR the 

relative difference in the symbol error probability decreases. 

Table 1 Symbol Error Probability for ML and MAP detectors in function of SNR:  

a) Case 1 and Case 2, b) Case3 and Case 4 

a) 

Pe(SNR) CASE 1 CASE 2 

SNR[dB] ML MAP ML MAP 

0 0.519 0.519 0.461 0.441 

5 0.299 0.299 0.201 0.194 

10 0.080 0.080 0.025 0.024 

15 0.002 0.002 6.9110
-5

 6.7310
-5

 

20 7.9010
-9 

7.9010
-9

 1.6110
-12

 1.5710
-12

 

b) 

Pe(SNR) CASE 3 CASE 4 

SNR[dB] ML MAP ML MAP 

0 0.517 0.481 0.519 0.436 

5 0.337 0.320 0.277 0.243 

10 0.124 0.119 0.059 0.054 

15 9.3610
-3

 9.0510
-3

 9.1710
-4

 8.3510
-4

 

20 6.0310
-6

 5.8410
-6

 4.3410
-9

 3.9710
-9

 

 

 



224 S.A. VLAJKOV, A.Ž. JOVANOVIĆ, Z.H. PERIĆ 
 

 

Fig. 3 Relative difference in the symbol error probabilities of the ML and MAP detectors. 

As we have already noticed the PAM constellation with the increasing distribution of 

the constellation point probabilities does not represent a good constellation because its 

symbol error probability is significantly higher than that for the conventional PAM 

constellation. This points out that the nonuniform distribution of the constellation points 

probabilities does not necessarily improve the symbol error probability. Because of that, 

determination of constellation points probabilities with the aim to reduce the symbol 

error probability is an important task. In this paper we propose that the constellation 

points probabilities follows the Gaussian distribution. Namely, we assume that the 

constellation point probability Pi is equal with the probability that Gaussian variable with 

zero mean value and unit variance belongs to segment ai  d2, ai + d2: 

 

2
,,1,

2
erfc

2

)1(
erfc

2

1 M
i

iddi
P

i

































 
 . (22) 

For M = 16 and PAM constellation with equidistant constellation points whose probabilities 

are defined with (22) we calculate symbol error probability by utilizing (17) and tabulate the 

obtained results in Table 2. In Table 2 we also tabulate the symbol error probability of the 

conventional PAM constellation. One can evident that the PAM constellation we propose 

achieves significantly lower symbol error probability than it is in the case of conventional 

PAM constellation, whereby the achieved gain grows with the SNR. Thus, for SNR = 20 dB 

it is reached that the symbol error probability of our PAM constellation is for 2.57106 times 

lower than that of its conventional counterpart. Furthermore, the considered PAM 

constellation outperform the best PAM constellation in 17 whose symbol error probability 

amounts 4.5710
-6

 for SNR = 20 dB. Unlike the PAM constellation defined with (22), the 

PAM constellation in 17 is characterized with only two different constellation points 

probabilities, so that it represents the PAM constellation of smaller complexity than it is the 

PAM constellation proposed in this paper.  



 Comparative Analysis of ML and MAP Detectors for PAM Constellations in AWGN Channel 225 

Table 2 Symbol Error Probability for 16-PAM Constellation Based on (22)  

 

SNR[dB] Pe
 

Pe
conventional PAM

 

0 0.515 0.712 

5 0.265 0.549 

10 0.051 0.311 

15 5.3610
-4

 0.079 

20 7.8610
-10

 2.0210
-3

 

4. CONCLUSION 

In this paper, for PAM constellation with equidistant and non-equiprobable symbols the 

comparative analysis of the MAP and ML detectors has been performed. One of the results of 

analysis is formula for the symbol error probability of MAP detector. It has been also noticed 

that in all observed cases, except for the first one, with the increase of the signal-to-noise ratio 

the relative difference in the symbol error probabilities of the ML and MAP detectors 

decreases so that it can be expected that it becomes negligible for higher SNR values. This 

means that in the channels with higher values of signal-to-noise ratio the detector of smaller 

complexity - the ML detector can be used. The another finding is that when the probability 

distribution has „zigzag‟ form, the choice of the detector is of great importance since it 

significantly influences the symbol error probability. In such situations, the MAP detector is 

preferable solution. It has been observed that the increasing distribution of the constellation 

points probabilities negatively effects the symbol error probability, while the decreasing 

distribution decreases the symbol error probability. Finally, this observation has helped us to 

define the method for determining constellation points probabilities. The proposed method 

has led to the symbol error probability of PAM constellation being significantly decreased in 

comparison to that for the conventional PAM constellation. 

Acknowledgement: This paper was realized as a part of the projects "Development and 

implementation of next-generation systems, devices and software based on software radio for radio 

and radar networks" (TR-32051) and "Development of Dialogue Systems for Serbian and Other 

South Slavic Languages" (TR 32035), financed by Ministry of Education, Science and 

Technological Development of the Republic of Serbia. 

REFERENCES 

[1] J. G. Proakis, Digital communications, McGraw-Hill Science, 2001. 

[2] P. Ivaniš, D. Drajić, Introduction to Information Theory and Coding (In Serbain), Akademska misao, 

2018. 

[3] A. Alvarado, E. Agrell, F. Brannstrom, "Asymptotic comparison of ML and MAP detectors for 

multidimensional constellations", IEEE Transactions on Information Theory, vol. 64, no. 2, pp. 1231–

1240, 2018.  

[4] M. Ivanov, F. Brannstrom, A. Alvarado, et al., "On the exact BER of bit-wise demodulators for one-

dimensional constellations", IEEE Transactions on Communications, vol. 61, no. 4, pp. 1450–1459, 

2013.  

[5] H. Kuai, F. Alajaji, G. Takahara, "Tight error bounds for nonuniform signaling over AWGN channels", 

IEEE Transactions on Information Theory, vol. 46, no. 7, pp 2712–2718, 2000.  



226 S.A. VLAJKOV, A.Ž. JOVANOVIĆ, Z.H. PERIĆ 
 

[6] L. Wei, I. Korn, "Optimal M-amplitude shift keying/quadrature amplitude shift keying with non-equal 

symbol probabilities", IET Communications, vol. 5, no. 6, pp 745–752, 2011.  

[7] L. Wei, "Optimized M-ary orthogonal and bi-orthogonal signaling using coherent receiver with non-equal 

symbol probabilities", IEEE Communications Letter, vol. 16, no. 6, pp 793–796, 2012.  

[8] I. Korn, J. P. Fonseka, S. Xing, "Optimal binary communication with nonequal probabilities", IEEE 

Transactions on Communications, vol. 51, no. 9, pp 1435–1438, 2003. 

[9] V. Ipatov, "Comments on Optimal binary communication with nonequal probabilities", IEEE Transactions on 

Communications, vol. 55, no. 1, pp. 231, 2007. 

[10] Z. H. Perić, "Nonlinear transformation of one-dimensional constellation points in order to error 

probability decreasing", Facta Universitatis, Series: Electronics and Energetics, vol. 11, no. 3, pp. 291–

299, 1998.  

[11] Z. H. Perić, M. S. Bogosavljević, "Performance of nonuniform PAM constellations for Gaussian 

channel", Electronics and Electrical Engineering, vol. 52, no. 3, pp. 27–30, 2004.  

[12] Z. H. Perić, N. Milošević, A. Ž. Jovanović, et al., "Design of piecewise uniform PAM constellation", In 

Proceedings of the XII International SAUM Conference on Systems, Automatic Control and 

Measurements, Niš, Serbia, November 2014, pp. 109–111. 

[13] Z. H. Perić, A. Ž. Jovanović, S. A. Vlajkov, "Comparative analysis of various PAM constellations", In 

Proceedings of the XIII International SAUM Conference on Systems, Automatic Control and 

Measurements, Niš, Serbia, November 2016, pp. 27-30, 2016. 

[14] I. B. Djordjevic, A. Ž. Jovanovic, Z. H. Peric, T. Wang, "Optimized vector-quantization-based signal 

constellation design (OVQ-SCD) for multidimensional optical transport", In Proceedings of thd CLEO: Science 

and Innovations, San Jose, California United States, June 8-13, 2014.  

[15] I. B. Đorđević, A. Ž. Jovanović, Z. H. Perić, T. Wang, "Multidimensional optical transport based on optimized 

vector-quantization-inspired signal constellation design", IEEE Transactions on Communications, vol. 62,  no. 

9, pp. 3262–3273, 2014.  

[16] I. B. Đorđević, A. Ž. Jovanović, M. Cvijetić, Z. H. Perić, "Multidimensional vector quantization-based 

signal constellation design enabling beyond 1 Pb/s serial optical transport networks", IEEE Photonics 

Journal, vol. 5, no. 4, 2013.  

[17] S. A. Vlajkov, A. Ž. Jovanović, Z. H. Perić, "Approach in companding-quantisation-inspired PAM 

constellation design", IET Communications, Vol. 12, No. 18, pp. 2305–2314, 2018.