EUROPEAN JOURNAL OF PURE AND APPLIED MATHEMATICS Vol. 12, No. 3, 2019, 790-820 ISSN 1307-5543 – www.ejpam.com Published by New York Business Global Divergence Measures Estimation and Its Asymptotic Normality Theory in the discrete case Amadou Diadie Ba1,∗, Gane Samb Lo1,2,3 1 U.F.R. S.A.T., LERSTAD, Gaston Berger University, Saint Louis, Sénégal 2 LASTA, Pierre et Marie University, Paris, France 3 African University of Sciences and Technology, Abuja, Nigeria Abstract. In this paper we provide the asymptotic theory of the general of φ-divergences mea- sures, which include the most common divergence measures : Rényi and Tsallis families and the Kullback-Leibler measure. We are interested in divergence measures in the discrete case. One sided and two-sided statistical tests are derived as well as symmetrized estimators. Almost sure rates of convergence and asymptotic normality theorem are obtained in the general case, and next particularized for the Rényi and Tsallis families and for the Kullback-Leibler measure as well. Our theoretical results are validated by simulations. 2010 Mathematics Subject Classifications: 62G05, 62G07, 62F05 Key Words and Phrases: φ-divergence measure estimation, Rényi, Tsallis, Kullback-Leibler divergence measures, symmetrized divergence measures 1. Introduction 1.1. Motivations In this paper, we study the convergence of φ−divergence measure estimator for empirical discrete probability distributions supported on a finite set. Let throughout the following X = {c1, c2, · · · , cr} (r ≥ 2) be a finite countable space. The probability distributions on X are finite dimensional vectors p in P(X ) = { p = (pc)c∈X : pc ≥ 0, ∀c ∈ X and ∑ c∈X pc = 1 } . A divergence measure on P(X ) is a function D : (P(X ))2 −→ R (p,q) 7−→ D(p,q) (1) ∗Corresponding author. DOI: https://doi.org/10.29020/nybg.ejpam.v12i3.3437 Email addresses: ba.amadou-diadie@ugb.edu.sn (A.D.Ba), gane-samb.lo@ugb.edu.sn (G.S. Lo) http://www.ejpam.com 790 c© 2019 EJPAM All rights reserved. Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 791 such that D(p,p) = 0 for any p such that (p,p) in the domain of application of D. The function D is not necessarily a mapping. And if it is, it is not always symmetric and it does neither have to be a metric. In lack of symmetry, the following more general notation is more appropriate : D : P1(X )× P2(X ) −→ R (p,q) 7−→ D(p,q), (2) where P1(X ) and P2(X ) are two families of probability distributions on X , not necessarily the same. To better explain our concern, let us introduce some of the most celebrated divergence measures. Let (p,q) ∈ P(X ) × P(X ) with X = {c1, c2, · · · , cr}, and let X and Y two randoms variables such that P(X = cj) = pj , and P(Y = cj) = qj , j ∈ {1, · · · , r}, and set p = (p1, · · · , pr)t and q = (q1, · · · , qr)t. The four most popular divergence measures are : (1) The L2 2-divergence measure : DL2(p,q) = r∑ j=1 (pj − qj)2. (3) (2) The family of Rényi’s divergence measures indexed by α 6= 1, α > 0, known under the name of Rényi-α : DR,α(p,q) = 1 α− 1 log  r∑ j=1 pαj q 1−α j  . (4) (3) The family of Tsallis divergence measures indexed by α 6= 1, α > 0, also known under the name of Tsallis-α : DT,α(p,q) = 1 α− 1  r∑ j=1 pαj q 1−α j − 1  . (5) (4) The Kullback-Leibler divergence measure DKL(p,q) = r∑ j=1 pj log(pj/qj). (6) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 792 The latter, the Kullback-Leibler divergence measure, may be interpreted as a limit case of both the Rényi’s family and the Tsallis’ one by letting α → 1. As well, for α near 1, the Tsallis family DT,α(p,q) may be seen as derived from DR,α(p,q) based on the first order expansion of the logarithm function in the neighborhood of the unity. Here for ease of notation we refer the notation log as the natural logarithm. From this small sample of divergence measures, we may give the following remarks : For both the Rényi and the Tsallis families, we may have computation problems. So without loss of generality, suppose pj > 0 and qj > 0, ∀j ∈ D = {1, 2, · · · , r} (BD) (7) If Assumption (7) holds, we do not have to worry about summation problems, especially for Tsallis, Rényi and Kulback-Leibler measures, in the computations arising in estimation theories. This explains why Assumption (7) is systematically used in a great number of works in that topic, for example, in [20], [10], [8], and recently in [1] to cite a few. It is clear from the very form of these divergence measures that we do not have symmetry, unless for the special case where α = 1/2. So we define the following symmetric version of divergence measures D(s)(p,q) = D(p,q) +D(q,p) 2 , provided that D(p,q) and D(q,p) are finite. Both families are build on the following summation Sα(p,q) = ∑ j∈D pαj q 1−α j , with α 6= 1, α > 0. Although we are focusing on the aforementioned divergence measures in this paper, it is worth mentioning that there exist quite a few number of them. Let us cite for example the ones named after : Ali-Silvey or f -divergence (see [6]), Cauchy-Schwarz, Jeffrey divergence (see [5]), Chernoff (see [5]) , Jensen-Shannon (see [5]). According to [3], there is more than a dozen of different divergence measures in the literature. Before coming back to our divergence measures estimation of interest, we want to high- light some important applications of them. Indeed, divergence has proven to be useful in applications. Let us cite a few of them : (a) They heavily intervene in information theory and recently in machine learning. Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 793 (b) They have been used as similarity measures in image registration or multimedia clas- sification (see [17]). (c) They are also used as loss functions in evaluating and optimizing the performance of density estimation methods (see [8]). (d) Divergence estimates can also be used to determine sample sizes required to achieve given performance levels in hypothesis testing. (e) There has been a growing interest in applying divergence to various fields of science and engineering for the purpose of estimation, classification, etc. (See [2] and [13]). (f) Divergence also plays a central role in the frame of large deviations results including the asymptotic rate of decrease of error probability in binary hypothesis testing problems. (g) The estimation of divergence between the samples drawn from unknown distributions gauges the distance between those distributions. Divergence estimates can then be used in clustering and in particular for deciding whether the samples come from the same dis- tribution by comparing the estimate to a threshold. (h) Divergence gauges how differently two random variables are distributed and it provides a useful measure of discrepancy between distributions. The reader may find more applications and descriptions in the following papers : [11], [7], [18], [9], [17], and [15]. In the next subsection, we describe the frame in which we place the estimation problems we deal in this paper. 1.2. Statistical Estimations The divergence measures may be applied to two statistical problems among others. (A) First, it may be used as a fitting problem as described here. Let X1, X2, · · · a sample of replications of X with an unknown probability distribution p and we want to test the hypothesis that p is equal to a known and fixed probability p0. Theoretically, we can answer this question by estimating a divergence measure D(p,p0) by a plug-in estimator D(p̂n,p) where, for each n ≥ 1, p is replaced by an estimator p̂n of the probability law, which is based on sample X1, X2, ..., Xn, to be precised. From there establishing an asymptotic theory of ∆n = D(p̂n,p0)−D(p,p0) is thought to be necessary to conclude. Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 794 (B) Next, it may be used as tool of comparing for two distributions. We may have two samples and wonder whether they come from the same probability distribution. Here, we also may two different cases. (B1) In the first, we have two independent samples X1, X2, .... and Y1, Y2, .... respectively from a random variable X and Y according the probability distributions p and q. Here the estimated divergence D(p̂n, q̂m), where n and m are the sizes of the available samples, is the natural estimator of D(p,q) on which depends the statistical test of the hypothesis : p = q. (B2) But the data may also be paired (X,Y ), (X1, Y1), (X2, Y2), ..., that is Xi and Yi are measurements of the same case i = 1, 2, ... In such a situation, testing the equality of the margins pX = pY should be based on an estimator p̂ (n) X,Y of the joint probability law of the couple (X,Y ) based of the paired observations (Xi, Yi), i = 1, 2, . . . , n. We did not encounter the approach (B2) in the literature. In the (B1) approach, almost all the papers used the same sample size, at the exception of [19], for the double-size estimation problem. In our view, the study case should rely on the available data so that using the same sample size may lead to a loss of information. To apply their method, one should take the minimum of the two sizes and then loose information. We suggest to come back to a general case and then study the asymptotic theory of D(p̂n, q̂m) based on samples X1, X2, .., Xn. and Y1, Y2, ..., Ym. In this paper, we will systematically use arbitrary samples sizes. 1.3. Previous work In the context of the situation (B1), there are several papers dealing with the estimation of the divergence measures. As we are concerned in this paper by the weak laws of the estimators, our review on that problematic did not return significant things. Instead, the literature presented us many kinds of results on almost-sure efficiency of the estimation, with rates of convergences and laws of the iterated logarithm, Lp (p = 1, 2) convergences, etc. To be precise, [4] used recent techniques based on functional empirical process to pro- vide a series of interesting rates of convergence of the estimators in the case of one-sided approach for the class de Rényi, Tsallis, Kullback-Leibler to cite a few. Unfortunately, the authors did not address the problem of integrability, taking that the divergence measures are finite. Although the results should be correct under the boundedness assumption (7) (BD) we described earlier, a new formulation in that frame would be welcome. In the context of the situation (B1), we may cite first the works of [20] and [10]. They both used divergence measures based on probability density functions and concentrated of Rényi-α, Tsallis-α and Kullback-Leibler. Specifically, [10] defined Reyni and Tsallis estimators by correcting the plug-in estimator Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 795 and established that, as long as DR,α(p,q) ≥ c and DT,α(p,q) ≥ c, for some constant c > 0, then E |DR,α(p̂n, q̂n)−DR,α(p,q)| ≤ c ( n−1/2 + n− 3s 2s+d ) and E |DT,α(p̂n, q̂n)−DT,α(p,q)| ≤ c ( n−1/2 + n− 3s 2s+d ) . [19] used a k−nearest-neighbor approach to prove that if |α− 1| < k, (α 6= 1) then lim n,m→∞ E [DT,α(p̂n, q̂m)−DT,α(p,q)]2 = 0 and lim n,m→∞ E (DR,α(p̂n, q̂m)) = DR,α(p,q). There has been recent interest in deriving convergence rates for divergence estimators (see [16] and [10]). The rates are typically derived in terms of smoothness s of the densities. Similarly, [21] showed that when s > d a k-nearest-neighbor style estimator achieves rate n−2/d (in absolute error) ignoring logarithmic factors. In a follow up work, the authors improved this result to O(n−1/2) using an ensemble of weak estimators, but they require s > d orders of smoothness. [20] provided an estimator for Rényi−α divergences as well as general density functionals that uses a mirror image kernel density estimator. They obtained exponential inequalities for the deviation of the estimators from the true value. [12] studied an ε−nearest neighbor estimator for the L2−divergence that enjoys the same rate of convergence as the projection-based estimator of [10]. 1.4. Main contributions Our main contribution may be summarized as follows : for data sampled from one or two unknown random variables, we derive almost sure convergency and central limit theorems for empirical φ− divergences. We will focus on divergence measures between discrete prob- ability distribution. As well, our results applied to the approaches (A) and (B1) defined above. As a consequence, we estimate divergence measures by their plug-in counterparts, meaning that we replace the probability mass function (p.m.f.) in the expression of the divergence measure by a nonparametric estimator of the p.m.f. We also wish to get first general laws for an arbitrary functional of the form J(p,q) = ∑ j∈D φ(pj , qj), (8) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 796 where φ : (0, 1)2 → R is a twice continously differentiable function. The results on the functional J(p,q), which is also known under the name of φ-divergence, will lead to those on the particular cases of the Tsallis, Rényi, and Kullback-Leibler measures. 1.5. Overview of the paper The rest of the paper is organized as follows. In Section 2, we define estimators of the p.m.f.’s pj and qj based on two i.i.d. samples according respectively to p and to q. These ones allow us to define the empirical φ−divergences. In Section 3, we will give our full results for functional J(p,q) both one sided and two-sided approaches. In Section 4, we will particularize the results for specific measures we already described. In Section 5 we present some simulations confirming our results. Finally in Section 6, we conclude. 2. Empirical φ− divergence 2.1. Notations and main results Before we state the main results we need a few definitions. Let X and Y two randoms variables defined on the probability distributions (X ,A,P) with X = {c1, c2, · · · , cr} and p = (pj)1≤j≤r and q = (qj)1≤j≤r two discrete probability distributions on X such that, for any j ∈ D pj = P(X = cj) and qj = P(Y = cj). We suppose that (7) is satisfied that is ∀ j ∈ D, pj > 0 and qj > 0. Define the empirical probability distribution generated by i.i.d. random variablesX1, · · · , Xn from the probability distribution p as p̂n = (p̂cn)c∈X , where p̂ cj n = 1 n n∑ i=1 1cj (Xi) (9) where 1cj (Xi) = { 1 if Xi = cj 0 otherwise for any j ∈ D. q̂m is defined in the same way by Y1, · · · , Ym i.i.d.∼ q that is q̂m = (q̂cm)c∈X , where q̂ cj m = 1 m m∑ i=1 1cj (Yi). (10) For a given j ∈ D, this empirical estimator p̂ cj n of pj is strongly consistent and asymptot- ically normal. Precisely, when n tends to infinity, p̂ cj n − pj a.s.−→ 0 (11) √ n(p̂ cj n − pj) D Zpj , (12) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 797 where Zpj d∼ N (0, pj(1− pj)). We denote by a.s.−→ the almost sure convergence and D the convergence in distribution. The notation d∼ denote the equality in distribution. These asymptotic properties derive from the law of large numbers and central limit theo- rem. For sake of simplicity, we introduce the two following notations : ∆ cj pn = p̂ cj n − pj , ∆ cj qm = q̂ cj m − qj , ∀ j ∈ D, an = sup j∈D |∆cj pn |, bm = sup j∈D |∆cj qm |, and cn,m = max(an, bm). (13) For any j ∈ D, set δn(pj) = √ n/pj∆ cj pn and δm(qj) = √ m/qj∆ cj qm . We recall that, since for a fixed j ∈ D, np̂cjn has a binomial distribution with parameters n and success probability pj , we have E ( p̂ cj n ) = pj and V(p̂ cj n ) = pj(1− pj) n . Furthermore, by the strong law of large numbers, we know that ∆ cj pn a.s.−→ 0, as n→ +∞, for a fixed j ∈ D. And finally, by the asymptotic Gaussian limit of the multinomial law (see for example Chapter 1, Section 4 in [14]), we have( δn(pj), j ∈ D ) D Z(p) L∼ N (0,Σp), as n→ +∞, (14) and ( δm(qj), j ∈ D) ) D Z(q) L∼ N (0,Σq), as m→ +∞, (15) where Z(p) = (Zpj , j ∈ D)t and Z(q) = (Zqj , j ∈ D)t are two independent centered Gaussian random vectors of dimension r having respectively the following elements : (Σp)(i,j) = (1− pj)δij − √ pipj(1− δij), (i, j) ∈ D2 (16) (Σq)(i,j) = (1− qj)δij − √ qiqj(1− δij), (i, j) ∈ D2, (17) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 798 where δij = { 1 for i = j 0 for i 6= j . For a fixed j ∈ D, we have also E ( q̂ cj m ) = qj , V(q̂ cj m) = qj(1− qj) m , and ∆ cj qm a.s.−→ 0, as m→ +∞. 2.2. φ-divergence measure Definition 1. The φ-divergence between the two probability distributions p and q is given by J(p, q) = ∑ j∈D φ(pj , qj), (18) where φ : [0, 1]2 → R is a measurable function having continuous second order partial derivatives. The results on the functional J(p,q) will lead to those on the particular cases of the Tsallis, Rényi, and Kullback-Leibler measures. Based on (9) and (10), we will use the following empirical φ-divergences : J(p̂n,q) = ∑ j∈D φ(p̂ cj n , qj), J(p, q̂m) = ∑ j∈D φ(pj , q̂ cj m), and J(p̂n, q̂m) = ∑ j∈D φ(p̂ cj n , q̂ cj m). Denote φ (1) 1 (s, t) = ∂φ ∂s (s, t), φ (1) 2 (s, t) = ∂φ ∂t (s, t), φ (2) 1 (s, t) = ∂2φ ∂s2 (s, t), and φ (2) 2 (s, t) = ∂2φ ∂t2 (s, t), φ (2) 1,2(s, t) = φ (2) 2,1(s, t) = ∂2φ ∂s∂t (s, t). Set A1,p = ∑ j∈D |φ(1)1 (pj , qj)|, A2,q = ∑ j∈D |φ(1)2 (pj , qj)|, (19) A3,q = ∑ j∈D |φ(1)1 (qj , pj)|, and A4,p = ∑ j∈D |φ(1)2 (qj , pj)|. (20) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 799 3. Statements of the main results 3.1. Main results Here are our main results. The first concerns the almost sure efficiency of the estimators. Theorem 1. Let p and q two probability distributions and p̂n and q̂m be generated by i.i.d. samples X1, · · · , Xn and Y1, · · · , Ym according respectively to p and q and given by (9) and (10), the assumption (7) be satisfied. Then the following asymptotic results hold. (a) One sample lim sup n→+∞ |J(p̂n, q)− J(p, q)| an ≤ A1,p, a.s., (21) lim sup m→+∞ |J(p, q̂m)− J(p, q)| bm ≤ A2,q, a.s., (22) (b) Two samples : lim sup (n,m)→(+∞,+∞) |J(p̂n, q̂m)− J(p, q)| cn,m ≤ A1,p +A2,q a.s., (23) where an, bm and cn,m are as in (13) and A1,p and A2,q as in (19). Proof. In the proof, we will systematically use the mean values theorem. In the mul- tivariate handling, we prefer to use the Taylor-Lagrange-Cauchy as stated in [22], page 230. For a fixed j ∈ D, we have φ(p̂ cj n , qj) = φ(pj + ∆ cj pn , qj) (24) = φ(pj , qj) + ∆ cj pnφ (1) 1 (pj + θ1,j∆ cj pn , qj), by applying the mean value theorem to the function (.) 7→ φ((.), qj) and where θ1,j is some number lying between 0 and 1. In the sequel, any θi,j , i = 1, 2, · · · satisfies |θi,j | < 1. By applying again the mean values theorem to the function (.) 7→ φ (1) 1 ((.), qj), we have φ (1) 1 (pj + θ1,j∆ cj pn , qj) = φ (1) 1 (pj , qj) + θ1,j∆ cj pnφ (2) 1 (pj + θ2,j∆ cj pn , qj). We can write (24) as φ(p̂ cj n , qj) = φ(pj , qj) + ∆ cj pnφ (1) 1 (pj , qj) + θ1,j(∆ cj pn)2φ (2) 1 (pj + θ2,j∆ cj pn , qj). Now we have, by summation over j ∈ D, J(p̂n,q)− J(p,q) = ∑ j∈D ∆ cj pnφ (1) 1 (pj , qj) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 800 + ∑ j∈D θ1,j(∆ cj pn)2φ (2) 1 (pj + θ2,j∆ cj pn , qj), (25) hence |J(p̂n,q)− J(p,q)| ≤ an ∑ j∈D |φ(1)1 (pj , qj)| + a2n ∑ j∈D |φ(2)1 (pj + θ2,j∆ cj pn , qj)|. Therefore |J(p̂n,q)− J(p,q) an ≤ A1,p + an ∑ j∈D |φ(2)1 (pj + θ2,j∆ cj pn , qj)|, and then lim sup n→∞ |J(p̂n,q)− J(p,q) an ≤ A1,p, since A1,p <∞, an a.s.−→ 0, and∑ j∈D |φ(2)1 (pj + θ2,j∆ cj pn , qj)| → ∑ j∈D |φ(2)1 (pj , qj)| <∞ as n→∞. This proves (21). Formula (22) is obtained in a similar way. We only need to adapt the result concerning the first coordinate to the second. The proof of (23) comes by splitting ∑ j∈D ( φ(p̂ cj n , q̂ cj m)− φ(pj , qj) ) , into the following two terms ∑ j∈D ( φ(p̂ cj n , q̂ cj m)− φ(pj , qj) ) = ∑ j∈D ( φ(p̂ cj n , q̂ cj m)− φ(pj , q̂ cj m) ) + ∑ j∈D ( φ(pj , q̂ cj m)− φ(pj , qj) ) ≡ In,1 + In,2. We already know how to handle In,2. As to In,1, we may still use the Taylor-Lagrange- Cauchy formula since we have, for a fixed j ∈ D,∥∥(p̂ cj n , q̂ cj m)− (pj , q̂ cj m) ∥∥ ∞ = ∥∥(p̂ cj n − pj), 0) ∥∥ ∞ = an → 0. By the Taylor-Lagrange-Cauchy (see [22], page 230), we have Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 801 In,1 = ∑ j∈D ∆ cj pnφ(p̂ cj n + θj∆ cj pn , q̂ cj m) ≤ an ∑ j∈D |φ(p̂ cj n + θj∆ cj pn , q̂ cj m)| = an(A1 + o(1)). From there, the combination of these remarks directs to the result. The second main result concerns the asymptotic normality of the estimators. Let V1(p,q) = ∑ j∈D pj(1− pj)(φ(1)1 (pj , qj)) 2 − 2 ∑ (i,j)∈D2, i 6=j pipjφ (1) 1 (pi, qi)φ (1) 1 (pj , qj)(26) and V2(p,q) = ∑ j∈D qj(1− qj)(φ(1)2 (pj , qj)) 2 − 2 ∑ (i,j)∈D2, i 6=j qiqjφ (1) 2 (pi, qi)φ (1) 2 (pj , qj).(27) Theorem 2. Under the same assumptions as in Theorem 1, the following central limit theorems hold. (a) One sample : as n→ +∞, √ n(J(p̂n, q)− J(p, q)) D N (0, V1(p, q)) , (28) √ m(J(p, q̂m)− J(p, q)) D N (0, V2(p, q)) . (29) (b) Two samples : for (n,m)→ (+∞,+∞) and nm/(n+m)→ γ ∈ (0, 1),( nm mV1(p, q) + nV2(p, q) )1/2 (J(p̂n, q̂m)− J(p, q)) D N (0, 1) . (30) Proof. Let us prove (28). By going back to (25), we have √ n(J(p̂n,q)− J(p,q)) = ∑ j∈D √ pjδn(pj)φ (1) 1 (pj , qj) + √ nR1,n where R1,n = ∑ j∈D θ1,j(∆ cj pn)2φ (2) 1 (pj + θ2,j∆ cj pn , qj). Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 802 Now using Formula (14) above, we get,∑ j∈D √ pjδn(pj)φ (1) 1 (pj , qj) D ∑ j∈D φ (1) 1 (pj , qj) √ pjZpj , as n→ +∞ which follows a centered normal law of variance V1(p,q) : V1(p,q) = ∑ j∈D (1− pj)(φ(1)1 (pj , qj)) 2 − 2 ∑ (i,j)∈D2, i 6=j √ pipjφ (1) 1 (pi, qi)φ (1) 1 (pj , qj) since V ∑ j∈D φ (1) 1 (pj , qj) √ pjZpj  = ∑ j∈D V(φ (1) 1 (pj , qj) √ pjZpj ) + 2 ∑ (i,j)∈D2, i 6=j Cov(φ (1) 1 (pi, qi) √ piZpi , φ (1) 1 (pj , qj) √ pjZpj ) = ∑ j∈D pj(φ (1) 1 (pj , qj)) 2V(Zpj ) + 2 ∑ (i,j)∈D2, i 6=j √ pipjφ (1) 1 (pi, qi)φ (1) 1 (pj , qj)Cov(Zpi , Zpj ) = ∑ j∈D pj(1− pj)(φ(1)1 (pj , qj)) 2 − 2 ∑ (i,j)∈D2, i 6=j pipjφ (1) 1 (pi, qi)φ (1) 1 (pj , qj). Finally, the proof will be complete if we show that √ nR1,n converges, in probability, to zero, as n tends to infinity. We have∣∣√nR1,n ∣∣ ≤ √na2n∑ j∈D |φ(2)1 (pj + θ2,j∆ cj pn , qj)|. (31) Let show that √ na2n = oP(1). By the Bienaymé-Tchebychev inequality, we have, for any ε > 0 and for j ∈ D, P( √ n(p̂ cj n − pj)2 ≥ ε) = P ( |p̂cjn − pj | ≥ √ ε n1/4 ) ≤ pj(1− pj) εn1/2 , which implies that √ na2n converges in probability to 0 as n→ +∞. Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 803 Finally from (31) we have √ nR1,n P→ 0 as n→ +∞ which implies √ n(J(p̂n,q)− J(p,q)) D N (0, V1(p,q)) , as n→ +∞. This ends the proof of (28). The result (29) is obtained by a symmetry argument by swapping the roles of p and q. Now, it remains to prove Formula (30) of the theorem. Let us use bi-variate Taylor- Lagrange-Cauchy formula to get, J(p̂n, q̂m)− J(p,q) = ∑ j∈D ∆ cj pnφ (1) 1 (pj , qj) + ∑ j∈D ∆ cj qmφ (1) 2 (pj , qj) 1 2 ∑ j∈D ( (∆ cj pn)2φ (2) 1 + ∆ cj pn∆ cj qmφ (2) 1,2 + (∆ cj qm)2φ (2) 2 )( u cj n , v cj m ) , where (u cj n , v cj n ) = (pj + θ∆ cj pn , qj + θj∆ cj qm). Thus we get J(p̂n, q̂m)− J(p,q) = 1√ n Nn(p) + 1√ m Nm(q) +Rn,m, where Nn(p) = ∑ j∈D √ pjδn(pj)φ (1) 1 (pj , qj) D ∑ j∈D φ (1) 1 (pj , qj)Zpj , as n→ +∞, Nm(q) = ∑ j∈D √ qjδm(qj)φ (1) 2 (pj , qj) D ∑ j∈D φ (1) 2 (pj , qj)Zqj , as m→ +∞, and Rn,m is given by 1 2 ∑ j∈D ( (∆ cj pn)2φ (2) 1 + ∆ cj pn∆ cj qmφ (2) 1,2 + (∆ cj qm)2φ (2) 2 )( u cj n , v cj m ) . First, we have that Nn(p) and Nm(p) are independents and hence Nn(p) L∼ N (0, V1(p,q)) and Nm(q) L∼ N (0, V2(p,q)) . Therefore 1√ n ∑ j∈D √ pjδn(pj)φ (1) 1 (pj , qj) + 1√ m ∑ j∈D √ qjδm(qj)φ (1) 2 (pj , qj) = N ( 0, V1(p,q) n + V2(p,q) m ) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 804 + oP ( 1√ n ) + oP ( 1√ m ) . Thus J(p̂n, q̂m)− J(p,q) = N ( 0, V1(p,q) n + V2(p,q) m ) + oP ( 1√ n ) + oP ( 1√ m ) +Rn,m. Next, we have 1√ V1(p,q) n + V2(p,q) m (J(p̂n, q̂m)− J(p,q)) = N (0, 1) + oP  1√ n 1√ V1(p,q) n + V2(p,q) m  + oP  1√ m 1√ V1(p,q) n + V2(p,q) m  + 1√ V1(p,q) n + V2(p,q) m Rn,m. That leads to√ nm mV1(p,q) + nV2(p,q) (J(p̂n, q̂m)− J(p,q)) = N (0, 1) + oP(1) + √ nm mV1(p,q) + nV2(p,q) Rn,m, since m/(mV1(p,q) + nV2(p,q)) and m/(nV1(p,q) + nV2(p,q)) are bounded, and then oP  1√ n 1√ V1(p,q) n + V2(p,q) m  = oP (√ m mV1(p,q) + nV2(p,q) ) = oP(1), and oP  1√ m 1√ V1(p,q) n + V2(p,q) m  = oP (√ n mV1(p,q) + nV2(p,q) ) = oP(1). It remains to prove that ∣∣∣∣√ nm mV1(p,q)+nV2(p,q) Rn,m ∣∣∣∣ = oP(1). But we have, by the continuity assumptions on φ and on its partial derivatives and by the uniform converges of ∆ cj pn and ∆ cj qm to zero, that ∣∣∣∣√ nm mV1(p,q) + nV2(p,q) Rn,m ∣∣∣∣ ≤ Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 805 1 2 √na2n( ∑ j∈D φ (2) 1 (pj , qj) + o(1)) (√ m mV1(p,q) + nV2(p,q) ) + 1 2 √mb2m( ∑ j∈D φ (2) 2 (pj , qj) + o(1)) (√ n mV1(p,q) + nV2(p,q) ) + 1 2 √nambm( ∑ j∈D φ (2) 2 (pj , qj) + o(1)) (√ n mV1(p,q) + nV2(p,q) ) . As previously, we have √ na2n = oP(1), √ mb2m = oP(1) and √ nambm = oP(1). From there, the conclusion is immediate. 3.2. Direct extensions Quite a few number of divergence measures are not symmetric. Among these non- symmetric measures are some of the most interesting ones. For such measures, estimators of the form J(p̂n,q), J(p, q̂m) and J(p̂n, q̂m) are not equal to J(q, p̂n), J(q̂m,p) and J(q̂m, p̂n) respectively. In one-sided tests, we have to decide whether the hypothesis p = q, for q known and fixed, is true based on data from p. In such a case, we may use one of the statistics J(p̂n,q) or J(q, p̂n) to perform the tests. We may have information that allows us to prefer one of them. If not, it is better to use both of them, upon the finiteness of both J(p,q) and J(q,p), in a symmetrized form as J (s)(p,q) = J(p,q) + J(q,p) 2 . (32) The same situation applies when we face double-side tests, i.e., testing p = q from data generated by p et q. Asymptotic a.e. efficiency. Theorem 3. Under the same assumptions as in Theorem 1, the following hold. (a) One sample : lim sup n→+∞ ∣∣J (s)(p̂n, q)− J (s)(p, q) ∣∣ an ≤ 1 2 (A1,p +A4,p) a.e., (33) lim sup n→+∞ ∣∣J (s)(p, q̂m)− J (s)(p, q) ∣∣ bn ≤ 1 2 (A2,q +A3,q) a.e., (34) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 806 (b) Two samples : lim sup (n,m)→(+∞,+∞) ∣∣J (s)(p̂n, q̂m)− J (s)(p, q) ∣∣ cn,m ≤ 1 2 (A1,p +A2,q +A3,q +A4,p) , a.e. (35) Proof. The proof is established by considering the following equalities and by using Theorem 1 J (s)(p̂n,q)− J (s)(p,q) = 1 2 (J(p̂n,q)− J(p,q)) + 1 2 (J(q, p̂n)− J(q,p)) , (36) J (s)(p, q̂m)− J (s)(p,q)) = 1 2 (J(p, q̂m)− J(p,q)) + 1 2 (J(q̂m,p)− J(q,p)) , (37) J (s)(p̂n, q̂m)− J (s)(p,q)) = 1 2 (J(p̂n, q̂m)− J(p,q)) + 1 2 (J(q̂m, p̂n)− J(q,p)) .(38) Asymptotic Normality. In addition to V1(p,q) and V2(p,q) defined in (26) and (27), denote V3(p,q) = ∑ j∈D qj(1− qj)(φ(1)1 (qj , pj)) 2 − 2 ∑ (i,j)∈D2, i 6=j qiqjφ (1) 1 (qi, pi)φ (1) 1 (qj , pj),(39) V4(p,q) = ∑ j∈D pj(1− pj)(φ(1)2 (qj , pj)) 2 − 2 ∑ (i,j)∈D2, i 6=j pipjφ (1) 2 (qi, pi)φ (1) 2 (qj , pj),(40) and finally V1:4(p,q) = 1 4 (V1(p,q) + V4(p,q)) and V2:3(p,q) = 1 4 (V2(p,q) + V3(p,q)). We have Theorem 4. Under the same assumptions as in Theorem 1, the following hold. (a) One sample : as n→ +∞, √ n ( J (s)(p̂n, q)− J (s)(p, q) ) D N (0, V1:4(p, q)), (41) √ n ( J (s)(p, q̂n)− J (s)(p, q) ) D N (0, V2:3(p, q)). (42) (b) Two samples : for (n,m)→ (+∞,+∞) and nm/(n+m)→ γ ∈ (0, 1), ( nm mV1:4(p, q) + nV2:3(p, q) )1/2( J (s)(p̂n, q̂m)− J (s)(p, q) ) D N (0, 1). (43) Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 807 Proof. The proof follows directly by using equalities (36), (37), and (38) above and Theorem 2. Remark : We have just seen that Theorems 3 and 4 are direct consequences of the main Theorems 1 and 2. The same will be true for the following Corollaries concerning the particular cases which proofs will be omitted. 4. Particular Cases 4.1. Rényi and Tsallis families These two families are expressed through the summation Sα(p,q) = ∑ j∈D pαj q 1−α j , α > 0, α 6= 1, (44) which is of the form of φ−divergence measure with φ(x, y) = xαy1−α, (x, y) ∈ {(pj , qj), j ∈ D}. A-(a)- The asymptotic behavior of the Tsallis divergence measure. Denote AT,α,1 = α |α− 1| ∑ j∈D (pj/qj) α−1 and AT,α,2 = ∑ j∈D (pj/qj) α . We have Corollary 1. Under the same assumptions as in Theorem 1, and for any α > 0, α 6= 1, the following hold. (a) One sample : lim sup n→+∞ |DT,α(p̂n, q)−DT,α(p, q)| an ≤ AT,α,1 a.s., lim sup n→+∞ |DT,α(p, q̂n)−DT,α(p, q)| bn ≤ AT,α,2 a.s. (b) Two samples : lim sup (n,m)→(+∞,+∞) |DT,α(p̂n, q̂m)−DT,α(p, q)| cn,m ≤ AT,α,1 +AT,α,2 a.s. Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 808 Denote VT,α,1(p,q) = ( α α− 1 )2(∑ j∈D pj(1− pj)(pj/qj)2α−2 − 2 ∑ (i,j)∈D2, i 6=j (pipj) α(qiqj) α−1 ) and VT,α,2(p,q) = ∑ j∈D qj(1− qj)(pj/qj)2α − 2 ∑ (i,j)∈D2, i 6=j (pipj) α(qiqj) 1−α. We have Corollary 2. Under the same assumptions as in Theorem 1, and for any α > 0, α 6= 1, the following hold. (a) One sample : as n→ +∞, √ n (DT,α(p̂n, q)−DT,α(p, q)) D N (0, VT,α,1(p, q)), √ n (DT,α(p, q̂n)−DT,α(p, q)) D N (0, VT,α,2(p, q)). (b) Two samples : for (n,m)→ (+∞,+∞) and nm/(n+m)→ γ ∈ (0, 1),( nm nVT,α,2(p, q) +mVT,α,1(p, q) )1/2 (DT,α(p̂n, q̂m)−DT,α(p, q)) D N (0, 1). As to the symmetrized form D(s) T,α(p,q) = DT,α(p,q) +DT,α(q,p) 2 , we need the supplementaries notations : AT,α,3 = α |α− 1| ∑ j∈D (qj/pj) α−1 , AT,α,4 = ∑ j∈D (qj/pj) α , VT,α,3(p,q) = ( α α− 1 )2 ∑ j∈D qj(1− qj)(qj/pj)2−2α − 2 ∑ (i,j)∈D2, i 6=j (qiqj) 2−α(pipj) α−1  , and VT,α,4(p,q) = ∑ j∈D pj(1− pj)(qj/pj)2α − 2 ∑ (i,j)∈D2, i 6=j (pipj) 1−α(qiqj) α. We have Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 809 Corollary 3. Under the same assumptions as in Theorem 1, and for any α > 0, α 6= 1, the following hold. (a) One sample : lim sup n→+∞ |D(s) T,α(p̂n, q)−D(s) T,α(p, q)| an ≤ (AT,α,1 +AT,α,4) /2 =: A (s) T,α,1 a.s, lim sup n→+∞ |D(s) T,α(p, q̂n)−D(s) T,α(p, q)| bn ≤ (AT,α,2 +AT,α,3) /2 =: A (s) T,α,2 a.s. (b) Two samples : lim sup (n,m)→(+∞,+∞) |D(s) T,α(p̂n, q̂m)−D(s) T,α(p, q)| cn,m ≤ A(s) T,α,1 +A (s) T,α,2 a.s. Denote VT,α,1:4(p,q) = VT,α,1(p,q) + VT,α,4(p,q) and VT,α,2:3(p,q) = VT,α,2(p,q) + VT,α,3(p,q). We have Corollary 4. Under the same assumptions as in Theorem 1, and for any α > 0, α 6= 1, the following hold. (a) One sample : as n→ +∞, √ n ( D(s) T,α(p̂n, q)−D(s) T,α(p, q) ) D N (0, VT,α,1:4(p, q)), √ n ( D(s) T,α(p, q̂n)−D(s) T,α(p, q) ) D N (0, VT,α,2:3(p, q)). (b) Two samples : for (n,m)→ (+∞,+∞) and nm/(n+m)→ γ ∈ (0, 1),( nm mVT,α,1:4(p, q) + nVT,α,2:3(p, q) )1/2( D(s) T,α(p̂n, q̂m)−D(s) T,α(p, q) ) D N (0, 1). A-(b)- The asymptotic behavior of the Rényi-α divergence measure. The treatment of the asymptotic behavior of the Rényi-α, α > 0, α 6= 1 is obtained from Part (A)-(a) by expansions and by the application of the delta method. We first remark that DR,α(p,q) = 1 α− 1 log (Sα(p,q)) . Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 810 Corollary 5. Under the same assumptions as in Theorem 1, and for any α > 0, α 6= 1, the following hold. (a) One sample : lim sup n→+∞ |DR,α(p̂n, q)−DR,α(p, q)| an ≤ AT,α,1 Sα(p, q) =: AR,α,1 a.s., lim sup n→+∞ |DR,α(p, q̂n)−DR,α(p, q)| bn ≤ AT,α,2 Sα(p, q) =: AR,α,2 a.s. (b) Two samples : lim sup (n,m)→(+∞,+∞) |DR,α(p̂n, q̂m)−DR,α(p, q)| cn,m ≤ AR,α,1 +AR,α,2 a.s. Denote VR,α,1(p,q) = VT,α,1(p,q) S2α(p,q) and VR,α,2(p,q) = VT,α,2(p,q) S2α(p,q) . We have Corollary 6. Under the same assumptions as in Theorem 1, and for any α > 0, α 6= 1, the following hold (a) One sample : as n→ +∞, √ n ( DR,α(p̂n, q)−DR,α(p, q) ) D N (0, VR,α,1(p, q)), √ n ( DR,α(p, q̂n)−DR,α(p, q) ) D N (0, VR,α,2(p, q)). (b) Two samples : for (n,m)→ (+∞,+∞) and nm/(n+m)→ γ ∈ (0, 1),( nm nVR,α,2(p, q) +mVR,α,1(p, q) )1/2( DR,α(p̂n, q̂m)−DR,α(p, q) ) D N (0, 1). As to the symmetrized form D(s) R,α(p,q) = DR,α(p,q)−DR,α(q,p) 2 , we need the supplementary notations : AR,α,3 = AT,α,3 Sα(p,q) , AR,α,4 = AT,α,4 Sα(p,q) VR,α,3(p,q) = VT,α,3(p,q) S2α(p,q) , and VR,α,4(p,q) = VT,α,4(p,q) S2α(p,q) . Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 811 Corollary 7. Under the same assumptions as in Theorem 1, and for any α > 0, α 6= 1, the following hold. (a) One sample : lim sup n→+∞ |D(s) R,α(p̂n, q)−D(s) R,α(p, q)| an ≤ (AR,α,1 +AR,α,4)/2 =: A (s) R,α,1, a.s., lim sup n→+∞ |D(s) R,α(p, q̂n)−D(s) R,α(p, q)| an ≤ (AR,α,2 +AR,α,3)/2 =: A (s) R,α,2, a.s. (b) Two samples : lim sup (n,m)→(+∞,+∞) |D(s) R,α(p̂n, q̂m)−D(s) R,α(p, q)| cn,m ≤ A(s) R,α,1 +A (s) R,α,2, a.s. Denote VR,α,1:4(p,q) = VR,α,1(p,q) + VR,α,4(p,q) and VR,α,2:3(p,q) = VR,α,2(p,q) + VR,α,3(p,q). We have Corollary 8. Under the same assumptions as in Theorem 1, and for any α > 0, α 6= 1, the following hold. (a) One sample : as n→ +∞, √ n ( D(s) R,α(p̂n, q)−D(s) R,α(p, q) ) D N (0, VR,α,1:4(p, q)), √ n ( D(s) R,α(p, q̂n)−D(s) R,α(p, q) ) D N (0, VR,α,2:3(p, q)). (b) Two samples : as (n,m)→ (+∞,+∞) and nm/(n+m)→ γ ∈ (0, 1),( nm nVR,α,2:3(p, q) +mVR,α,1:4(p, q) )1/2( D(s) R,α(p̂n, q̂m)−D(s) R,α(p, q) ) D N (0, 1). B - Kulback-Leibler Measure Here we have DKL(p,q) = ∑ j∈D φ(pj , qj), where φ(x, y) = x log(x/y), (x, y) ∈ {(pj , qj), j ∈ D}. We have Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 812 Corollary 9. Under the same assumptions as in Theorem 1, the following hold. (a) One sample : lim sup n→+∞ |DKL(p̂n, q)−DKL(p, q)| an ≤ ∑ j∈D |1 + log(pj/qj)| =: AKL,1(p, q), a.s., lim sup n→+∞ |DKL(p, q̂n)−DKL(p, q)| bn ≤ ∑ j∈D (pj/qj) =: AKL,2(p, q), a.s. (b) Two samples : lim sup (n,m)→(+∞,+∞) |DKL(p̂n, q̂m)−DKL(p, q)| cn,m ≤ AKL,1(p, q) +AKL,2(p, q), a.s. Denote VKL,1(p,q) = ∑ j∈D pj(1− pj) (1 + log(pj/qj)) 2 − 2 ∑ (i,j)∈D2, i 6=j pipj(1 + log(pi/qi))(1 + log(pj/qj)) and VKL,2(p,q) = ∑ j∈D qj(1− qj)(pj/qj)2 − 2 ∑ (i,j)∈D2, i 6=j pipj . We have Corollary 10. Under the same assumptions as in Theorem 1, the following hold. (a) One sample : as n→ +∞ √ n (DKL(p̂n, q)−DKL(p, q)) D N (0, VKL,1(p, q)), √ n (DKL(p, q̂n)−DKL(p, q)) D N (0, VKL,2(p, q)). (b) Two samples : as (n,m)→ (+∞,+∞) and nm/(n+m)→ γ ∈ (0, 1),( nm nVKL,2(p, q) +mVKL,1(p, q) )1/2 (DKL(p̂n, q̂m)−DKL(p, q)) D N (0, 1). As to the symmetrized form D(s) KL(p,q) = DKL(p,q) +DKL(q,p) 2 , Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 813 we need the supplementary notations : AKL,3(p,q) = ∑ j∈D |1 + log(qj/pj)| , AKL,4(p,q) = ∑ j∈D qj/pj VKL,3(p,q) = ∑ j∈D qj(1− qj)(1 + log(qj/pj)) 2 −2 ∑ (i,j)∈D2, i 6=j qiqj(1 + log(qi/pi))(1 + log(qj/pj)), and VKL,4(p,q) = ∑ j∈D pj(1− pj)(qj/pj)2 − 2 ∑ (i,j)∈D2, i 6=j qiqj . We have Corollary 11. Under the same assumptions as in Theorem 1, the following hold. (a) One sample : lim sup n→+∞ |D(s) KL(p̂n, q)−D(s) KL(p, q)| an ≤ (AKL,1(p, q) +AKL,4(p, q))/2 =: A (s) KL,1(p, q), a.s., lim sup n→+∞ |D(s) KL(p, q̂n)−D(s) KL(p, q)| bn ≤ (AKL,2(p, q) +AKL,3(p, q))/2 =: A (s) KL,2(p, q), a.s. (b) Two samples : lim sup (n,m)→(+∞,+∞) |D(s) KL(p̂n, q̂m)−D(s) KL(p, q)| cn,m ≤ A(s) KL,1(p, q) +A (s) KL,2(p, q), a.s. Denote VKL,1:4(p,q) = VKL,1(p,q) + VKL,4(p,q), VKL,2:3(p,q) = VKL,2(p,q) + VKL,3(p,q). We have Corollary 12. Under the same assumptions as in Theorem 1, the following hold. (a) One sample : as n→ +∞, √ n ( D(s) KL(p̂n, q)−D(s) KL(p, q) ) D N (0, VKL,1:4(p, q)), √ n ( D(s) KL(p, q̂n)−D(s) KL(p, q) ) D N (0, VKL,2:3(p, q)). Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 814 (b) Two samples : for (n,m)→ (+∞,+∞) and nm/(n+m)→ γ ∈ (0, 1), ( nm mVKL,1:4(p, q) + nVKL,2:3(p, q) )1/2( D(s) KL(p̂n, q̂m)−D(s) KL(p, q) ) D N (0, 1). 5. Simulations To assess the performance of ours estimators, we present a simulation study on a finite sample. For simplicity, in our experiments we consider tree outcomes for the randoms variables X and Y , c1, c2, c3 with respectives p.m.f.’s p1, p2, p3 and q1, q2, q3. Our aim is to compare the performance of the divergences measures estimators as well as their symmetrized forms with one or two samples when sample sizes increase. Suppose that p1 = 0.4, p2 = 0.25, p3 = 0.35 and q1 = 0.27, q2 = 0.32, q3 = 0.41. We set first α = 0.99 since lim α→1 DT,α(p,q) = lim α→1 DR,α(p,q) = DKL(p,q) and second α = 0.5 since DT,0.5(p,q) = DT,0.5(q,p) and DR,0.5(p,q) = DR,0.5(q,p). True and the symmetrized form of our interest divergence measures are then DT,0.99(p,q) ≈ 0.03969, D(s) T,0.99(p,q) ≈ 0.03854, DR,0.99(p,q) ≈ 0.03970, D(s) R,0.99(p,q) ≈ 0.03854, DKL(p,q) ≈ 0.04012, and D(s) KL(p,q) ≈ 0.03893. and DR,0.5(p,q) ≈ 0.01951452. In each figures 1, 2, 3, 4, 5, 6, 7, and 8, left panels represent the plots of divergence mea- sure estimator, built from sample sizes of n = 100, 200, · · · , 30000, and the true divergence measure (represented by horizontal black line). The middle panels show the histograms of the data and where the red line represents the plots of the theoretical normal distribution calculated from the same mean and the same standard deviation of the data. The right panels concern the Q-Q plot of the data which display the observed values against normally distributed data (represented by the red line). We see that the underlying distribution of the data is normal since the points fall along a straight line. 6. Conclusion This paper joins a growing body of literature on estimating divergence measures in the discrete case and on finite sets. We adopted the plug-in method and we derived almost sure rates of convergence and asymptotic normality of the most common divergence measures in one sample, two samples as well as symmetrical form of divergence measures, all this, by means of the functional φ−divergence measure. Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 815 Figure 1 displays the performances of the Stallis divergence estimators of order α = 0.99, DT,α(p̂n,q), DT,α(p, q̂m), and DT,α(p̂n, q̂m). Figure 1: Plots when samples sizes increase, histograms and normal Q-Q plots of DT,α(pn, q), DT,α(p, qm), and DT,α(pn, qm) (α = 0.99) versus N (0, 1). Figure 2 concerns the performances of the symmetrized form of Stallis divergence estima- tors D(s) T,α(p̂n,q), D(s) T,α(p, q̂m) and D(s) T,α(p̂n, q̂m). Figure 2: Plots when samples sizes increase, histograms and normal Q-Q plots of D(s) T,α(pn, q), D (s) T,α(p, qm), and D(s) T,α(pn, qm) (α = 0.99) versus N (0, 1). Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 816 Figure 3 displays the performances of the Rényi divergence estimators of order α = 0.99, DR,α(p̂n,q), DR,α(p, q̂m), and DR,α(p̂n, q̂m). Figure 3: Plots when samples sizes increase, histograms and normal Q-Q plots of DR,α(pn, q), DR,α(p, qm), and DR,α(pn, qm) (α = 0.99) versus N (0, 1). Figure 4 concerns the performances of the symmetrized form of Rényi divergence estimators D(s) R,α(p̂n,q), D(s) R,α(p, q̂m), and D(s) R,α(p̂n, q̂m). Figure 4: Plots when samples sizes increase, histograms and normal Q-Q plots of D(s) R,α(pn, q), D (s) R,α(p, qm), and D(s) R,α(pn, qm) (α = 0.99) versus N (0, 1). Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 817 Figure 5 displays the performances of the Rényi divergence estimators of order α = 0.5, DR,α(p̂n,q), DR,α(p, q̂m), and DR,α(p̂n, q̂m). Figure 5: Plots when samples sizes increase, histogram and normal Q-Q plots of DR,α(pn, q), DR,α(p, qm), and DR,α(pn, qm) (α = 0.5) versus N (0, 1). Figure 6 concerns the performances of the symmetrized form of Rényi divergence estimators D(s) R,α(p̂n,q), D(s) R,α(p, q̂m), and D(s) R,α(p̂n, q̂m). Figure 6: Plots when samples sizes increase, histograms and normal Q-Q plots of D(s) R,α(pn, q), D (s) R,α(p, qm), and D(s) R,α(pn, qm) (α = 0.5) versus N (0, 1). Ba A.D., Lo G.S. / Eur. J. Pure Appl. Math, 12 (3) (2019), 790-820 818 Figure 7 displays the performances of the Kullback-Leibler estimators, DKL(p̂n,q), DKL(p, q̂m), and DKL(p̂n, q̂m). Figure 7: Plots when samples sizes increase, histograms and normal Q-Q plots of DKL(pn, q), DKL(p, qm), and DKL(pn, qm) versus N (0, 1). Figure 8 concerns the performances of the symmetrized form of Kullback-Leibler estima- tors, D(s) KL(p̂n,q), D(s) KL(p, q̂m), and D(s) KL(p̂n, q̂m). Figure 8: Plots when samples sizes increase, histograms and normal Q-Q plots of D(s) KL(pn, q), D (s) KL(p, qm), and D(s) KL(pn, qm) versus N (0, 1). REFERENCES 819 References [1] Lô G.S. Bâ A.D. and Gaston Berger University Bâ, D. Divergence measures estima- tion and its asymptotic normality theory using wavelets empirical processes i. Journal of Statistical Theory and Applications, 17:158–171, 2018. [2] P. K. Bhattacharya. Efficient estimation of a shift parameter from grouped data. The Annals of Mathematical Statistics, 38:1770–1787, 1967. [3] Andrzej Cichocki and Shun-ichi Amari. Families of alpha- beta- and gamma- diver- gences: Flexible and robust measures of similarities. Entropy, 12:1532–1568, 2010. [4] Hamza Dhaker. Kernel-type estimators of divergence measures and its strong uniform consistency. American Journal of Theoretical and Applied Statistics, 5:13, 2016. [5] Atif Ahmet Evren. Some applications of kullback-leibler and jeffrey’s divergences in multinomial population. Journal of Selcuk University Natural and Applied Science, 1, 2012. [6] Topsoe F. Some inequalities for information divergences and related measures of discrimination. IEEE Transactions on Information Theory, 46, 2000. [7] K. Fukunaga and R.R. Hayes. The reduced parzen classifier. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11:423–425, 1989. [8] Peter Hall. On kullback-leibler loss and density estimation. The Annals of Statistics, 15:1491–1519, 1987. [9] Trevor Hastie and Robert Tibshirani. Classification by pairwise coupling. The Annals of Statistics, 26:451–471, 1998. [10] Akshay Krishnamurthy, Kirthevasan Kandasamy, Barnabas Poczos, and Larry Wasserman. Nonparametric estimation of renyi divergence and friends. PMLR, pages 919–927, 2014. [11] S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Math- ematical Statistics, 22:79–86, 1951. [12] David Kllberg and Oleg Seleznjev. Estimation of entropy-type integral functionals. Communications in Statistics - Theory and Methods, 45:887–905, 2016. [13] Ce Liu, Ce Liu, and Hueng-Yeung Shum. Kullback-leibler boosting, 2015. [14] Gane Samb Lo. Weak Convergence (IA). Sequences of Random Vectors. Statistics and Probability African Society, 2016. [15] David Mackay. Information theory, inference, and learning algorithms, 1995. REFERENCES 820 [16] Kevin R. Moon and Alfred O. Hero. Ensemble estimation of multivariate f-divergence. 2014 IEEE International Symposium on Information Theory, 2014. [17] Pedro Moreno, Purdy Ho, and Nuno Vasconcelos. A kullback-leibler divergence based kernel for svm classification in multimedia applications, 2004. [18] Timo Ojala, Matti Pietikinen, and David Harwood. A comparative study of texture measures with classification based on featured distributions. Pattern Recognition, 29:51–59, 1996. [19] Barnabas Poczos. On the estimation of alpha-divergences, 2015. [20] Shashank Singh and Barnabas Poczos. Generalized exponential concentration in- equality for renyi divergence estimation. PMLR, pages 333–341, 2014. [21] Kumar Sricharan, Dennis Wei, and Alfred O. Hero. Ensemble estimators for multi- variate entropy estimation. IEEE Transactions on Information Theory, 59:4374–4388, 2013. [22] Georges Valiron. Cours d’analyse mathématique. Masson et Cie, 1966.