INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL
ISSN 1841-9836, 12(4), 475-491, August 2017.

Asymptotically Unbiased Estimation of A Nonsymmetric
Dependence Measure Applied to Sensor Data Analytics and

Financial Time Series

A. Caţaron, R. Andonie, Y. Chueh

Angel Caţaron
Department of Electronics and Computers
Transilvania University of Braşov, Romania
cataron@unitbv.ro

Razvan Andonie
1. Department of Computer Science
Central Washington University, USA
2. Department of Electronics and Computers
Transilvania University of Braşov, Romania
*Corresponding author: andonie@cwu.edu

Yvonne Chueh
Department of Mathematics
Central Washington University, USA
chueh@cwu.edu

Abstract: A fundamental concept frequently applied to statistical machine learning
is the detection of dependencies between unknown random variables found from data
samples. In previous work, we have introduced a nonparametric unilateral dependence
measure based on Onicescu’s information energy and a kNN method for estimating
this measure from an available sample set of discrete or continuous variables. This
paper provides the formal proofs which show that the estimator is asymptotically
unbiased and has asymptotic zero variance when the sample size increases. It implies
that the estimator has good statistical qualities. We investigate the performance of
the estimator for data analysis applications in sensor data analysis and financial time
series.
Keywords: machine learning, sensor data analytics, financial time series, statistical
inference, information energy, nonsymmetric dependence measure, big data analytics.

1 Introduction

Statistical machine learning is based on the strong assumption that we use a representative
training set of samples to infer a model. In this case, we select a random sample of the population,
perform a statistical analysis on this sample, and use these results as an estimation to the desired
statistical characteristics of the population as a whole. The accuracy of the estimation depends
on the representativeness of the data sample. We gauge the representativeness of a sample by how
well its statistical characteristics reflect the probabilistic characteristics of the entire population.
Many standard techniques may be used to select a representative sample set [16]. However, if we
do not use expert knowledge, selecting the most representative training set from a given dataset
was proved to be computationally difficult (NP-hard) [10].

The problem is actually more complex, since in most applications the complete dataset is
either unknown or too large to be analyzed. Therefore, we have to rely on a more or less
representative training set. For example, a common statistical machine learning problem is to
estimate information theory measures (such as entropy) from available training sets. This can be

Copyright © 2006-2017 by CCC Publications



476 A. Caţaron, R. Andonie, Y. Chueh

reduced to the construction of an estimate of a density function from the observed data [21]. We
refer here only to nonparametric estimation, where less rigid assumptions will be made about
the distribution of the observed data. Although it will be assumed that the distribution has
the probability density f, the data will be allowed to speak for themselves in determining the
estimate of f more than would be the case if f were constrained to fall in a given parametric
family.

The estimation of information theory measures has an important application area - the de-
tection of dependency relationships between unknown random variables represented by data
samples. There are two information theory strategies one can adopt when studying the rela-
tionship between two random variables: the first is to measure their interdependence thought as
a mutual attribute and the second is to measure how much one system depends on the other.
In the first case we have symmetric (bilateral) measures of dependence, whereas in the second
one we have nonsymmetric (unilateral) measures. The literature review summarizes these two
strategies as follows.

Strategy I. Several symmetric dependence measure were proposed (see [20]). Among them,
the Shannon entropy based mutual information (MI), MI(X,Y ) = H(X) + H(Y )−H(X,Y ) =
MI(Y,X), which measures the information interdependence between two random variables X
and Y . Estimating MI is known to be a non-trivial task [3]. Naïve estimations (which attempt
to construct a histogram where every point is the center of a sampling interval) are plagued with
both systematic (bias) and statistical errors. Other more sophisticated estimation methods were
proposed [19], [23], [14]: kernel density estimators, B-spline estimators, k-th nearest neighbor
(kNN) estimators, and wavelet density estimators. An ideal estimator does not exist, instead
the choice of the estimator depends on the structure of data to be analyzed. It is not possible
to design an estimator that minimizes both the bias and the variance to arbitrarily small values.
The existing studies have shown that there is always a delicate trade off between the two types
of errors [3].

Strategy II. Several continuous entropy-like nonsymmetric dependence measures have been
proposed [15]. But information measures can also refer to certainty (not only to uncertainty, like
Shannon’s entropy), and probability can be considered as a measure of certainty. For instance,
Onicescu’s information energy (IE) was interpreted by several authors as a measure of expected
commonness, a measure of average certainty, or as a measure of concentration. The second strat-
egy can be illustrated by a unilateral dependence measure defined for discrete random variables
by Andonie et al. [2]: o(X,Y ) = IE(X|Y ) − IE(X), where IE(X) =

∑n
k=1 p

2
k is Onicescu’s

information energy of a discrete random variable X with probabilities pk, and IE(X|Y ) is the
conditional information energy between two variables, as defined in [18].

For a continuous random variable X with probability density function f(x), the IE is [11,18]:

IE(X) = E(f(X)) =

∫ +∞
−∞

f(x)f(x)dx =

∫ +∞
−∞

f2(x)dx (1)

where X is a random variable with probability density function f(x). In other words, IE(X)
is the expectation of the values of a density function f. For two continuous random variables X
and Y , with their joint probability density function f(x,y), we introduced the continuous version
of the o(X,Y ) measure in [1,8]:

o(X,Y ) = IE(X|Y ) − IE(X) =
∫ +∞
−∞

∫ +∞
−∞

f(x,y)f(x|y)dy dx−
∫ +∞
−∞

f2(x)dx (2)

The discrete and the continuous o(X,Y ) measure the “additional” average certainty (or in-
formation) of X occurring under the condition that Y has already or simultaneously occurred



Asymptotically Unbiased Estimation of A Nonsymmetric Dependence Measure Applied to
Sensor Data Analytics and Financial Time Series 477

(or is certain) “over” the average certainty of X when the certainty (or information) of Y is not
available. Thus, o(X,Y ) can be regarded as an indicator of the unilateral dependence of X upon
Y . As a unilateral measure, o(X,Y ) gives a different insight than the MI into the dependencies
between pairs of random variables.

The o(X,Y ) measure can easily be computed if the data sample is extracted from known
parametric probability distributions. When the underlying distribution of data sample is un-
known, the o(X,Y ) has to be estimated. More formally, we have to estimate o(X,Y ) from a
random sample x1, x2, ..., xn of n d-dimensional observations from a distribution with the un-
known probability density f(x). This problem is even more difficult if the number of available
points is small.

Contributions. In previous work [4,5], we introduced a kNN method for estimating IE and
o(X,Y ) from data samples of discrete or continuous variables. In the present paper, we give
a more detailed description of this method. For the first time, we provide the formal proofs
showing that the estimator is asymptotically unbiased and has asymptotic zero variance when
the sample size increases. This implies that the estimator has good statistical qualities. The
potential application of our results in statistical machine learning for multivariate data drives
the motivation for the present work and we refer to two real world applications. The first one is a
study of the unilateral interactions between temperature sensor data in a refrigerator room. The
second one is an analysis of the dynamics of interaction between assets and liabilities evidenced
by the historical event that matches the time series formed by the unilateral dependence measures
observed from the financial time series reported by Kodak and Apple. This evolution is captured
by our unilateral dependence measure over time.

The rest of the paper is structured as follows. In Section 2 we summarize our previous work,
we review the kNN estimation method for the continuous o(X,Y ) measure, and we derive our IE
estimator in a d-dimensional space. In Section 3 we analyze and prove the asymptotic unbiased
behavior of this IE estimator. In Section 4 we derive the o(X,Y ) estimator for the purpose
of measuring the unilateral dependence relation between a pair of random variables. Section 5
discusses applications and Section 6 contains our final remarks.

2 Background

Fitting multi-dimensional data to joint density functions is challenging. We aim to summarize
the kNN estimation technique which can be used for this problem. We will refer to our previous
results on IE and o(X,Y ) kNN estimation.

2.1 kNN estimation of the unilateral dependence measure in an unidimen-
sional space

Our goal is to approximate IE and o(X,Y ) from the available dataset, for random variables
X and Y with unknown densities, using the kNN method. The kNN estimators represent an
attempt to adapt the amount of smoothing to the “local” density of data. The degree of smoothing
is controlled by an integer k, chosen to be considerably smaller than the sample size. We define
the distance Ri,k,n between two points on the line as |xi −xk| out of n available points, and for
each xi we define the distances Ri,1,n ≤ Ri,2,n ≤ . . . ≤ Ri,n,n, arranged in ascending order, from
xi to the points of the sample.

The kNN density estimate f(xi) is defined by [21]:

f̂(xi) =
k

2nRi,k,n



478 A. Caţaron, R. Andonie, Y. Chueh

The kNN was used for non-parametrical estimate of the entropy based on the k-th nearest
neighbor distance between n points in a sample, where k is a fixed parameter and k ≤ n − 1.
Based on the first nearest neighbor distances, Leonenko et al. [13] introduced an asymptotic
unbiased and consistent estimator Hn of the entropy H(f) in a multidimensional space. When
the sample points are very close one to each other, small fluctuations in their distances produce
high fluctuations of Hn. In order to overcome this problem, Singh et al. [22] defined an entropy
estimator based on the k-th nearest neighbor distances. A kNN estimator of the Kullback-Leibler
divergence was introduced by Wang et al. [24]. Faivishevsky et al. used a mean of several kNN
estimators, corresponding to different values of k, to increase the smoothness of the estimation [9].

The kNN method works well if the value of k is optimally chosen, and it outperforms his-
togram methods [23]. However, there is no model selection method for determining the number
of nearest neighbors k, and this is a known limitation.

2.2 kNN estimation of the IE and the o(X, Y ) measure

In [5], we introduced a kNN IE estimator, and we stated in [4], without mathematical proofs,
that this estimator is consistent. By definition [12], a statistic whose mathematical expectation
is equal to a parameter is called unbiased estimator of that parameter. Otherwise, the statistic is
said to be biased. A statistical estimator that converges asymptotically to a parameter is called
consistent estimator of that parameter. In the following, we will review this result, introducing
also the basic mathematical notations used in Section 3.

The IE is the average of f(x), therefore we have to estimate f(x). The n observations from
our samples have the same probability 1

n
. A convenient estimator of the IE is:

ˆIE
(n)
k (X) =

1

n

n∑
i=1

f̂(xi). (3)

The mass probability at value xi determined by its k nearest surrounding points and stan-
dardized by the sphere volume Vk(xi) these k nearest observations occupy measures the density
of probability at value xi. The k-th nearest neighbor estimate is defined by [4], [21]:

f̂(xi) =
k
n

Vk(xi)
, i = 1...n (4)

where Vk(xi) is the volume of the d-dimensional sphere centered in xi, with the radius Ri,k,n
measuring the Euclidean distance from xi to the k-th nearest data point. The estimate (4) can
be re-written as:

f̂(xi) =
k

nVdR
d
i,k,n

, i = 1...n. (5)

given Vk(xi) = VdRdi,k,n where Vd is the radius of the unit sphere in d-dimensions, that is V1 = 2,
V2 = π, V3 = 4π3 , etc.

By substituting f̂(xi) in (3), we finally obtain the following IE approximation:

ˆIE
(n)
k (f) =

1

n

n∑
i=1

k

nVdR
d
i,k,n

. (6)

In [6], we showed how to estimate the o(X,Y ) dependence measure from an available sample
set of discrete or continuous variables and we experimentally compared the results of the kNN
estimator with the naïve histogram estimator.



Asymptotically Unbiased Estimation of A Nonsymmetric Dependence Measure Applied to
Sensor Data Analytics and Financial Time Series 479

3 Asymptotic behavior of the IE approximator

Consistency of an estimator means that as the sample size gets large the estimate gets closer
and closer to the true value of the parameter. Unbiasedness is a statement about the expected
value of the sampling distribution of the estimator. The ideal situation, of course, is to have an
unbiased consistent estimator. This may be very difficult to achieve. Yet unbiasedness is not
essential, and just a little bias is permitted as long as the estimator is consistent. Therefore, an
asymptotically unbiased consistent estimator may be acceptable.

In [4], we stated (without proofs) that the IE approximator is asymptotically unbiased and
consistent. In the this section, we provide the formal proofs for this statement.

Lemma 1. Let us consider the identically distributed random variables T (n)1 , T
(n)
2 , ..., T

(n)
n

defined by

T
(n)
i =

k

nVdR
d
i,k,n

. (7)

For a given x, the random variable Tx has the following probability density function:

hTx(y) =
f(x)

[
k
y
f(x)

]k
y2(k − 1)!

e
−k
y
f(x)

, 0 < y < ∞.

Proof: See Appendix 6. 2

Theorem 2 (Asymptotically Unbiasedness). The information energy estimator ˆIE
(n)
k (f) is asymp-

totically unbiased.

Proof:
We will find the asymptotic value of the average:

lim
n→∞

E
[

ˆIE
(n)
k (f)

]
.

From Lemma 1, for a given x, the random variable Tx has the pdf:

hTx(y) =
f(x)

[
k
y
f(x)

]k
y2(k − 1)!

e
−k
y
f(x)

, 0 < y < ∞.

The conditional expectation of T (n)1 |X1 = x, given the fixed center X1 = x is:

limn→∞E
[
T

(n)
1 |X1 = x

]
=

∫ ∞
0

yh(y)dy

=

∫ ∞
o

f(x) (kf(x))
k

(k − 1)!
·y−1−ke−

kf(x)
y dy

=
f(x) (kf(x))

k

(k − 1)!

∫ ∞
0

y−1−ke
−kf(x)

y dy.

We make the following change of variable: u = kf(x)
y

, with y ∈ (0, +∞). The corresponding
u ∈ (+∞, 0) is decreasing. We have:

du = −
1

y2
kf(x)dy



480 A. Caţaron, R. Andonie, Y. Chueh

and

y−1−k =

(
u

kf(x)

)k+1
.

We obtain:∫ ∞
0

yh(y)dy =
f(x) (kf(x))

k

(k − 1)!

∫ 0
∞
−uk+1e−udu ·

1

(kf(x))k+1

(
kf(x)

u

)2
·

1

kf(x)
.

The negative sign before uk+1 allows us to change the limits of the integral:

∫ ∞
0

yh(y)dy =
f(x) (kf(x))

k

(k − 1)!

∫ ∞
0

uk−1e−udu ·
1

(kf(x))k

=
f(x)

(k − 1)!

∫ ∞
0

uk−1e−udu =
f(x)

(k − 1)!
Γ(k) = f(x).

We take the complete expectation of the above, then we have:

limn→∞E
[
E
[
T

(n)
1 |X1 = x

]]
= limn→∞E

[
T

(n)
1

]
= E[f(x)].

Let us define Ĝ(n)(f) = 1
n

∑n
i=1 T

(n)
i , the unbiased estimator of f(x) (like in [22]). We obtain:

E
[
Ĝ(n)(f)

]
=

1

n

n∑
i=1

E
[
T

(n)
i

]
=

1

n
[E [f(x)] + · · · + E [f(x)]] = E [f(x)] .

limn→∞E
[
Ĝ(n)(f)

]
= E [f(x)] =

∫
f(x) ·f(x)dx = IE(f).

This proves that our IE estimator is asymptotically unbiased. 2

Next, we aim to prove that it has asymptotic zero variance.

Lemma 3. Given the identically distributed random variables T (n)1 , T
(n)
2 , ..., T

(n)
n defined by (7),

we have:
lim
n→∞

1

n
E
[
T

(n)2

1

]
− lim
n→∞

1

n
E
[
T

(n)
1

]2
= 0.

Proof: See Appendix 6. 2

Lemma 4. Given the identically distributed random variables T (n)1 , T
(n)
2 , ..., T

(n)
n defined by (7),

the following relation is true:

lim
n→∞

n(n− 1)
n2

(
E
[
T

(n)
1 T

(n)
2

]
−E

[
T

(n)
1

]
E
[
T

(n)
2

])
= 0.

Proof: See Appendix 6. 2

Theorem 5 (Asymptotic Zero Variance).

lim
n→∞

V ar
[
Ĝ

(n)
k (f)

]
= 0.



Asymptotically Unbiased Estimation of A Nonsymmetric Dependence Measure Applied to
Sensor Data Analytics and Financial Time Series 481

Proof: T (n)1 ,T
(n)
2 , . . . ,T

(n)
n are identically distributed.

V ar
[
Ĝ

(n)
k (f)

]
=

1

n2


 n∑
i=1

V ar
[
T

(n)
i

]
+
∑
i 6=j
i<j

2Cov
(
T

(n)
i ,T

(n)
j

)
=

1

n
V ar

[
T

(n)
1 +

n(n− 1)
n2

]
Cov

(
T

(n)
1 ,T

(n)
2

)
.

lim
n→∞

V ar
[
Ĝ

(n)
k (f)

]
= lim

n→∞

1

n
E
[
T

(n)2

1

]
− lim
n→∞

1

n
E
[
T

(n)
1

]2
+ lim
n→∞

n(n− 1)
n2

(
E
[
T

(n)
1 T

(n)
2

]
−E

[
T

(n)
1

]
E
[
T

(n)
2

])
.

From Lemma 3 and Lemma 4, we obtain:

lim
n→∞

V ar
[
Ĝ

(n)
k (f)

]
= 0.

2

Our main result is synthesized in:

Theorem 6 (Consistency). The information energy estimator ˆIE
(n)
k (f) is consistent.

Proof: This results from Theorems 2 and 5, and the following property: An asymptotically
unbiased estimator with asymptotic zero variance is consistent [17]. 2

4 The kNN o(X, Y ) estimator

Our goal is to infer o(X,Y ) from the random samples x1, x2, ..., xn. We will use the results
from Section 2.2 to deduct the kNN estimator of o(X,Y ).

First, we substitute ÎE
(n)

k (X) from eq. (6) in eq. (2):

ô(X,Y ) = ÎE
(n)

k (X|Y ) − ÎE
(n)

k (X) (8)

where:

ÎE
(n)

k (X|Y ) =
m∑
j=1

f̂(yj)ÎE
(n)

k (X|yj) (9)

and

ÎE
(n)

k (X) =
1

n

n∑
i=1

k1

nVd1(X)R
d1
i

, (10)

is an adaptation of eq. (6).



482 A. Caţaron, R. Andonie, Y. Chueh

We can write:

ÎE
(n)

k (X|yj) =
1

n

n∑
i=1

f̂(xi|yj) =
1

n

n∑
i=1

f̂(xi,yj)

f̂(yj)
=

1

n

n∑
i=1

f̂(xi,yj)

f̂(yj)
, (11)

where

f̂(xi|yj) =
f̂(xi,yj)

f̂(yj)
. (12)

From (9) and (11) we can write:

ÎE
(n)

k (X|Y ) =
m∑
j=1

f̂(yj)
1

n

n∑
i=1

f̂(xi,yj)

f̂(yj)
=

1

n

m∑
j=1

n∑
i=1

f̂(xi,yj).

The estimate of the joint probability density can be written as:

f̂(xi,yj) =
k2

pVd2(X,Y )R
d2
i,j

,

where p is the number of pairs (xi,yj), then re-write the eq. (9) as:

ÎE
(n)

k (X|Y ) =
k2

npVd2(X,Y )

m∑
j=1

n∑
i=1

1

Rd2i,j
.

Ri is the Euclidean distance between the reference point xi and its kth1 nearest neighbor, when
the points are drawn from the one-dimensional probability distribution f(x): Ri = ‖xi −xi,k1‖.
Similarly, Rj is the Euclidean distance between the reference point yj and its kth1 nearest neighbor,
when the points are drawn from the one-dimensional probability distribution f(Y ): Rj = ‖yj −
yj,k1‖. Then, Rij is the Euclidean distance between the reference point (xi,yj) and its k

th
2

nearest neighbor, when the points are drawn from the joint probability distribution f(X,Y ):
Rij =

√
(xij −xij,k2 )2 + (yij −yij,k2 )2.

The estimate of o(X,Y ) is:

ô(X,Y ) =
k2

npVd2(X,Y )

m∑
j=1

n∑
i=1

1

Rd2i,j
−

k1
n2Vd1(X)

n∑
i=1

1

Rd1i
. (13)

Although we do not have a general method to set the nearest neighbor parameter, Silverman
[21] suggested that k should be proportional to

n4/(d+4). (14)

In our case, the optimal values of k1 and k2 may not be equal, because the these two param-
eters refer to different samples. The R code for computing the estimate of o(X,Y ) is publicly
available on GitHub1.

5 Applications

There are tremendous opportunities for applications using o(X,Y ) estimation to analyze
potential dependencies between data represented by samples. We illustrate with an example
with numerical implementations and two applications using real-world data, all extracted from
our previous work and summarized here.

1https://github.com/cataron/information-energy



Asymptotically Unbiased Estimation of A Nonsymmetric Dependence Measure Applied to
Sensor Data Analytics and Financial Time Series 483

Figure 1: The kNN estimated values, where k was determined with formula (14). The theoretical
values IE(X) = 1.12, IE(X|Y ) = 1.1351, o(X,Y ) = 0.0151, IE(Y ) = 1.128, IE(Y |X) =
1.14787, o(Y,X) = 0.01987 are marked with dashed lines. The estimates converge towards their
theoretical values for an increasing number of samples, under various k values (the horizontal
axis): 2, 4, 6, 8, 10.

5.1 A specified joint probability distribution

Let us illustrate with the following joint probability density function from [6]:

fX,Y (x,y) =
6

5

(
x + y2

)
,x ∈ [0, 1],y ∈ [0, 1], (15)

which has the marginal probability density functions

fX(x) =

∫ 1
0

6

5

(
x + y2

)
dy =

6

5

(
x +

1

3

)
(16)

and

fY (y) =

∫ 1
0

6

5

(
x + y2

)
dx =

6

5

(
1

2
+ y2

)
. (17)

The conditional probability density function is:

fX|Y (x|y) =
fX,Y (x,y)

fY (y)
=
x + y2

1
2

+ y2
,

and the theoretical value of o(X,Y ) is:

o(X,Y ) = IE(X|Y ) − IE(X)

IE(X) =

∫ 1
0
f2(x)dx = 1.12

IE(X|Y ) =
∫ 1

0

∫ 1
0
fX,Y (x,y)fX|Y (x|y)dxdy = 1.1351

and

o(X,Y ) = 1.1351 − 1.12 = 0.0151.



484 A. Caţaron, R. Andonie, Y. Chueh

Similarly, we obtain IE(Y ) = 1.128, IE(Y |X) = 1.14787, and o(Y,X) = 0.01987. We
observe that o(X,Y ) < o(Y,X). The theoretical values of IE(X), IE(Y ), IE(X|Y ), IE(Y |X),
o(X,Y ), and (Y,X) are represented in Fig. 1 with dashed lines. Their kNN estimates are
represented with continuous lines, for an increasing number of data samples.

5.2 Sensor data

When studying the interaction between two random variable, why is a unilateral dependence
measure useful, and why do we not simply use the well-known MI? Let us consider two sets
of experiments, characterized respectively by random variables X and Y . The experiments run
simultaneously and interact probabilistically. Our question is which variable influences proba-
bilistically more the other one. Thus, X can be viewed as X|Y and Y can be viewed as Y |X.
While the correlation quantifies linear dependence and MI describes the degree of interdepen-
dence between two random variables, o(X,Y ) helps us understand which random variable, X or
Y , has a higher influence on the other one. If both X and X|Y are available, we can estimate
IE(X|Y ) as well as o(X,Y ), and similarly for Y and Y |X.

When the data is acquired from real world experiments or from simulators, we need to store
series of values for X and Y featuring the two phenomena running independently, as well as
measurements of values generated by the two phenomena running simultaneously, in order to
capture X|Y and Y |X. Moreover, the precision of the IE(X|Y ) and o(X,Y ) estimators increase
when more values of X|Y are available for each value of Y .

In our application, which can be found in [6], we study the evolution of the temperature
measured at the surface of two packages, X and Y , introduced in a refrigerated room. When one
package at the outside temperature is placed in the room, it is getting cold under the influence
of the air conditioning devices. The situation changes when a second package is placed in the
room because it will change the ambient conditions and the temperature may decrease slower.
Emulating the real world circumstances, we are able to measure the temperatures with various
experimental setups. When we only have one package in the room, we measure the time series
of temperatures as samples of an unconditional random variables. When we have both packages
in the room, we measure simultaneously the time series of temperatures as conditional random
values.

According to these experiments, we can assess if package X has a stronger influence on
package Y , or vice versa. This might lead to the decision to remove the sensor which is more
influenced by the other.

5.3 Financial time series

Studying the relationship between stocks is a very interesting business case for finance. This
is just an example which can be extended to all kind of financial parameters of companies.
In [7] we have presented an application using the public available data offered by Kodak and
Apple for the period 1999–2014. We analyzed the unilateral dependence between the Kodak and
Apple series in order to understand how they influence one to each other. We aim to describe
in the following our general approach for detecting unilateral dependencies between time series
variables, without specifically referring to the Kodak-Apple case.

The information energy and the unilateral dependence measure both are the result of two
random variables. The estimation becomes more precise when the number of observations is
high. When we analyze time series, we consider the series as a set of n observations xi, i = 1...n
of the random variable X. Our goal is to find synchronous relations between two time series
which have been considered as two samples of two random variables X and Y . A point xi from
the first series was paired with a point yi from the second one producing a joint observation, with



Asymptotically Unbiased Estimation of A Nonsymmetric Dependence Measure Applied to
Sensor Data Analytics and Financial Time Series 485

i = 1, . . . ,n. Therefore, from two time series with n points each we obtain n pairs (xi,yi). The
unilateral dependence o(X,Y ) between X and Y can be estimated with formula (13) based on
X and (X,Y ), while the unilateral dependence o(Y,X) can be estimated with the same formula
based on Y and (Y,X). We note that the set of pairs (X,Y ) is equal with the set of pairs (Y,X)
since for each observation xi exactly one observation yi is used.

It is interesting to study the evolution of the unilateral dependence in time. The values
o(X,Y ) and o(Y,X) can be estimated at the moment of time tm using the history of the m
observations of xi and yi between the initial moments t0 and tm. A new set of two observations
xm+1 and ym+1 allow us to re-estimate o(X,Y ) and o(Y,X).

Our model captures not only the unilateral dependencies between two simultaneous time
series, but also how these dependencies evolve in time, which can be valuable and insightful for
forecast or investment in financial research or applications.

6 Conclusion

We introduced a non-parametric asymptotically unbiased and consistent estimator of the
IE, and the unilateral dependency measure o(X,Y ) between random variables X and Y . We
estimated o(X,Y ) from available data samples using the kNN technique. In our applications,
we showed how the nonsymmetric dependence measure can provide information that cannot be
expressed by a symmetric measure. The examples illustrated in this paper are all one-dimensional
random variables, for the purpose of simplicity. The kNN estimation can be also applied on multi-
dimensional variables. For instance, the o(X,Y ) measure can be used to analyze the unilateral
dependence between bi-variate time-series data acquired from various fields.

Because of the mathematical properties proved here, the o(X,Y ) estimator works the best
for large data sets, so it is suitable for big data analytics. Our method can be applied to both
continuous and discrete variable spaces, meaning that we can use it both in classification and
regression problems.

Bibliography

[1] Andonie R., Caţaron A. (2004), An informational energy LVQ approach for feature ranking,
European Symposium on Artificial Neural Networks 2004, pages In d-side publications, 471–
476, 2004.

[2] Andonie R. (1986), Interacting systems and informational energy, Foundation of Control
Engineering, 11, 53–59, 1986.

[3] Bonachela J.A., Hinrichsen H., Miguel A. Munoz M.A. (2008), Entropy estimates of small
data sets, MATH.THEOR., 41(20), 1-20, 2008.

[4] Caţaron A., Andonie R., Chueh Y. (2013), Asymptotically unbiased estimator of the infor-
mational energy with kNN, International Journal of Computers Communications & Control,
8(5), 689–698, 2013.

[5] Caţaron A., Andonie R. (2012), How to infer the informational energy from small datasets,
Optimization of Electrical and Electronic Equipment (OPTIM), 2012 13th International
Conference on, 1065 –1070, 2012.

[6] Caţaron A., Andonie R., Chueh Y. (2014), kNN estimation of the unilateral dependency
measure between random variables, 2014 IEEE Symposium on Computational Intelligence
and Data Mining, (CIDM 2014), Orlando, FL, USA, 471–478, 2014.



486 A. Caţaron, R. Andonie, Y. Chueh

[7] Caţaron A., Andonie R., Chueh Y. (2015), Financial data analysis using the informational
energy unilateral dependency measure, Proceedings of the International Joint Conference
on Neural Networks, (IJCNN 2015), Killarney, Ireland, 1-8, 2015.

[8] Chueh Y., Caţaron A., Andonie R. (2016), Mortality rate modeling of joint lives and survivor
insurance contracts tested by a novel unilateral dependence measure, 2016 IEEE Symposium
Series on Computational Intelligence, SSCI 2016, Athens, Greece, December 6-9, 2016, 1–8,
2016.

[9] Faivishevsky L., Goldberger J. (2008), ICA based on a smooth estimation of the differential
entropy, NIPS, 1-8, 2008.

[10] Gamez J.E., Modave F., Kosheleva O. (2008), Selecting the most representative sample
is NP-hard: Need for expert (fuzzy) knowledge, Fuzzy Systems, 2008. FUZZ-IEEE 2008.
(IEEE World Congress on Computational Intelligence). IEEE International Conference on,
1069–1074, 2008.

[11] Guiaşu S. (1977), Information theory with applications, McGraw Hill, New York, 1977.

[12] Hogg R.V., McKean J., Allen T. Craig A.T. (2006), Introduction To Mathematical Statistics,
6/E, Pearson Education, 2006.

[13] Kozachenko L. F., Leonenko N. N. (1987), Sample estimate of the entropy of a random
vector, Probl. Peredachi Inf., 23(2), 9–16, 1987.

[14] Kraskov A., Stögbauer H., Grassberger P. (2004), Estimating mutual information, Phys.
Rev. E, 69, 1–16, 2004.

[15] Li H. (2015), On nonsymmetric nonparametric measures of dependence, arXiv:1502.03850,
2015.

[16] Lohr H. (1999), Sampling: Design and Analysis, Duxbury Press, 1999.

[17] Miller M., Miller M. (2003), John E. Freund’s mathematical statistics with applications,
Pearson/Prentice Hall, Upper Saddle River, New Jersey, 7th edition, 2003.

[18] Onicescu O. (1966), Theorie de l’information. Energie informationelle, C. R. Acad. Sci.
Paris, Ser. A–B, 263, 841–842, 1966.

[19] Paninski L. (2003), Estimation of entropy and mutual information, Neural Comput., 15,
1191–1253, 2003.

[20] Schweizer B., Wolff E. F. (1981), On nonparametric measures of dependence for random
variables, Ann. Statist., 9:879–885, 1981.

[21] Silverman B.W. (1986), Density Estimation for Statistics and Data Analysis (Chapman &
Hall/CRC Monographs on Statistics & Applied Probability), Chapman and Hall/CRC, 1986.

[22] Singh H., Misra N., Hnizdo V., Fedorowicz A., Demchuk E. (2003), Nearest neightboor
estimates of entropy, American Journal of Mathematical and Management Sciences, 23,
301–321, 2003.

[23] Walters-Williams J., Li Y. (2009), Estimation of mutual information: A survey, Proceedings
of the 4th International Conference on Rough Sets and Knowledge Technology, Springer-
Verlag, Berlin, Heidelberg, 389–396, 2009.



Asymptotically Unbiased Estimation of A Nonsymmetric Dependence Measure Applied to
Sensor Data Analytics and Financial Time Series 487

[24] Wang Q., Kulkarni S. R., Verdu S. (2006), A nearest-neighbor approach to estimating
divergence between continuous random vectors, Proc. of the IEEE International Symposium
on Information Theory, Seattle, WA, 242-246, 2006.

Appendix

Proof of Lemma 1

The random variables T (n)1 , T
(n)
2 , ..., T

(n)
n are identically distributed, therefore:

E
[

ˆIE
(n)
k (f)

]
= E

[
T

(n)
1

]
.

Let us denote ρr,n =
(

k
nVdr

)1
d
, with ρr,n → 0 when n →∞. For a real number r we have:

P
[
T

(n)
1 > r|X1 = x

]
= P [ρr,n > R1,k,n|X1 = x] , (18)

because:

T
(n)
1 > r ⇔

k

nVdR
d
1,k,n

> r ⇒
k

nVdr
> Rd1,k,n ⇒

(
k

nVdr

)1
d

> R1,k,n

We write the probability P
[
T

(n)
1 > r|X1 = x

]
as a binomial distribution, and approximate

it with the Poisson probability function.
The binomial distribution is given by f(k; n,p) =

(
n
k

)
pk(1 −p)n−k, and the binomial distri-

bution of the cumulative distribution function is P(X ≤ x) =
∑

xi≤x P(X = xi) =
∑

xi≤x f(x).

The probability expressed by the Poisson formula is P(x) ≈ e
−np(np)x

x!
, where p is the successful

probability out of n samples.
Using the Poisson approximation for the binomial distribution by letting the sample size

n →∞ we get:

P[ρr,n > R1,k,n|X1 = x] = 1 −P [R1,k,n ≥ ρr,n]

= 1 −
k∑
i=1

(
n− 1
i

)[
P(Sρr,n)(1 −P(Sρr,n))

]n−i−1
= P [Tx > r].

We note that:

P(Sρr,n) =

∫
Sρr,n

f(t)dt,

ρr,n =

(
k

nVdr

)1
d

,

Vρ = Vdρ
d = Vd

k

nVdr
=

k

nr
⇒ n =

k

rVρ
.



488 A. Caţaron, R. Andonie, Y. Chueh

Let n →∞, then:

lim
n→∞

nP(Sρr,n) =
k

r
lim
n→∞

P(Sρ)

Vρ
=
k

r
f(x).

Thus, from the Poisson approximation we obtain:

P [Tx > r] = 1 −
k∑
i=0

(
k
r
f(x)

)i
i!

e−
k
r
f(x).

To calculate the pdf for random variable Tx, we differentiate its cumulative density function
with respect to r:

d

dr
P [Tx ≤ r] =

d

dr
[1 −P [Tx > r]] =

d

dr

[
1 −

(
1 −

k∑
i=0

[
k
r
f(x)

]i
i!

e−
k
r
f(x)

)]

=
d

dr

k∑
i=0

[
k
r
f(x)

]i
i!

e−
k
r
f(x) =

d

dr

[
e−

k
r
f(x) +

∑
i=1

k

[
k
r
f(x)

]i
i!

e−
k
r
f(x)

]

=
kf(x)

r2
e−

k
r
f(x) +

k∑
i=1

[
k
r
f(x)

]i−1
(i− 1)!

(
−
kf(x)

r2

)
e−

k
r
f(x) +

k∑
i=1

[
k
r
f(x)

]i
i!

·
kf(x)

r2
e−

k
r
f(x)

=
kf(x)

r2
e−

k
r
f(x) −

k∑
i=1

kf(x)

r2
·
[
k
r
f(x)

]i−1
(i− 1)!

e−
k
r
f(x) +

k∑
i=1

kf(x)

r2
·
[
k
r
f(x)

]i
i!

e−
k
r
f(x)

=
kf(x)

r2
e−

k
r
f(x) −

kf(x)

r2
e−

k
r
f(x) +

kf(x)

r2
·
[
k
r
f(x)

]k
k!

e−
k
r
f(x) =

e−
k
r
f(x)f(x)

[
k
r
f(x)

]k
r2(k − 1)!

.

Then, for a given x, the random variable Tx has the pdf:

hTx(y) =
f(x)

[
k
y
f(x)

]k
y2(k − 1)!

e
−k
y
f(x)

, 0 < y < ∞.

Proof of Lemma 3

We have:

lim
n→∞

E
[
T

(n)2

1

∣∣∣X1 = x] = ∫ ∞
0

y2h(y)dy

=

∫ ∞
0

f(x) (kf(x))
k

(k − 1)!
y−ke

−kf(x)
y dy

=
f(x) (kf(x))

k

(k − 1)!

∫ ∞
0

y−ke
−kf(x)

y dy.

We make the following substitution: u = kf(x)
y

, with y ∈ (0, +∞), u ∈ (+∞, 0) decreasing,

and y = kf(x)
u

. Then du = − 1
y2
kf(x)dy and y−k =

(
u

kf(x)

)k
.



Asymptotically Unbiased Estimation of A Nonsymmetric Dependence Measure Applied to
Sensor Data Analytics and Financial Time Series 489

We obtain:

lim
n→∞

E

[(
T

(n)
1

)2
|X1 = x

]
=
f(x) (kf(x))

k

(k − 1)!

∫ 0
∞
−
(

u

kf(x)

)k
e
−kf(x)

y y2
1

kf(x)
du

=
f(x) (kf(x))

k

(k − 1)!

∫ 0
∞
−uk−2e−u

(
1

kf(x)

)k−1
du

=
f(x) (kf(x))

k

(k − 1)!

∫ ∞
0
−uk−2e−udu

=
kf(x)2

(k − 1)!
Γ(k − 1) =

k

k − 1
f(x)2.

lim
n→∞

E

[(
T

(n)
1

)2]
= lim

n→∞
E

[
E

[(
T

(n)
1

)2
|X1 = x

]]
= lim

n→∞
E

[
k

k − 1
f(x)2

]
=

k

k − 1
E
[
f(x)2

]
=

k

k − 1

[
V ar [f(x)] + (E [f(x)])

2
]

=
k

k − 1
[
V ar [f(x)] + IE(f)2

]
.

lim
n→∞

V ar
[
T

(n)
1

]
= lim

n→∞
E

[(
T

(n)
1

)2]
− lim
n→∞

(
E
[
T

(n)
1

])2
=

k

k − 1
V ar [f(x)] +

k

k − 1
IE(f)2 − IE(f)2

=
k

k − 1
V ar [f(x)] +

1

k − 1
IE(f)2.

lim
n→∞

1

n
V ar

[
T

(n)
1

]
= lim

n→∞

1

n

[
k

k − 1
V ar [f(x)] +

1

k − 1
IE(f)2

]
= 0.

Proof of Lemma 4

The relation above can be proved if the limiting covariance between T (n)1 and T
(n)
2 is zero. This

can be achieved by showing limiting probability distributions of T (n)1 and T
(n)
2 are independent.



490 A. Caţaron, R. Andonie, Y. Chueh

Given the real number r, s > 0:

P
[
T

(n)
1 > r,T

(n)
1 > s|X1 = x,X2 = y

]
= P [R1,k,n < ρr,n,R2,k,n < ρs,n|X1 = x,X2 = y]
= P

[
at least k of X3,X4, . . . ,Xn ∈ Sρr,n;x and

at least k of X3,X4, . . . ,Xn ∈ Sρs,n;y
]
.

Note: For x 6= y, ρr,n → 0 and ρs,n → 0 as n → +∞. For large enough n, we have
Sρr,n;x ∩Sρs,n;y = Φ.

We obtain:

P
[
T

(n)
1 > r,T

(n)
1 > s|X1 = x,X2 = y

]
=

∑
k≤i,j

i+j≤n−2

(n− 2)!
i!j!(n− 2 − i− j)!

[
P(Sρr,n;x)

]i [
P(Sρs,n;x)

]j ·
[
1 −P(Sρr,n;x) −P(Sρs,n;y )

]n−2−i−j
where P(Sρr,n;x) =

∫
Sρr,n;x

f(t)dt.
Recall:

lim
n→∞

nP(Sρr,n;x) =
k

r
f(x) = lim

n→∞

k

r
·
P(Sρr,n;x)

Vρ
.

Then, we have:

P
[
T

(n)
1 > r,T

(n)
1 > s|X1 = x,X2 = y

]
=

∑
k≤i,j

i+j≤n−2

(n− 2)!
i!j!(n− 2 − i− j)!

(
k

nr

)i (
k

ns

)j
·

(
P(Sρr,n;x)

Vρr,n

)i (
P(Sρs,n;x)

Vρs,n

)j
·[

1 −
1

n

(
k

r
·
P(Sρr,n;x)

Vρr,n
+
k

s

P(Sρs,n;x)

Vρs,n

)]n−2−i−j

Since:

(n− 2)!
(n− 2 − i− j)!ninj

→ 1 as n →∞[
1 −

1

n

(
k

r
·
P(Sρr )

Vρr
+
k

s

P(Sρs)

Vρs

)]n−2−i−j
→ e−(

k
r
f(x)+ k

s
f(y)) as n →∞

lim
n→∞

(
1 −

1

n

)n
= e−1.



Asymptotically Unbiased Estimation of A Nonsymmetric Dependence Measure Applied to
Sensor Data Analytics and Financial Time Series 491

we obtain:

lim
n→∞

P
[
T

(n)
1 > r,T

(n)
2 > s|X1 = x,X2 = y

]
=

n−2−k∑
i=k

n−2−k∑
j=k

ki

i!ri
[f(x)]

i ·
kj

j!sj
[f(y)]

j ·e−(
k
r
f(x)+ k

s
f(y))

=

[
n−2−k∑
i=k

ki

i!ri
[f(x)]

i
e−

k
r
f(x)

]
·


n−2−k∑

j=k

kj

j!sj
[f(y)]

j
e−

k
s
f(y)




= P [Tx > r] ·P [Ty > s],

where for given z, the random variable Tz has the pdf:

hTx =
f(z)

(
k
y
f(z)

)k
(k − 1)!y2

e
−kf(z)

y =
kkf(z)k+1

(k − 1)!yk+2
e
−kf(z)

y .

Therefore, by the theorem of independent variables:

lim
n→∞

E[T
(n)
1 T

(n)
2 ] = limn→∞

[
E[T

(n)
1 ] ·E[T

(n)
2 ]
]

⇒ lim
n→∞

Cov[T
(n)
1 ,T

(n)
2 ] = 0.