

































Volume 2, Issue 2

Empirical Active Information 47

Empirical Active Information
Eric Holloway
DOI: 10.33014/issn.2640-5652.2.2.holloway.4

Active information is the log ratio of the performance of a
search with (!) and without (") information about the tar-
get it is searching for, using the notation of self information
# ($) = ! log2 $,

#+ = log2
!

"
(1)

= # (") ! # (!). (2)

As proven in Bartlett (2020), active information is a canoni-
cal specified complexity, and therefore by Montañez (2018),

Pr(#+ > %) " 2!! . (3)

This is posed in an ideal setting where we know both of
these performance values.

However, in the empirical setting, although we can per-
haps take the baseline as a given value, we must infer the
performance of the search with target information by obser-
vation. In this case, we will only consider a search across
binary strings. All finite domains can be cast as binary
strings, so the proof here is general enough for all finite
domains, although perhaps not always the most convenient
representation.

We’ve observed & bits and of those ' are zeros. We can
think of this in terms of a prediction task, where ones cor-
respond to correct predictions and zeros correspond to in-
correct predictions.

For sake of demonstration we set

' " &/2, (4)

which we can always guarantee by flipping the prediction if
more than half are incorrect.

From the assumption in (4) we define the probabilities

$ = '/& (5)
( = 1 ! '/& (6)

such that $ " (.

We want to figure out what kind of prediction accuracy
we can expect on unseen data, and we want to err on the
side of caution and not be over optimistic. Stated more
formally, what is the Bernoulli distribution probability )
of ones that generated this sequence, based on a certain
confidence probability of *?

First, here is an upper bound on the binomial quotient tail
derived from the binomial cumulative distribution:

"!
#=0

"
&

+

#
$$%(&% "

"!
#=0

"
&

+

#
$#(%!# " 1 (7)

"!
#=0

"
&

+

#
" $!$%(!&% (8)

= 2%' ($) . (9)

The left hand side of (7) is derived by keeping the proba-
bility constant during the summation, instead of changing
as + is incremented. This guarantees the left hand side is a
lower bound on the binomial cumulative distribution, since
we’ve selected the smallest probability in the summation
and kept it constant.

The jump from $!$%(!&% in (8) to 2%' ($) in (9) is based on
the entropy function:

, ($) = !$ log2 $ ! ( log2 ( (10)
2' ($) = $!$(!& . (11)

Now, let us return to our goal of estimating the accuracy
on unseen data. The estimate can be wrong in two ways.

1. Too low. If so, then great!

2. Too high. If so, we want to know the probability the
estimate is too high.

So, stated more formally, we want to pick an ) for our
estimate such that any Bernoulli distribution with a smaller
probability " of generating a one, i.e. 0.5 " " < ) , will have
a probability less than * of generating ' or fewer zeros. In
other words, * is the probability we’ve over estimated our
model accuracy.

The probability * is thus:

* =
"!

#=0

"
&

+

#
(1 ! ))#)%!# "

"!
#=0

"
&

+

#
)% (12)

" 2%' ($))% (13)
log2 * " &(, ($) + log2 )). (14)

The substitution to get the right hand side in (12) is the
same sort of reasoning used to get the left hand side in (7),
except reversed. In (7) the goal was to get a lower bound

https://dx.doi.org/10.33014/issn.2640-5652.2.2.holloway.4


48 Letters and Notes

on the cumulative distribution. In (12) the goal is to get
an upper bound. So, we pick the largest probability in the
summation, )%, which is larger than (1 ! ))#)%!# because
) # 0.5.

The reduction from (12) to (13) is based on the substitu-
tion from (9). The use of the logarithm in (14) measures
the probability in bits to make the formula a bit easier to
manipulate.

Now, we want to derive ) from a predefined value for *, for
example * = 2!(. This means we need to pick a value for
) such that the right hand side of (14) evaluates to !-. If
we substitute in

) = 2!' ($)!(/%, (15)

we achieve just that result,

log2 * " &(, ($) + log2 (2!' ($)!(/%)) (16)
= &(, ($) ! , ($) ! -/&) = !-. (17)

As long as &, ($)+- < & then from (15) ) > 0.5, and we are
able to predict better than 50/50 odds. Here we can also see
why our very first assumption in (4) is necessary. If ' > &/2
then since the entropy function , ($) is symmetrical we can
substitute in a value for $ such that (15) gives us a estimate
that is higher than the observed accuracy rate, i.e. ( < ) ,
which is a contradiction since the estimate is a lower bound
and so should be lower than the observed accuracy.

Now we can finalize our concept of empirical active informa-
tion per sample by substituting in ! = ) for (1) and setting
- = 0,

#)+ = # (") ! , ($). (18)

And as at the beginning, the probability bound still applies.

Pr(#)+ > %) " 2!! . (19)

Bartlett, Jonathan (2020). “Active Information is a Spec-
ified Complexity Model”. In: Communications of the
Blyth Institute 2.2.

Montañez, George D (2018). “A Unified Model of Complex
Specified Information”. In: BIO-Complexity 2018.


	About This Journal
	The Purpose of the Journal
	Paper Submission Policies
	Other Journal Content

	From the Editors
	Annie CrawfordAnnie CrawfordMetaphor and Meaning  in the Teleological Language of Biology
	Introduction
	The History of Teleology in the Biological Sciences
	Teleological language is Essential to Biology
	If teleological language is essential to biology, then life must be teleological
	Conclusion: Life All the Way Down

	Salvador CordovaSalvador CordovaFisher's Fundamental Theorem of Natural Selection  Isn't Fundamental After All
	Introduction: The Problem of Defining Fitness
	Absolute ``Darwinian'' Fitness vs. Relative Fitness
	Mean and Variance of Relative Fitnesses
	Numerical Examples to Illustrate Fisher's Theorem
	Discussion
	Conclusion

	Eric HollowayEric HollowayTutorial: Bioinformatics Basics
	Introduction
	Genetic Code and Sequence Translation
	Sequencing and Assembly
	Accessing Data
	Finding Things With BLAST
	Summary

	Letters and Notes
	Eric HollowayYou Cannot Get Meaning From Randomness
	Jonathan BartlettActive Information is a Specified Complexity Model
	Eric HollowayEvolution in the Valley of Illusions
	Jonathan BartlettIs Active Information Applicable to Biology?
	Eric HollowayEmpirical Active Information

	News

