ENTROPY

ENTROPY

It is the average number of bits required to represent information without any loss.

Entropy are:

Shannon’s entropy

First order entropy

Entropy is average self-information associated with the random experiment.

For an event A we have set of outcome A(i) then the entropy for event A can be represent as:

EXAMPLE:

Consider the following sequence:

Q=1 2 3 2 3 4 5 4 5 6 7 8 9 8 9 10

Find the entropy of the sequence:

Solution:

         H(s) =0.25+0.375+0.375+0.375+0.375+0.25+0.25+0.375+0.375+0.25

    =3.22 bits/sample