ENTROPY

ENTROPY

It is the average number of bits required to represent information without any loss.

Entropy are:

Shannon’s entropy

First order entropy

Entropy is average self-information associated with the random experiment.

For an event A we have set of outcome A(i) then the entropy for event A can be represent as:

EXAMPLE:

Consider the following sequence:

Q=1 2 3 2 3 4 5 4 5 6 7 8 9 8 9 10

Find the entropy of the sequence:

Solution:

         H(s) =0.25+0.375+0.375+0.375+0.375+0.25+0.25+0.375+0.375+0.25

    =3.22 bits/sample

MATHEMATICAL PRELIMINARIES FOR LOSSLESS COMPRESSION MODELS SELF INFORMATION

MATHEMATICAL PRELIMINARIES FOR LOSSLESS COMPRESSION MODELS

Self-Information

Suppose we have an element A which is set of outcomes of some random experiments.

If P(A) is the probability that the event is occurred then the self-information associated with the event A then event A can be represented as:

Here, b=unit of Information

So, b= bits or binary (b=2)

Here, we take calculation of Fair and Unfair Coin

Fair Coin:

Unfair Coin:

If we toss the coin 8 time. If we get only 1 time head then other 7 time we get tail.

We take the probability of Head:

We take the probability of Tail:

SORT TRICK:

Data compression

Introduction of Compression Technology

What is Data Compression?

Data compression uses different techniques to reduce number of bits required to represent an image or video in the form of bits.

it is an art or science of representing information in compressed form.

Applications :  Audio
                          Text
                          Photography / Image

                          Video

There are 2 compression techniques:

1) Lossy compression

2) Lossless compression

What is lossy compression?

Original Data ≠ Decompressed Data
Loss of Information

Its compression ratio is > 4:1
Quality of Information is Change
Applications:  Audio,Video, Image
In lossy compression data is loss after compression.

What is lossless compression?

Original Data = Decompressed Data

No loss of information

Its compression ratio is 4:1
Quality of Information is not Change
Applications: Text (applicable on text)
In lossless compression data is not loss after compression.


Measures of Performance:
1) Compression Ratio
2) Rate
3) Distortion
4) Space saving / Saving percentage

1.Compression Ratio
Compression ratio is the ratio between number of bits to represent an information in form of bits before compression.

Compression Ratio= Bits before compression /bits after compression

2.Rate
Rate is the average number of bits required to represent a single sample of information.


3.Distortion: 
It is the difference between the original data & reconstructed data after decompress. It shows the amount of information lost due to compression.


4. Space saving/ Saving Percentage
It is the reduction in size relative to the uncompressed data.

Space saving = 1- (compressed size / uncompressed size)