The Theory of Information and Coding [2 ed.] 0511037147, 0521000955, 9780511037146

Table of contents :
Cover......Page 1
Half-title......Page 3
Series-title......Page 4
Title......Page 5
Copyright......Page 6
Contents......Page 7
Editor’s statement......Page 10
Foreword......Page 11
Preface to the first edition......Page 12
Preface to the second edition......Page 14
Introduction......Page 15
Problems......Page 26
Notes......Page 27
Part one Information theory......Page 29
1.1 Discrete random variables......Page 31
1.2 Discrete random vectors......Page 47
1.3 Nondiscrete random variables and vectors......Page 51
Problems......Page 58
Notes......Page 63
2.1 The capacity-cost function......Page 64
2.2 The channel coding theorem......Page 72
Problems......Page 82
Notes......Page 87
3.1 The rate-distortion function......Page 89
3.2 The source coding theorem......Page 98
Problems......Page 105
Notes......Page 107
4.1 The Gaussian channel......Page 109
4.2 The Gaussian source......Page 113
Problems......Page 119
Notes......Page 124
5 The source–channel coding theorem......Page 126
Problems......Page 134
Notes......Page 136
6.2 The channel coding theorem......Page 137
6.3 The source coding theorem......Page 145
Part two Coding theory......Page 151
7.1 Introduction: The generator and parity-check matrices......Page 153
7.2 Syndrome decoding on q-ary symmetric channels......Page 157
7.3 Hamming geometry and code performance......Page 160
7.4 Hamming codes......Page 162
7.5 Syndrome decoding on general q-ary channels......Page 163
7.6 Weight enumerators and the MacWilliams identities......Page 167
Problems......Page 172
Notes......Page 179
8.1 Introduction......Page 181
8.2 Shift-register encoders for cyclic codes......Page 195
8.3 Cyclic Hamming codes4......Page 209
8.4 Burst-error correction......Page 213
8.5 Decoding burst-error-correcting cyclic codes......Page 229
Problems......Page 234
Notes......Page 242
9.1 Introduction1......Page 244
9.2 BCH codes as cyclic codes......Page 248
9.3 Decoding BCH codes, part one: the key equation......Page 250
9.4 Euclid's algorithm for polynomials......Page 258
9.5 Decoding BCH codes, Part two: the algorithms......Page 263
9.6 Reed–solomon codes......Page 267
9.7 Decoding when erasures are present......Page 280
9.8 The (23, 12) Golay code......Page 291
Problems......Page 296
Notes......Page 306
10.1 Introduction......Page 307
10.2 State diagrams, trellises, and Viterbi decoding......Page 314
10.3 Path enumerators and error bounds......Page 321
10.4 Sequential decoding10......Page 327
Problems......Page 336
Notes......Page 343
11.1 Introduction......Page 344
11.2 Uniquely decodable variable-length codes......Page 345
11.3 Matching codes to sources......Page 348
11.4 The construction of optimal UD codes (Huffman's algorithm)......Page 351
Problems......Page 356
Notes......Page 359
12.2 Block codes......Page 361
12.3 Convolutional codes......Page 371
12.4 A comparison of block and convolutional codes......Page 373
12.5 Source codes......Page 377
Appendix A Probability theory......Page 380
Appendix B Convex functions and Jensen’s inequality......Page 384
Appendix C Finite fields......Page 389
Appendix D Path enumeration in directed graphs......Page 394
2. An annotated bibliography of the theory of information and coding......Page 398
3. Original papers cited in the text......Page 400
Index of theorems......Page 402
Index......Page 404

Citation preview

This page intentionally left blank



 




    

  
   


          
 !"#  $%& '! & ()

*& *&+ , ,   % &-




 




    

 . ) 11 12 1( 13 21 22 26 27 2) 2: 2( 23 6< 61 62 66 6. 67 6) 6: 6( 63 .< .1 .2 .6 .. .7 .) .: .( .3 7< 71 72 7. 77 7) 7: 7( 73 )< )1 )2 )6 ). )7 )) ): )( )3 :< :1 :2 :) :( (< (1 (2 (6 (. (7

/ & 0 Symmetry and separation of variables  - Permanents /  0&  / 0 * Continued fractions      0 /
% Mathematical theory of entropy    The Cauchy problem   &4 5 0&&    & &-*&& Birkhoff interpolation /  !& Graph theory 0   Field extensions and Galois theory 0   The one-dimensional heat equation

   Computation and automata  /*& 8&9 Theory of matroids   %*    &  0  &!%& Regular variation   &!*&;  '  $; Rational approximation of real functions  /*& 8&9 Combinatorial geometrics  *   =&*! Algorithmic algebraic number theory 0 -4&  0 * >& Functional equations containing several variables  5!-4   *4&"#   & Iterative functional equations  ' >4!  Factorization calculus and geometric probability  $&>&%  &   ,, Volterra integral and functional equations  $&   *  Basic hypergeometric series
 %&& Comparison of statistical experiments

 &! & Intervals methods for systems of equations  5&-*!# Exact constants in approximation theory   !   0 +& Combinatorial matrix theory  /*& 8&9 Matroid applications  # Operator algebras in dynamical systems / %& Model theory  *  ' # General orthogonal polynomials  -*&& Convex bodies     0 =>-4+# Stochastic equations in in®nite dimensions

 ?&   '&%  ! ,&  /*&   =&%& Oriented matroids
 
%   !-*& Stopping times and directed processes    Computation with ®nitely presented groups

  & Banach algebras and the general theory of  -algebras  -&!@ Handbook of categorical algebra I  -&!@ Handbook of categorical algebra II  -&!@ Handbook of categorical algebra III

 5#   &> Introduction to the modern theory of dynamical systems '  -*#; Combinatorial methods in discrete mathematics '  -*#; Probabilistic methods in discrete mathematics   * Skew Fields -* 0 & Geometric tomography &%&  #& 0  && ;&  Pade approximants 0 5?-&# Bounded arithmetic, propositional logic, and complex theory   & Geometric applications of Fourier series and spherical harmonics    In®nite dimensional optimization and control theory

  * $ Minkowski geometry   $  
  %*; Nonnegative matrices and applications 5
%& Sperner theory  ;&#;-  "    - Eigenspaces of graphs  &%&  >&&   &!@ Combinatorial species and tree-like structures      /-* Representations of the classical groups

 &*  0!%-#&   &4 Design Theory volume I 2 ed

 &-*  0 /&4& Orthonormal systems and Banach space geometry &%&
 &" -* #&+  ? + Special Functions  -- Quantum ®eld theory for mathematicians

  ;; Geometry of sporadic groups I

 &*  0!%-#&   &4 Design Theory volume II 2 ed   # Lie's Structural Approach to PDE Systems   !#   A! Orthogonal polynomials of several variables 0 +>&+ The foundations of mathematics in the theory of sets    &    &+     ;  Navier±Stokes equations and turbulence  &   &#& Geometries on Surfaces  5 #     Asymptotics and Mellin±Barnes integrals



 




    



The Theory of Information and Coding Second Edition

 0 -


California Institute of Technology

          The Pitt Building, Trumpington Street, Cambridge, United Kingdom    The Edinburgh Building, Cambridge CB2 2RU, UK 40 West 20th Street, New York, NY 10011-4211, USA 477 Williamstown Road, Port Melbourne, VIC 3207, Australia Ruiz de Alarcón 13, 28014 Madrid, Spain Dock House, The Waterfront, Cape Town 8001, South Africa http://www.cambridge.org © Cambridge University Press 2004 First published in printed format 2002 ISBN 0-511-03714-7 eBook (Adobe Reader) ISBN 0-521-00095-5 hardback

Contents

Editor's statement

page ;

Section editor's foreword

@

Preface to the ®rst edition

@

Preface to the second edition

@

Introduction >&  &

1 12 16

Part one: Information theory 1

Entropy and mutual information 11 -&&  ;>& 12 -&&  ;&- 16 -&&  ;>&  ;&- >&  &

1: 1: 66 6: .. .3

2

Discrete memoryless channels and their capacity±cost functions 21 *& -$-+B- ,!- 22 *& -*& -% *&& >&  &

7< 7< 7( )( :6

3

Discrete memoryless sources and their rate-distortion functions 61 *& & ,!- 62 *& !-& -% *&& >&  &

:7 :7 (. 31 36

;

;

Contents

4

The Gaussian channel and source .1 *& ! -*& .2 *& ! !-& >&  &

37 37 33 1&  &

112 12< 122

6

Survey of advanced topics for part one )1 !- )2 *& -*& -% *&& )6 *& !-& -% *&&

126 126 126 161

Part two: Coding theory 7

Linear codes :1 !-D *& %&&  $+-*&-# -& :2 + & &-%  q+ + &- -*& :6  % %& &+  -& $&, -& :.  % -& :7 + & &-%  %&& q+ -*& :) /&%* &! &  *& -/  && >&  &

163 163 1.6 1.) 1.( 1.3 176 17( 1)7

8

Cyclic codes (1 !- (2 *,&%& &-& , -+-- -& (6 +--  % -& (. !& -&- (7 &-% >!& -&-% -+-- -& >&  &

1): 1): 1(1 137 133 217 22< 22(

9

BCH, Reed±Solomon, and related codes 31 !- 32  -&  -+-- -& 36 &-%  -&  &D *& #&+ &H! 3.
!-I %* , $+  37 &-%  -&  "D *& %*  3) &&B  -& 3: &-% "*& &!& & $&&

26< 26< 26. 26) 2.. 2.3 276 2))

Preface 3(

*& 826129 + -& >&  &

; 2:: 2(2 232

10

Convolutional codes 1& %4&  &- &-* & - $%  &-%4>& >-* , $&&+ *& - ! >& , ;! &  &- " >& &-&&   &  && -*%&   *$& * * &&$& " #& *& - & "&+ !& "*&&   &&&  & --&>&  J&  "*-*  - >& $$& >! "*&&  *  +& $&&& >&-!& , !,J-& ,  ,  *&+   !--& +  -& $+ *& -  ! , ;&+ & &%&&% $>&   * &,   $  !-* ,K!% &&;  *& $$@  , ,!-  *& -&   *&& , $>>+    & "*&  & * - &  *& - - ,,  %& *& >- &!  * J& & >&-!& , *& & ! !$!% , &&-*   ,J-! , +& "*    $&-  %& ! *& >- &!  *& &&; & >& -
&-& * !--&&&  %;%  $&& * -*&;& * >?&-;& $&*$ *& J ,  # ianarlo ota

;

&"

  , ,    *& *& , "* "& - - !-

  & , --&    ;   !-* !$ *& $&--!$ , $*$*&   %;& &   *;% &-*%+ /& "&  *& %&! , !& * *& &-% *  %& - , $>&  &&  &-%  %  &-% ,  - >& $$-*&   +& -  -$& "+D * -- $$& , 13.( # *& >* ,  &" -*$& , *& -  *& $ *+ +& *&& * %"  %%&% &!&  * K&%% J&   & ,  & %+ &;& * >&- & $ , ! + %!%&

*& $&& %$* 8-!+ " %$*  &9   &@-&& !-  *& " $&- , - !-D -%   

*& J 8"*-*  *& !>?&- ,  "9   &&% ! , *& $"&  >&!+ , %&>L *& &- >&%  >>+ *&+ "*-* *& -*$& >&%! >+ * &-*&  ;&  !&@$&-& "+ ark 5ac General Editor &-  >>+

 
 * *& - *&+ ,  !- Bell System Tech. J. 27 813.(9 !-D 6:3B6(2L  &D -&& && +&  6(2B.&  $ . O9 *" * Pe ˆ O1 p9Q ‡ pO1 Q9 "*&&   1 2n Qˆ 2 O2 n‡19 : n 2

0.5 0.6 0.7 0.8 0.9 0.10

0.11

O>9 *" * *& $&, -& , *& %! R ˆ 2n -*& & +& &@-+ *&  & ;!& , Pe 

! % R > 1  R   *" *"  -;& *& ;%!& NN- K$$%II &%+  $ .   On k9 -&  *& && , % >+ &-& *& ,"% ;&- ! % *&+ & NN+II ;& , -& " , *& R ˆ .=:  % -&D 11&%  #& J& ->! , *& " "+  J  *& & $+ *& !>?&- %&" ! " *!& , &&-* $$&  ,  *&+ & $!>*& &-* +&

*! *  !;&+ -#"&%&  *& !H!& ,*& , ,  *&+ &+  *& > , * 13.( $$& !   *&   !;&+ -#"&%&  *&   $ $13.( ->!  *& !>?&-M &+ &;&+ & , * $$& -& NN *& - *&+ , - !-II * $;&  >&  $-&& !-& , &&-* & , &&  O &@ $&   13:6 -&- , *& #&+ $$&  *& &;&$ & , ,  *&+ F27G * " *& !*  -!* , 12 , *& .3 $$& -&  *& !* $$&& & * *&&  &9 2 O$ 19  *&  %&& , & -*& 72 O59  *& >+ + &- -*& 6 O$ 19  & & p   -& *& transition $>>+  *& crossover $>>+ , *&  . O$ 69  *& & *& binomial &; , * , ! , Pe &&;& ,!*& - coef®cient NK ONNN -*& KII9 && *& ! >& , "+ , -*% K >?&- "*! &$& ! ,  & , N >?&- ! &-+   &H!  N ON 19 ON 29 . . . ON K ‡ 19=KOK 19 . . . 2 . 1 O ,!*& $$&& , >  -&,J -& && 5!* F:G ' 19  *&& & 2k n‡1 ,,&& "+ *& -*& -! -!& &@-+ k &  % *& 2n ‡ 1  & >
-* !-* $& --! "* 2 n‡1 $>>+ pk O1 p92 n‡1 k   *& & pk O1 p92 n‡1 k  &H!  *& k $>>+ * &@-+ k , *& 2n ‡ 1 -$& , *& !-& > & &-&;&  & 7 O$ 79  &- $-&  + &   $& , + $  *   *& % ,  && && *& NN&&II  *& & $&   NN+ $ II   $+-*&-# ,!& * ,&-! -%&  !&  %&>%&

Part one ,  *&+


$+ 

1 !! , 

1.1 Discrete random variables !$$& X   -&&  ;>& *  & "*& %& R ˆ fx1  x2  . . .g  J&  -!>& & pi ˆ PfX ˆ xi g O $>> - & %+ -! $$&@ 9 *& entropy , X  &J& >+
1 HOX 9 ˆ pi % : O1:19 p i i>1

* &J && &>  *& >& , *& %*  $!$&+ &, !$&-J& , &-&+ *"&;& "& * && *& >&b &$+ >+ H b OX 9  + * *& &$+ , X  >&% &!&  >&b ! & 2 ! & -& bits O>+ %9  >&e ! & -& nats O! %9 &- , pi ˆ +  & >+L && > 119 + , R  J& *& ! O119 +  -;&%&L  * -& "& & HOX 9 ˆ ‡1 Example 1.1 & X &$&& *& !- & ,  %&  ,  , & *& R ˆ f1 2 6 . 7 )g  pi ˆ 1) , &-* i && HOX 9 ˆ % ) ˆ 2:7( > ˆ 1::3  h Example 1.2 & R ˆ f+ PfX ˆ + &$+ ,!- H 2 O p9 "*-* " &J& 
H O H  *&& %*+ ,,&& "+D HOX 9  *& &$+ , *&  ;>& X L HO p9 ˆ p % p O1 p9 %O1 p9 , < < p < 1L  HO p1  p2  . . .  pr 9 ˆ  pi % pi 1 , p   $>>+ ;&-9 h 1 2 1 Example 1.3 , *& !  && >+ A  , *&  nˆ2 On % n9 ;>& X  &J& >+ PfX ˆ ng ˆ OAn %2 n9 1 , n ˆ 2 6 . . .  *& HOX 9 ˆ ‡1 O&& > 129 h  ! ! * HOX 9 - >& *!%* ,   >! X D

&!& , *& ,"% *%

O9 *&  ! , NN, II $;& >+  >&; , X  O>9 ! NN!-&+II >! X  O-9 *& NN &II , X   *& &@ ,&" $%$* "& " -! *&& $$&& , + >! *& && *! >&   &&+ * HOX 9 &  ,- &!& *&& *%   &&$ *& - &&  "& && *&& & + $>& ,!- ,   ;>& X * *& *& $$&&  >& -!& >&" >! + HOX 9 "  , *& !+ , - !- $>&   &-* x 2 R &J& IOx9 ˆ % PfX ˆ xg *& I   &"  ;>&  HOX 9   ;&%& *& ,!- IOx9 O&& % 129 - >& &$&&  *&  ! , ,  $;& >+ *& &;& fX ˆ xg

--%  * &$& *& & $>>&  &;&  *& & ,  "& &-&;& "*&  --! -& &;& O& * --! "* $>>+ 19 $;&  ,  "*&&  !#&+ &;& $;&  ;&+ %&  ! , ,   &@ $& !$$& +! ;&  -&

1.1 Discrete random variables

13

Figure 1.2 *& ,!- IOx9

"* -! "& + NN+&  II H!& , +! #& NN/ ;&  >& 127PII  %  NNII "& +! "! *;& %& ;&+ & ,  -& !-* &@& & %&;+  &@-&&%+  $>>& ;&&+ , +! %  NN+&II +! "! *;& && !-* , "  , $&$& ;& *& -&  #& *&  & H!&  "! %&  NNII  ,&" "! %&  NN+&II  *& ;&%&  ! , ,  $;& "! >& HO p9 "*&& p ˆ P f%&  &* > 127g &;& ?! >&,& &-&;% *& -&I &$+ +! "! $>>+ >& %*+ @!L * &K&- *& ,- *     ! , !-&+ &@ >! *& "& HO p9  &H!+  &!& , * !-&+1 + ,  $& -&! "#& "&& %&  &- *& -&I "& *& "! >&- & &@& &+ >&  %* >&%  !$&- *& -& , >&%  -*& * "+ + NNII * &K&- *& ,- * *&  ;>& X &$&&% *& -&I &$+   ;&+   && HO p9 &!& *&  & , X 

  & -&& &@ $& &J& X >+ PfX ˆ &; , *& NN>II X $;& & NN>II , ,  ! J *&& --& *& @ ! $>& ;!& , HOX 9  &  , *& 4& , R Theorem 1.1 Let X assume values in R ˆ fx1  x2  . . .  xr g Then < < HOX 9 < % r Furthermore HOX 9 ˆ < iff pi ˆ 1 for some i and HOX 9 ˆ % r iff pi ˆ 1=r for all i Proof -& &-* pi  < 1 &-* & pi % p i 1  O119  > !&  ;>&  *&  NN II # ,  ;>&  +  & * *& @ ! ;!& , *& ,!- HO p1  p2  . . .  pr 9  p ˆ O p1  . . .  pr 9  %& ;& *& r 1  &  $&@ f pi > & X  Y  H!+ HOX jY 9 -& *& conditional entropy 2 , X  %;& Y   &   * &+ "& !-&  & & &   x  *& %& , X  y  *& %& , Y  &J&D pOx9 ˆ PfX ˆ xg pO y9 ˆ PfY ˆ yg pOx y9 ˆ PfX ˆ x Y ˆ yg

O1:29

pOxj y9 ˆ PfX ˆ xjY ˆ yg ˆ pOx y9= pO y9 pO yjx9 ˆ PfY ˆ yjX ˆ xg ˆ pOx y9= pOx9: O *   --+  >%!!  , >!&+ &-&+ $$ $& !>-$ " >& & , &@ $& pX Ox9 pY j X O y x9 * && " & *"&;& + "*& -! ! >& & !>!& , *& && x yL &&
@ $& 1)9 ! &J D   1 HOX jY 9 ˆ E % pOxj y9 ˆ


x y

pOx y9 %

1 : pOxj y9

O1:69

O  O169 "& >&;& *&  & -;&  "&  , ! O119D < % < 1 ˆ &*;  %;&& >+  r 3 s @ , transition probabilities O pO yjx99 *& ! >& pO yjx9 &$&& *& $>>+ * y " >& *& !$! %;& * x  *& $! &+ *& ! >& pO yjx9 ! ,+


pO yjx9 >
& pO yjx9 Example 1.4 O*& >+ + &- -*& &+ -!&  *&  !-9 && r ˆ s ˆ 2  *& %$* # #& *D

22

Entropy and mutual information

Example 1.5 O *& >+ &!& -*&9 && r ˆ 2 s ˆ 6 *& $! & >&& NN&& NN& !-* & #&+ * < ! ?  1 ! ?  *&

1.1 Discrete random variables ! $ PfY ˆ 1jX ˆ & &>& h

!$$& " * *& $!     & &&-& --%   $>>+ >! pOx9  f+ PfX ˆ xg ˆ pOx9

x 2 f& Y "*-* " &$&& *& output , *& -*& *& ? >! , X  Y  %;& >+ pOx y9 ˆ PfX ˆ x Y ˆ yg ˆ PfX ˆ xgPfY ˆ yjX ˆ xg ˆ pOx9 pO yjx9  *&

% >! , Y  pO y9 ˆ PfY ˆ yg
ˆ PfY ˆ yjX ˆ xgPfX ˆ xg x

ˆ




pO yjx9 pOx9:

x

 +

pOxj y9 ˆ pOx y9= pO y9
ˆ pO yjx9 pOx9= pO yjx99 pOx99: x9

&-& -&$%  &;&+    $! >! *&&   $ ,  ;>&D X  *& NN$!II   Y *& NN!$!II ,  *& -*& ;&&+ %;& + $ OX  Y 9 , -&&  ;>&  *&& &@     $! >! !-* * X  *& $!  Y  *& !$!D  $+ &J& *& -*&I  $>>& >+ pO yjx9 ˆ PfY ˆ yjX ˆ xg  *& " %;& + && $ OX  Y 9 ,  ;>&   $>&  *# , Y   NN+II ;& , X  *   *& &! ,  % X *!%*  -&   Example 1.6 & X ! & *& ;!& 1 2 &-* "* $>>+ 1.  & Y ˆ X 2  *& -&$%   # #& *D

2.

Entropy and mutual information

 * &@ $& X  Y & !-&&  +&   -& * Y $;&  -&>&  ! , NN, II >! X O&& > 11&;& * Y ˆ y *& -& *& ! >& pOxj y9 ˆ PfX ˆ xjY ˆ yg , J@& y &$&& *& - >! , X  %;& * Y ˆ y "& &J& *& conditional entropy , X  %;& Y ˆ yD
1 HOX jY ˆ y9 ˆ : pOxj y9 % pOxj y9 x

* H!+  &,   ;>& &J&  *& %& , Y L & ! &J& *& conditional entropy HOX jY 9   &@$&-D
HOX jY 9 ˆ pO y9 HOX jY ˆ y9 y

ˆ


y

ˆ


x y

pO y9




pOxj y9 %

x

pOx y9 %

1 pOxj y9

1  pOxj y9

 %&& & "*
H O169 *! ,  %;& $ X  Y ,  ;>& HOX jY 9 represents the amount of uncertainty remaining about X after Y has been observed Example 1.7 & *& ,"%   "*-*   $-! -& , *& >+ &!& -*& ,
@ $& 17D

1.1 Discrete random variables

27

&& pX O&; , Y &!-& ! !-&+ >! X  h /& " $&&  &-*- &

  HOX jY 9 * " >& !&,! &

Theorem 1.2 Let X  Y  Z be discrete random variables. Using obvious  notation Osee Eqs. O1299 de®ne, for each z AOz9 ˆ x y pO y9 pOzjx y9 Then HOX jY 9 < HO Z9 ‡ EO% A9: Proof

 HOX jY 9 ˆ E % ˆ




1 pOxj y9

pOx y z9 %

x yz

ˆ


z



pOz9

1 : pOxj y9


pOx y z9 1 % : pOz9 pOxj y9 x y

2)

Entropy and mutual information

 J@& z pOx y z9= pOz9 ˆ pOx yjz9   $>>+ >!   "& - $$+ 0&&I &H!+  *& & !  *& &!   
1
pOx y z9 HOX jY 9 < : pOz9 % pOz9 x y pOxj y9 z ˆ


z

pOz9 %



pOx y z9 1 ‡ : pOz9 % pOz9 pOxj y9 z x y

! pOx y z9= pOxj y9 ˆ pOx y z9 pO y9= pOx y9 ˆ pO y9 pOzjx y9:

h

Corollary ONNFano's inequalityII9 Let X and Y be random variables, each taking values in the set fx1  x2  . . .  xr g Let Pe ˆ PfX 6ˆ Y g Then HOX jY 9 < HOPe 9 ‡ Pe %Or

19:

Proof  *&& 12 &J& Z ˆ < , X ˆ Y  Z ˆ 1 , X 6ˆ Y  *& AO& ,&!&L && > 111G I &H!+ *  &&% *&!- &$& *# , HOX jY 9  *&  ! , ,  &&&  && & X -& Y  #" & "+  && & X   J && & "*&*&   X ˆ Y L , X ˆ Y  "& & & , *"&;& X 6ˆ Y  *&& & r 1 & % $>& , X && % "*&*&   X ˆ Y  &H!;&  && % *&  ;>& Z &J&  *& $,L -& HO Z9 ˆ HOPe 9  #& HOPe 9 >   * , X 6ˆ Y O* *$$& "* $>>+ Pe 9 *&  ! , ,  &&&  J ! "*-* , *& & % r 1 ;!& X *  >+ *&& 11   %Or 19 Example 1.8 /& $$+ I &H!+  *& -*& ,
@ $& 1: && rˆ PfX ˆ Y g ˆ 26 Pe ˆ 16 I >!  *! HOX jY 9 < 1 6  1 H 6 ‡ 6 % 2 ˆ 16 % 6 ‡ 26 % 62 ‡ 16 % 2 ˆ % 6 16 % 2 ˆ 1:272< > O &@ $& "*&& I &H!+ & >&& && >1119 h " -& HOX 9 &$&& ! !-&+ >! X >&,& "& #" Y  HOX jY 9 &$&& ! !-&+ ,& *& ,,&&-& HOX 9 HOX jY 9 ! &$&& *&  ! , ,  $;& >! X >+ Y *  $ H!+  -& *& mutual information >&"&& X  Y   && >+ IOX L Y 9D

1.1 Discrete random variables IOX L Y 9 ˆ HOX 9

HOX jY 9:

2: O1:.9

O 
@ $& 1: I 2 OX L Y 9 ˆ & &*& $;&  &%;& O&% 
@ $& 1: IO& & O *& ;&%&9 >+ >&;% *& !$! , *& -*& Theorem 1.3 For any discrete random variables X and Y IOX L Y 9 > && , &@ $& * *& -;-& ;OX L Y 9 , &@ $& &-
@ $& 1) *&&   &+ ;&J& ;OX L Y 9 ˆ < >! I 2 OX L Y 9 ˆ 1 >9 %
H O1.9BO1:9   $>&  $;&  &&+ &;&  $  ,- >! !! , D IOX L Y 9 ˆ IOY L X 9 IOX L Y 9 ˆ HOY 9

O1:(9 HOY jX 9

IOX L Y 9 ˆ HOX 9 ‡ HOY 9

O1:39 HOX  Y 9

O1:1+ HOX  Y 9 ˆ




pOx y9 %

x y

1 : pOx y9

O1:119

*& $, , *&& &*$ & &,  > 11. *&+ - >& &+ & & >&& >+ & , *& '& % *"  % 17    ,!,! &@&-&  %;& ,  &$& , &-* , *& &  $& >+ % 17  &@ $&
H O1(9 &@$&& *& NN !!+II , !! ,  L HOX  Y 9 ˆ HOX 9 ‡ HOY jX 9 >&- & NN! !-&+ >! X  Y  *& ! , ! !-&+ >! X  ! !-&+ >! Y  -& X  #"II    " , "& & %;& *&&  ;>& X  Y  Z "& &J& *& !! ,  IOX  Y L Z9 ONN*&  ! , ,  X  Y $;& >! ZII9 %!+ "*
H O1:9 >+

Figure 1.5

& - '& %

,
H O1.9  O1(9BO11+ *&  $>>& pO yjx9    2 >+ *&  $>>& pOzj y9 ˆ pOzjx y9 /& *;& &+ >&;& * %;& + $ ,  ;>& OX  Y 9   $>&  &;&    "* X  *& $!  Y  *& !$! "&;&    !& * ,

6
& *&& &@  $ ,  I !-* * X  Y  Z *;& *& &*$ , % 1) &&   -& *  &-&+  !,J-& - , *  * OX  Y  Z9 ,   #; -* *  pOzj y9 ˆ pOzjx y9 O& Z &$&  X + *!%* Y 9 " &I ! & * OX  Y  Z9   #; -*   % 1) *& >+

*&& 1. IOX L Z9 < IOX  Y L Z9  -& OX  Y  Z9   #; -* IOX  Y L Z9 ˆ IOY L Z9 &-& IOX L Z9 < IOY L Z9 " , OX  Y  Z9   #; -*   O Z Y  X 9 O&& > 1179  *&-& IOX L Z9 < IOX L Y 9 -& *   &@& &+  $ , *&&- $$ &+ , #; -* "& $+    *&&  Theorem 1.5 If OX  Y  Z9 is a Markov chain, then  IOX L Y 9 IOX L Z9 < IOY L Z9:

h

&,&% %  % 1) "& J *  I &  NN&#II ,  , *&  I & && - O& , Y   &J& ,!- , X  Z  &J& ,!- , Y 9 "& - *# , *& -&  % 1)   # , $-&% -J%! @-+ *&& 17 + *  $-&% - + &+ , M O   $ %&&4 , * &&
H O11799 Example 1.9 & X 1  X 2  X 6 >& &$&&  ;>&L *& OX 1  X 1 ‡ X 2  X 1 ‡ X 2 ‡ X 6 9   #; -*   IOX 1 L X 1 ‡ X 2 ‡ X 6 9 < IOX 1 L X 1 ‡ X 2 9 O&& > 11)  1639 h Example 1.10  % 1) ! & * X  &->& >+ PfX ˆ *  I & >+ + &- -*& "* & $>>+ p *& IOX L Y 9 ˆ 1

H 2 O p9

IOX L Z9 ˆ 1

H 2 F2 pO1

> p9G

>:

*&& " ,!- & $&  ,"D O  &@& , *

Figure 1.6  ,  *&I ;&" ,  #; -*

1.1 Discrete random variables

61

&@ $& && > 11(  12! *& -;&@+ , IOX L Y 9 "*&   ;&"&   ,!- &*& , *& $! $>>& pOx9  , *&  $>>& pO yjx9 Theorem 1.6 IOX L Y 9 is a convex \ function of the input probabilities pOx9 Proof /& *# , *&  $>>& pO yjx9  >&% J@&  -& " $!  ;>& X 1  X 2 "* $>>+ >! p1 Ox9  p2 Ox9 , X I $>>+ >!   -;&@ - > pOx9 ˆ á p1 Ox9 ‡ â p2 Ox9 "& ! *" * áIOX 1 L Y1 9 ‡ âIOX 2 L Y2 9 < IOX L Y 9 "*&& Y1  Y2  Y & *& -*& !$! -&$%  X 1  X 2   X &$&-;&+   * -& *& ,"% $! "*-* !& >;!  **D áIOX 1 L Y1 9 ‡ âIOX 2 L Y2 9 IOX L Y 9
pO yjx9
pO yjx9 ˆ á p1 Ox y9 % â p2 Ox y9 % ‡ p1 O y9 p2 O y9 x y x y

O&&
H:O1::99


pO yjx9 Fá p1 Ox y9 ‡ â p2 Ox y9G % pO y9 x y ˆá


x y

p1 Ox y9 %


pO y9 pO y9 p2 Ox y9 % ‡â : p1 O y9 p2 O y9 x y

O1:129

/& " $$+ 0&&I &H!+  &-* , *& >;& !   &@ $&

pO y9 pO y9 p1 Ox y9 % p1 Ox y9 < % : p p O y9 1 1 O y9 x y x y

62

Entropy and mutual information

!


x y

p1 Ox y9


pO y9
pO y9 p1 Ox y9 ˆ p1 O y9 p1 O y9 x y ˆ


pO y9 . p1 O y9 p1 O y9 y

ˆ 1: &-& *& J !

 O1129  < &   ;>& >!& --%  PfX ˆ ig ˆ pi 

*& IOX L X 9 ˆ HOX 9 ˆ HO p1  p2  . . .  pr 9 *& &! " ," ,

*&& 1) h Theorem 1.7 IOX L Y 9 is convex [ in the transition probabilities pO yjx9 Proof && *& $! $>>& pOx9 & J@& >! "& & %;& " & ,  $>>& p1 O yjx9  p2 O yjx9   -;&@ - > pO yjx9 ˆ á p1 O yjx9 ‡ â p2 O yjx9   &H!&  *" * IOX L Y 9 < áIOX L Y1 9 ‡ âIOX L Y2 9

O1:169

"*&& Y  Y1  Y2 & *& -*& !$! -&$%  *&  $>>& pO yjx9 p1 O yjx9  p2 O yjx9 % !% >;!  *& ,,&&-& >&"&& *& &,  %* & , O1169  O&&
H O1799
pOxj y9 Fá p1 Ox y9 ‡ â p2 Ox y9G % pOx9 x y
x y

ˆá


x y

*& J !

á p1 Ox y9 % p1 Ox y9 %

p1 Oxj y9 pOx9


x y

â p2 Ox y9 %

p2 Oxj y9 pOx9


pOxj y9 pOxj y9 p2 Ox y9 % ‡â : O1:1.9 p1 Oxj y9 p 2 Oxj y9 x y

 O11.9  >+ 0&&I &H!+

1.2 Discrete random vectors  
pOxj y9 < á % p1 Ox y9 p1 Oxj y9 x y ˆ á %


x y

ˆ á %


y

 + *& &- !

66

 pOxj y9 p1 O y9

p1 O y9 ˆ &"&&  $ ,  ;>&   *  ;>&  * &- "& " %&& 4& *& &J  &J& HOX9 HOXjY9  IOXL Y9 "*&& X  Y & >+  ;&- ! $ , ;&"  *   ;&- X ˆ OX 1  X 2  . . .  X n 9  ?!  J&  ,  ;>& X i  *& >! , X O*& ? >! , X 1  X 2  . . .  X n 9  *& ,!- pOx1  x2  . . .  xn 9 ˆ PfX 1 ˆ x1  X 2 ˆ x2  . . .  X n ˆ xn g "*&& &-* xi   *& %& , X i  %-&  *& &J  &- 11 *! -;-& *& && * HOX 9 HOX jY 9 IOX L Y 9 &$& +  *& >! ,!- pOx9 pO yjx9 &-    + "+  *& ,- * *& ;!& ! & >+ X  Y & & ! >& &-& "& -  &&+ &@& *&& &J  >+  ;&-L , &@ $& *& &$+ , X ˆ OX 1  . . .  X n 9  &J& 
1 HOX9 ˆ pOx9 %  pOx9 x "*&& *& !   &@&& ;&  ;&- x  *& %& , X  >;!+ *&&  11B1: &  !&

*& %&&4 , *&& 17  >+  ;&- *  $-!+  $ $$- "*-* "& " -! & *& & ,  - !- +& *"  % 1: O-, % !  - !- +& 9 $$+% *&& 17  *& !> #; -* OU X V9 "& %& IOUL V9 < IOXL V9  + IOXL V9 < IOXL Y9 &-& , *&  ;>& , % 1: IOUL V9 < IOXL Y9:

O1:179

* &!  -& *& data-processing theorem & >!+  + * *& ,  $-&% O*& "# & >+ *& &-&  &-& , % 1:9 - + &+ , M  + , &@ $& * *& + -*& !$! Y  % 1: - & ,  >! *& !-& &H!&-& U * & *& &-&I & & V O/*& *  !& *&&-+ *&  $-&% , *& &-&  &;&*&& &H!&  && * ,  !>&9 /& " - &   $ , &H!& ;;% IOXL Y9  n iˆ1 IOX i L Yi 9 "*&& X ˆ OX 1  X 2  . . .  X n 9  Y ˆ OY1  Y2  . . .  Yn 9 &  $ , n &  ;&-

Theorem 1.8 If the components OX 1  X 2  . . .  X n 9 of X are independent, then IOXL Y9 >

n


IOX i L Yi 9:

iˆ1

Figure 1.7 %&& -

!- +& 

1.2 Discrete random vectors

67

Proof &% E && &@$&-  *& ?  $& $-& , X  Y "& *;&   pOxjy9 IOXL Y9 ˆ E % Osee Eq: O1:799 pOx9   pOxjy9 ˆ E %  pOx1 9 pOx2 9 . . . pOxn 9 -& X 1  X 2  . . .  X n & ! & &$&&  *& *& *   n n

pOxi j yi 9 IOX i  Yi 9 ˆ E % pOxi 9 iˆ1 iˆ1 

 pOx1 j y1 9 . . . pOxn j yn 9 ˆ E % : pOx1 9 . . . pOxn 9

&-& n


IOX i  Yi 9

IOXL Y9

iˆ1



 pOx1 j y1 9 . . . pOxn j yn 9 ˆ E % pOxjy9   pOx1 j y1 9 . . . pOxn j yn 9 < % E ˆ< pOxjy9

>+ 0&&I &H!+ -& *  &@$&- 

pOx y9f


g ˆ pOx1 j y1 9 . . . pOxn j yn 9 pOy9 xy

xy

ˆ 1:

h

Example 1.11 & X 1  X 2  . . .  X n >& &$&& &-+ >!&  ;>& "* -  &$+ H  & ð >&  $& ! , *& & f1 2 . . .  ng  & Yi ˆ X ðOi9  *& IOXL Y9 ˆ nH >!  IOX i L Yi 9 ˆ kH "*&& k  *& ! >& , J@& $ , ð *  *& ! >& , &%& i "* ðOi9 ˆ i  $-! , ð *  J@& $ ,  &@ $& , ðOi9  i ‡ 1 O  n9 *& IOX i L Yi 9 ˆ < O&& > 1269 h , "& *# , OY1  Y2  . . .  Yn 9  *& n !$! ,  + -*& "*& *& $! & X 1  X 2  . . .  X n  *&& 1( & ! * , *& $! & &$&& Y $;& & ,  >! X * *&   ! ,

6)

Entropy and mutual information

,  $;& >! &-* X i >+ *& -&$% Yi  *& &@ *&& " & ! * , "& $ *& ! $ , &$&&-& >! *& X i  ! & & * *& OX Y9 -*&  & +& *  pO y1  . . .  yn jx1  . . .  xn 9 ˆ

n 

pO yi jxi 9

O1:1)9

iˆ1

*& !  H!& ,,&&M Theorem 1.9 If X ˆ OX 1  . . .  X n 9 and Y ˆ OY1  . . .  Yn 9 are random vectors and the channel is memoryless, that is, if O11)9 holds, then IOXL Y9
 1269 h Corollary If X ˆ OX 1  X 2  . . .  X n 9 then HOX9
&  ;&- "*-* - ! &  !-!>& ! >& , ;!& - $&&+ %! & & , * !>?&- $;&  >& ;&+ ,J-!   "& * -& !&;& "*  ,&" && &+ ,-  &,& *& &&& && &&"*&&7 , &

 "& * && *& &$+ HOX 9 - >& &J& ,  -&&  ;>& >!  "+ ! !  >& J&M "&;& *& &J , *& !! ,  >&"&&  $ X  Y ,  ;>& O ;&-9 - >& %&&4&  *& -&& -&   !-* & &&%  !&,! "+ *& #&+  * %&&4  *&  ,  discrete quantization ,   ;>& X  , X    ;>& "* >! ,!- FOx9 ˆ PfX < xg  , fSi  i ˆ 1 2 . . .g ˆ P   $) , *& & & R   J&  -!>& ! >& , &>&%!& &!>& !>& *& quantization of X >+

6(

Entropy and mutual information

P && >+ FX G p  >+

&&+ >+ FX G  *& -&& 

;>& &J&

PfFX G ˆ ig ˆ PfX 2 Si g  ˆ dFOx9: Si

, X  Y &  $ ,  ;>& "& &J& *& IOX L Y 9  IOX L Y 9 ˆ !$ IOFX G P L FY G Q 9 PQ

!! ,  O1:1:9

"*&& *& NN!$II  #& ;&  $ , $ P  Q , *& & &  + , X ˆ OX 1  . . .  X r 9  Y ˆ OY1  . . .  Ys 9 &  $ ,  ;&- IOXL Y9 ˆ !$ IOFXGL FYG9 O1:1:99 "*&& *& !$& !  #& ;&  $ FXG ˆ OFX 1 G . . .  FX r G9 FYG ˆ OFY1  . . .  FYs G9 , *& - $&  ;>&

$ P1    >&  &J& & , P2 , &;&+ &  P1   !>& ,  & &  P2  -& * , P1   &J& & , P2  FX G P 2   && - ,!- , FX G P 1    >+ *&& 17 O && > 1229 IOFX G P 2 L Y 9 < IOFX G P 1 L Y 9 , +  ;>& Y  * >&; *" * *& NN!$II  &J O11:9   ,-  # ,    *& $ P  Q >&- & -&%+ J& &&   &+  *" * , P1  P2  Q1  Q2 & $  FX G1 FX G2  FY G1  FY G2 & *& -&$% -&&  ;>& *&& &@ $ P  Q -&$%   ;>& FX G  FY G !-* * IOFX GL FY G9 > IOFX G i L FY G i 9

i ˆ 1 2:

O1:1(9

" , X  Y & &+ -&& "* %& fx1  x2  . . .g  f y1  y2  . . .g >+ &&-%  X $  *  & - & * & xi    Y $  *  & - & * & yj  -&+ IOFX GL FY G9 ˆ IOX L Y 9
H!+ -&+  &J& & -! -&& IOFX GL FY G9   &J O11:9 &!-&  ! && &J &J O11:9 , IOX  Y 9  ,+  $& >!    -;&& , -!+ - $!% IOX  Y 9 ,  %;& $ ,  ;>& &+ "& "! #&  *;&  , ! , IOX  Y 9 %! 
H O179 O1)9  O1:9 ;;% &% *& * !  >! !,!&+  %&&  !-* , ! &@: "&;& , X  Y & !,J-&+  * !-* , !  &@  "& * " && & ! ! & * X  Y *;&  -!! ? &+ & * *&&

1.3 Nondiscrete random variables and vectors

63

&@  -!! &%;& ,!- pOx y9 &J& , $ Ox y9 , & ! >& !-* * , A  B & &;  *& & &   PfX 2 A Y 2 Bg ˆ pOx y9 dx dy: B A

 * ! *& ;! && , X  Y & %;& >+ 1 1 pOx9 ˆ pOx y9 dy qO y9 ˆ pOx y9 dx 1

1

 *& - && >+ pOx y9 pOxj y9 ˆ  qO y9

pO yjx9 ˆ

& ! " ! & * *& &% 1 1 hOX 9 ˆ dx pOx9 % pOx9 1 1 1 hOX jY 9 ˆ pOx y9 % >* &@

1

1

pOx y9 : pOx9

O1:139 1 dx dy pOxj y9

O1:2+  &


, x 1 , x< , x1 ,


>&  -!>& & , $ !-* * Äxi ˆ xi x 1 J& å1 , Äxi , å2 ,  i  + -*& O yj 9 !-* * å1 , Ä yj , å2  & FX G && *& H!4  , X >& , *& $ , *& & & -% , *& *, $& &; Fxi 1  xi 9  & FY G && *& H!4 , Y -&$ %  *& $ -% , *& &; F y j 1  yj 9 /& " !-& *& -& D  xi pOi9 ˆ PfFX G ˆ ig ˆ pOx9 dx qO j9 ˆ PfFY G ˆ jg ˆ

xi

1

yj

1

 yj

qO y9 dy

pOi j9 ˆ PfFX G ˆ i FY G ˆ jg ˆ

 yj  xi yj

1

xi

pOx y9 dx dy 1

pOij j9 ˆ PfFX G ˆ ijFY G ˆ jg ˆ pOi j9=qO j9:

.
& -!! >+ *& ;!& *&& , &%(

&

pOi9 ˆ Äxi pOsi 9 qO j9 ˆ Ä yj qOtj 9

O1:219

,  & si 2 Fxi 1  xi G tj 2 F y j 1  yj G  + >+ ;#%  " &  &;!& *&& ( "& *;& pOi j9 ˆ

 yj  xi yj

1

xi

pOxj y9qO y9 dx dy 1

ˆ pOsij jtij 9

 yj  xi yj

1

xi

qO y9 dx dy 1

ˆ pOsij jtij 9Äxi Ä yj qOtj 9

O1:229

,  & $ Osij  tij 9  Fxi 1  xi G 3 F y j 1  yj G &-& pOij j9 ˆ Äxi pOsij jtij 9:

O1:269

" >+
H O1.9 IOFX GL FY G9 ˆ H OFX G9 HOFX GjFY G9 ˆ  1 i j pOi j9 % pOij j9  + O1219 "& *;& HOFX G9 ˆ




Äxi pOsi 9 %

i



i pOi9 % pOi9


1 1 Äxi pOsi 9 % ‡ : pOsi 9 ÄOx i9 i

1

O1:2.9

 O12.9 *& J !   $$@   *& &% , hOX 9 O&& O11399   ,   &!%* å2  " -;&%& *& &- ! -;&%& O  ! & * % å1 1 9 >&-!& Äxi . å1   + HOFX GjFY G9 ˆ




Äxi Ä yj pOsij jtij 9qOtj 9 %

i j

‡


i

Äxi %

1 pOsij jtij 9

1
Ä yj pOsij jtij 9qOtj 9: Äxi j

O1:279

*& J !  O1279  *& &% ,  &$,!- $$@   pOx y9 % pOxj y9 1    , å2   &!%*  " $$@ & hOX jY 9

*& &- !  &-  *& &- !  O12.9 -& >+ O1229

1.3 Nondiscrete random variables and vectors  
1
yj xi Ä yj pOsij jtij 9qOtj 9 ˆ pOx y9 dx dy Äxi j y j 1 xi 1 j ˆ

.1

1 . Äxi pOsi 9: Äxi

*! "& J+ > IOFX GL FY G9 ˆ




Äxi pOsi 9 %

i


i j

1 pOsi 9

Äxi Ä yj pOsij jtij 9qOtj 9 %

1 : pOsij jtij 9

O1:2)9

!  "& *;& &+ >&;& *& J !  O12)9   $$@   hOx9  *& &-  hOX jY 9  >+ #% *& X  Y $ -&%+ J& &  å2 !  1669  * & hOX 9  !!+ -& *& differential entropy , X   %!*  , *& +  absolute entropy

  *& -&& -& "& - &+ &@& *& &J , IOX L Y 9  hOX 9   ;&-  &J& IOXL Y9  %&& ?! #& -&& H!4 , &-* - $& &$&+  hOX9 ! & *& &@&-& ,   n & -!! &+ pOx9  &J& hOX9 ˆ pOx9 % pOx9 dx Example 1.13 & X ˆ OX 1  X 2  . . .  X n 9 "*&& *& X i & &$&&

.2

Entropy and mutual information

! O 9  ;>& "* *& &+ , X  *& ,!- gOx9 ˆ

n  iˆ1

O2ðó 2i 9

1=2

& ì i  ;-& ó 2i  *& 

&@$

Oxi

ì i 92

2ó 2i

*& ,,&& &$+ , X  &+ - $!&D  1 hOX9 ˆ gOx9 % dx gOx9   n 
Oxi 2 1=2 ˆ gOx9 % O2ðó i 9 &@$ iˆ1

ˆ

n 
iˆ1

O"*&& gi Oxi 9 ˆ O1= ˆ O-&



 g i Oxi 9

1 2

%O2ðó 2i 9 ‡

 2ðó 2i 9 &@$F Oxi

n
iˆ1

2 1 2F %O2ðó i 9

Oxi

 :

ì i 92

O1:2:9



2ó 2i

ì i 92

dx



2ó 2i

dxi 

ì i 92 =2ó 2i G9

‡ 1G

 gi Oxi 9 dxi ˆ 1  g i Oxi 9Oxi ì i 92 dxi ˆ ó 2i 9 n ˆ % 2ðeOó 21 . . . ó 2n 91= n : 2

 *& $&- -& n ˆ 1 "& *;& hOX 9 ˆ 12 % 2ðeó 2 :

h

   &&%   $ ,- *  %  n &  ;>& "* %;& ;-& &$&& !  ;>& *;& *& %& ,,&& &$+ Theorem 1.11 If X ˆ OX 1  X 2  . . .  X n 9 has density pOx9 and if EFOX i ì i 92 G ˆ ó 2i  i ˆ 1 2 . . .  n then hOX9 < On=29 % 2ðeOó 21 ó 22 . . . ó 2n 91= n  with equality iff pOx9 ˆ gOx9 Osee De®nition O12:99 almost everywhere  *& % &+ pi Ox9 , X i J& pi Ox9 dx ˆ 1 Proof + *+$*& pi Ox9Ox ì i 92 dx ˆ ó 2i  &-&  ," , *& - $! ,
@ $& 116 *

1.3 Nondiscrete random variables and vectors  1 n pOx9 % dx ˆ % 2ðeOó 21 . . . ó 2n 91= n : gOx9 2

.6

*! , Y &&  n &  ;&- >!& --%  *& ! &+ gOx9  gOx9 hOX9 hOY9 ˆ pOx9 % dx: pOx9 + 0&&I &H!+ *"&;& *& & &%   < % gOx9 dx ˆ  16.  1679 h ;% &!& >&K+ *!%* *& &&% $- , ,,&& &$+ "& &!  !  --& !! ,  !    *" "*  & &!& , -&>+ * *&&  16 1. 17 1(  13 &  !&  *& %&& -& *&&  16 1(  13 $&&  &! ,J-!&L *&& &  & & #  *& $, Theorem 1.3. *& ,- * IOXL Y9 > < ,"  &&+ , *& ,- * IOFXGL FYG9 > < , + -&& H!4 FXG FYG &;& , FXG  FYG & &$&&  & FXG  FYG   IOFXGL FYG9 ˆ < ,  H!4L *&-& IOXL Y9 ˆ < >+
H O11:9 + , X  Y & &$&&   $>&  J H!4 FXG  FYG "*-* &  &$&& O&& > 16)9   IOXL Y9 > IOFXGL FYG9 . &; & >;& "& *;& * , X 1  . . .  X n & &$&&  & FX 1 G FX 2 G . . .  FX n G   >+

*&& 1( IOFXGL FYG9 >

n


IOFX i GL FYi G9:

O1:2(9

iˆ1

!  *& H!4 , *& FX i G  FYi G >&- & -&%+ J& *& &, & , O12(9 $$-*& IOXL Y9  *& %* & $$-*& n iˆ1 IOX i L Yi 9 Theorem 1.9. && *&  $>&   $$$&+ %&&4& *& *+$*& pOyjx9 ˆ

n  iˆ1

pO yi jxi 9

O1:239

..

Entropy and mutual information

O&&
H O11)99   * $$&+ "! & ! , J&   !+ , - $>>+ >! O&& &% && F.G '  }'1&  >+ de®ning -*&  >& & +& , O1239 * , &;&+ H!4 , *& $!   !$! , *& -*& * -  !!+ &+  ;&,+ O&& > .2:9  * *& ;%& * *&& 13  &&+  $& IOFXGL FYG9
&  $;% * IOX  Y L Z9 > IOY L Z9  %&& -& , &;&+ H!4 , X  Y  Z "& %& IOFX G FY GL F ZG9 > IOFY GL F ZG9 &-+ , *&& 1. "&;&     &+  $;& *& &- $ , *&& 1. & * IOX  Y L Z9 ˆ IOY L Z9 ,, OX  Y  Z9 ,   #; -* *& $>&  * , OX  Y  Z9   #; -*  , FX G  F ZG & $ , X  Z  +  >& $>&  J  $ FY G , Y !-* * OFX G FY G F ZG9   #; -* O&& > 16:B1639 /& $&,&   >!& *& && "*  && $, >! &,& &  #& F2.G *$& 6 "*&& *& -& &!  $;& "* ,! %3

Problems 1.1 *" * *& -;& < % 1=< ˆ < $&  &J O119 , HOX 9  ,-& , "& #& &*& , *& ,"% " ! $D O9 HOX 9   -!! ,!- , *& $>>+ ;&- p ˆ O p1  p2  . . .9 O>9 , X  Y &  ;>& &J&  *&  &  $& $-&  , X ˆ Y & *& HOX 9 ˆ HOY 9 1.2 , X  &J&  
@ $& 16 *" * HOX 9 ˆ ‡1 1.3 & X >&  -&&  ;>&  & f >&  &;!& ,!- &J&  *& %& , X  *" * HOX 9 > HF f OX 9G "* &H!+ ,, f  &  &  *& & fxD PfX ˆ xg . >+ >! O p1  p2  . . .  pn 9   &%& m < < m < n

Problems

.7

m &J& qm ˆ 1 jˆ1 pj  *" * HO p1  . . .  pn 9 < HO p1  . . .  pm  qm 9 ‡ qm %On m9 /*& & &H!+ *P 1.7 & f Ox9 >& + ,!- &J& ,  x > 1 , X   -&&  ;>& "*  n %& R ˆ fx1  . . .  xn g &J& *& f entropy , X >+ H f OX 9 ˆ iˆ1 pi f O1= pi 9 "*&& pi ˆ PfX ˆ xi g O+ &$+D f Ox9 ˆ % x:9 O9 , f Ox9  -;&@ \ J *& >& $>& !$$& >!  HOX 9 * &$& +  n *" >+ &@ $& * *& $>>+ >! * -*&;& * @ ! +  >& !H!& O>9 , f Ox9 ˆ % x=x *" * H f OX 9 , %Oe9=e F f Ox9   -;&@ \G O-9 /# *&  $ O>9  *" *  ,- H f OX 9 < %O69=6 "* &H!+ ,, &@-+ *&& , *& pi I &H! 16  *& & & &  &H!&-& , &%;& & ! >& !-* * *& ! ZOs9 ˆ n e a n s -;&%& ,  !,J-&+ %& s O &@ $& , *&& & + J&+ +  aI * -  J&9  &-* á . < &J& ÖOá9 ˆ !$ f HOp9D pn an ˆ ág *" * ÖOá9  %;& $ & -+ >+ ÖOá9 ˆ % ZOs9 sZ9Os9= ZOs9 "*&& Z9Os9= ZOs9 ˆ á F HintD ,  an s pn an ˆ á &J&  &" $>>+ = ZOs9   >! >+ qn ˆ e $$+ 0&&I &H!+  *& ! pn %Oqn = pn 9G , a ˆ Oa1  a2  . . .  aN 9 * + J&+ + - $& *" *  %$* , ÖOá9 ;&! á # #& *D

;& *& -& , *& $ A B  C 1.9 & p ˆ O p1  p2  . . .9>&  -!>& $>>+ >! *  pn > < , 1 n ˆ 1 2 . . .   1 nˆ1 pn ˆ 1 *" * , nˆ1 pn % n -;&%& *& HOp9  J& ;&&+, HOp9  J&  p  & O& p1 > p2 >


9 *" * pn % n , 1 *" >+ &@ $& * *& & &   !& , *& ! $ , -+  $$& 1.10 & X  Y >& *&  ;>& &J& 
@ $& 1) *" * ;OX  Y 9 ˆ   *&   , "*-* X   + ;& , Y  1.11  &-&+  !,J-& -  *& ? $>>+ >! pOx y z9 , &H!+  *  *&& 12 % * &! , + r > 2  + ;!& < < Pe < 1 -!-  ;>& X  Y "* %& f1 2 . . .  rg , "*-* I &H!+ O*& -+  *&& 129  *$

.)

Entropy and mutual information

1.12 *" * I &H!+  $& >*  !$$& >!   "& >!  Pe  &  , HOX jY 9 &$& *& !$$& >! *&!-+ 1.13 !$$& X  Y &  ;>& "*-* >* ! & ;!&   J& %!$ &J& Z ˆ Y X  !% *&& 12 *" * HOX jY 9 < HO Z9 , X  Z & &$&& *" * HOX jY 9 ˆ HO Z9 1.14 '&,+
H O179BO11&   ;>& !, + >!&  f& Y ! % *& ;!& f>+ p  &H!;&   %&  "* & $>>+ 12F1 O1 2 p9 n G  *&-& *  n!1 IOX < L X n 9 ˆ &"&& X  Y %;& ZII9  &J& >+ IOX L Y j Z9 ˆ


x yz

pOx y z9 %

pOx yjz9  pOxjz9 pO yjz9

"*&& "& $ *& !! -;& >! *& !  ;&D O9 IOX L Y j Z9 ˆ IOY L X j Z9 O>9 IOX L Y  Z9 ˆ IOX L Z9 ‡ IOX L Y j Z9 O-9 IOX L Y j Z9 > < "* &H!+ ,, OX  Z Y 9   #; -* 1.20 O& #; & A ˆ Oaij 9 >&  r 3 r -*- @ *   r -*9 1 aij > <  a ˆ 1 , i ˆ >& ,    & PfY ˆ jjX ˆ ig ˆ aij  &  & J& --& , -$& , * -*&D

" & O p<  p1  . . .  p r 1 9 ˆ p >&  $>>+ ;&- "*-*  >& "* r 1 pi aij  j ˆ ;& #&-*  >!& --%  p O9 *" * X <  X 1  X 2  . . . & &-+ >!&  ;>& & "* - & *&+ &$&&P O *& &H!&-&

Problems

.:

X <  X 1  X 2  . . .  -&  J& #; -*L && && F.G ' 1 *$& A'  A' 9 O>9 *& O$& + >9 entropy , *& #; -* X <  X 1  . . .  &J&  H ˆ 

n!1

1 HOX <  X 1  . . . X n 1 9: n

*" * Hˆ


i j

pi aij %

1 : aij

O-9  $!& *& >& $>>+ >!  *& &$+ , *& #; -* "*&  $>>+ -& & %;& >+ 

 q  q1  . . .  q r 1 <   < < < 1  q1  q2  . . .  q<  á 1 á  < < < 1      .. : 1 1   â 1 â < <  .  2 2 < < 1 < q r 1  q<  . . .  q r 2 O  *&  &@ $& &-* "   &, -+-- *, , *& $&-&% &9 1.21  IOX L Y 9 -+ -;&@ \  pOx9P   -+ -;&@ [  pO yjx9P 1.22 & X ˆ OX 1  . . .  X n 9  Y ˆ OY1  . . .  Ym 9 >& -&&  ;&-  & f  g >& &;!& ,!- , n  m & ;>& &$&-;&+ *" * IF f OX9L gOY9G < IOXL Y9 1.23 !$$+ *& % & 
@ $& 111  112 n 1.24 *" * &H!+ *  *&& 1( ,, pOxjy9 ˆ iˆ1 pOxi j yi 9 ,  Ox y9 1.25 *" * &H!+ *  *&& 13 ,, Y1  Y2  . . .  Yn & &$&&  *&-& * &H!+ *  *& -+  *&& 13 ,, X 1  . . .  X n & &$&& 1.26 ;&  $;&D , X  Y ˆ OY1  Y2  . . .  Yn 9 & -&&  ;&- n IOXL Y9 < iˆ1 IOXL Yi 9 1.27 & X  Z >& &$&&  ;>& "* -!! &+ ,!-   & Y ˆ X ‡ Z , hOY 9  hO Z9 &@ *" * IOX L Y 9 ˆ hOY 9 hO Z9 *" * *&  & , ! * , X  Z &  ;&- 1.28 O!9 & X  Z >& &$&&  ;>& X -&&  Z "* -!! &+  & Y ˆ X ‡ Z *" * Y *  -!! &+  * , hOY 9  hO Z9 &@ *& IOX L Y 9 ˆ hOY 9 hO Z9 *" * *&  & &! * , X  Z &  ;&- 1.29 , X ˆ OX 1  . . .  X n 9 & &$&& !  ;>& "* & ì i  ;-& ó 2i  X9 ˆ OX 19  . . .  X 9n 9 &  "* & ì9i  ;-& ó 9i 2  *" * X ‡ X9 ˆ OX 1 ‡ X 19  . . .  X n ‡ X 9n 9 &  "* & ì i ‡ ì9i  ;-& ó 2i ‡ ó 9i 2  1.30 , X 1  X 2  . . .  X n & &$&&  ;>& "* ,,&& &$& hOX i 9 ˆ hi  *" *

.(

Entropy and mutual information hOX 1 ‡


‡ X n 9 >

1 2 %

n


!

e

2 hi



iˆ1

"* &H!+ ,, *& X i & ! "* ;-& ó 2i ˆ e 2 hi =2ðe 1.31 , X *  -!! &+ *" * !$ HOFX G9 ˆ ‡1 "*&& *& !$& !  #& ;&  -&& H!4 , X  1.32 !-   ;>& "*& &+ pOx9  -!! ,  & x !-* * hOX 9 ˆ 1 1.33 & X >&   n & ;&- "* -!! n & &+ pOx9  ,,&& &$+ hOX9 & f >&  -!!+ ,,&& >& && $$% ,
!-& n$-& En  &, *" * hF f OX9G ˆ hOX 9 ‡ pOx9 %jJ j dx "*&& J ˆ J Ox1  . . .  xn 9  *& 0-> , *& ,  f  1.34 & X ˆ OX 1  . . .  X n 9 >&   ;>& "* &+  !$$& EFOX i ì i 9OX j ì j 9G ˆ r? ,  i j ;& * hOX9
& "*& -;-& @  O r? 9 O&& && F.G ' 2 &- )9 1.35 & f Ox9 >&  -!! &;!& ,!- &J&   &; I *& >?&- , * $>&   J ! *" %& *& ,,&& &$+ hOX 9 - = I  >& , X *  &+ ,!- * J& pOx9 ˆ < , x 2 I pOx9 f Ox9 dx ˆ A "*&& ,O f 9 , A , !$O f 9 O &@ $& , I ˆ O 1 19  f Ox9 ˆ Ox ì92 "& " ;& >+  &" !&  *& &  & -& , *&& 1119 &J& GOs9 ˆ I e sf Ox9 dx *" * *&& &@ s< "* G9Os< 9=GOs< 9 ˆ A  &J& qOx9 ˆ e s< f Ox9 =GOs< 9 , x 2 I qOx9 ˆ < , x 2 = I *" * hOX 9 < % GOs< 9 ‡ s< A "* &H!+ ,, X I &+ ˆ qOx9   &;&+"*&& F HintD && > 1(G $$+ * %&& &-*H!&  *&& -&D O9 f Ox9 ˆ % x I ˆ O1 19 O>9 f Ox9 ˆ x I ˆ O& *" * *&& &@ -&& H!4 FX G  FY G "*-* &  &$&&

*& &@ *&& $>&  & , && "*  & #"&%& , %&& #; -*L && , &@ $& && F.G ' 2 &- '   1.37 & OX  Y  Z9 >&  #; -* !-* * , + H!4 FX G  F ZG *&& &@  H!4 FY G !-* * OFX G FY G F ZG9   -&& #; -* ;& * IOX L Y  Z9 ˆ IOX L Y 9 1.38 & #; -* , *& , OX  f OX 9 Z9  OX  Y  f OY 99 *" * !-* -* *;& *& $$&+ &->&  > 16:  *&-& * *&& 1. * , *&  F HintD  *& OX  f OX 9 Z9 -& , fSi g  *& X $ & f OSi 9 >& *& Y $G

Notes

.3

1.39 & X 1  X 2  X 6 >& &$&& &-+ >!& !  ; >& "* & <  ;-& 1 *" * *& &H!&-& OX  Y  Z9 "*&& X ˆ X 1  Y ˆ X 1 ‡ X 2   Z ˆ X 1 ‡ X 2 ‡ X 6   #; -* O&&
@ $& 139 ! *"  * , FX G  F ZG & H!4 , X  Z "*-* >* ! & & * " ;!& OFX G FY G F ZG9  &;&  #; -* , + H!4 FY G , Y 

Notes 1 O$ 139 !$$& +! -! # H!& Qn D NN/ ;&  >& nPII *# >! *" +! !-&+ O@&+M9 "! -&&  n &-&& 2 O$ 2! X 9 6 O$ 219  *@ *& - ?% NN-&&II & J&  countable

*&& "! >&  $>&  &% !  I *;& -!>& $!  !$! $*>& >!  &   -K- "* --&$& , *&&- !%& "& * !!+    . O$ 2:9 *&& &J #& && + "*& pOx9 pO y9 6ˆ ?&-  %;&  *& J *&& -*$& , #&I ># F2.G ) O$ 6:9 $ , *& & & R   -&- , !>& OSi 9 "* [ Si ˆ R  Si \ Sj ˆ ö , i 6ˆ j : O$ 6(9 * & &   -+ !&  #& F2.G   *" * IOX L Y 9  &*& J&  &H!  *& &@$&-& ;!&  *& ?  $& $-& , X  Y , *& %* , *& #+ &;;& dì xy =dì x 3 ì y  "*&& ì xy  *& ? $>>+ &!& , OX  Y 9  ì x 3 ì y  *& $!- , *& % X  Y &!& *! + >-+   dì xy IOX L Y 9 ˆ % dì xy : dì x 3 ì y  ( O$ .&% $$& *&&  * , f Ox9  -!! f Ox9 gOx9 dx ˆ f Ox 9 < S gOx9 dx ,  & x< 2 S O&& $ F2G *&& 1.1) , S &@ $&9 3 O$ ..9 &*$  >&, %!&  #&I $,   & *& && &!  * IOX  Y L Z9 ˆ IOY L Z9 ,, OX  Y  Z9   #; -* *! , *& conditional mutual information IOX L ZjY 9  &J& >+ IOX L ZjY 9 ˆ IOX  Y L Z9 IOY L Z9 *& &H!& &!  IOX L ZjY 9 ˆ < ,, OX  Y  Z9   #; -* *$& 6  #&  "*+ &;&  *& $$&& , - !! ,  & J %;&  >- &!&*&&- &J , IOX L ZjY 9  *& & %;& >;& *& $;& ;! && &+ $$&& #& IOX L ZjY 9 > < O&H!;&  *&& 1.9  IOX L ZjY 9 ˆ IO ZL X jY 9 O&& > 1139  J+ $;& * IOX L ZjY 9 ˆ < ,, OX  Y  Z9  #; +  &- 6) *& $;& * IOX L ZjY 9 ˆ IOX  Y L Z9 IOY L Z9  >!& *& &!  5 %;

2 -&& & +& -*&  *& -$-+B- ,!-

2.1 The capacity±cost function

discrete memoryless channel O L && $ 2< ,,9  -*-&4& >+ " J& &D AX  *& input alphabet  AY  *& output alphabet   & , transition probabilities pO yjx9 &J& , &-* x 2 AX  y 2 AY  "*-*  ,+ pO yjx9 > >& & -;&&+ $+&   r 3 s -*- @1 Q ˆ Oqxy 9 "*& " & &@& >+ AX  -!  >+ AY  !*& & -& "* &-* $! x *&&   &%;& ! >& bOx9 *& NN-II2 , x !+ AX " >& #&  f>+ * y1  . . .  yn  *& !$! %;& * x1  . . .  xn n  *& $!  *& $!- iˆ1 pOxi jxi 9 *& - , &% x1  . . .  xn  &J&  bOx9 ˆ

n


bOxi 9:

O2:19

iˆ1

, *& n $! & &->& $>>-+ >+ *&  ;>& X ˆ OX 1  X 2  . . .  X n 9 "* ? >! ,!- pOx9 ˆ pOx1  . . .  xn 9 *& average cost  &J& >+ EFbOX9G ˆ

n


EFbOX i 9G

iˆ1

ˆ




pOx9bOx9:

O2:29

x

 &-* n ˆ 1 2 . . .  "& &J& *& n* capacity±cost function Cn O â9 , *& -*& >+ Cn O â9 ˆ

@fIOXL Y9 D EFbOX9G < nâg

O2:69

"*&& *& @ !  O269  &@&& ;&  $ OX Y9 ˆ OOX 1  . . .  X n 9 OY1  . . .  Y n 99 , n &  ;&- , "*-* O9 *& - $>>& PfYjXg & -& "* *& %;&

72

Discrete memoryless channels and their capacity±cost functions

-*&  $>>& & PfY1 ˆ y1  . . .  Y n ˆ yn jX 1 ˆ x1  . . .  n X n ˆ xn g ˆ iˆ1 pO yi jxi 9  O9 *& $! ;&- X J& EFbOX9G < nâ /& * -  $! ;&- X  test sourceL ,  J& EFbOX9G < nâ "& * -  âadmissible &-& *& @ 4  O269  #& ;&  n dimensional âadmissible test sources &;& & # >! *& ,!- Cn O â9 &  &  & * ,  %;& @ ,  $>>& O pO yjx99 *& ,!- IOXL Y9   -!! ,!- , *& $! >! pOx9 *& & , !-* >!   ,+% pOx9bOx9 ˆ EFbOX9G < nâ   - $- !>& , r n   &
!-& $-&   *& ,!- IOXL Y9 -!+ -*&;&  @ ! ;!&6 *  *& & NN @II *& * NN!$II $$&  O269 &- -& * , "& &J& â  >+ â



ˆ

 bOx9

O2:.9

x2 AX

*& EFbOX9G > n:â     Cn O â9  &J& + , â > â  + >&;& * , â1 . â2  *& & , & !-& ,+% EFbOX9G < nâ2   !>& , *& & , & !-& ,+% EFbOX9G < nâ1    Cn O â1 9 > Cn O â2 9 & Cn O â9   -&% ,!- , â > â  

*& capacity±cost ,!-. , *& -*&  " &J& >+ 1 CO â9 ˆ !$ Cn O â9: n n

O2:79

*& ! >& CO â9 " ! !  &$&& *& @ !  ! , ,  * - >&  & &>+ ;& *& -*& $& ! ,  & , *& -*& ! >& !&  !-*  "+ * *& ;&%& -  < â $& ! ,  & *& $&-& & & , * &! *I channel coding theorem " >& $;&  &- 22 ! >?&-  * &-   &;&$ &-*H!& , - $!% CO â9 ,  %;&    - ,!- /& >&% >+ *"% * *& ,!- Cn O â9 &  convex Theorem 2.1 Cn O â9 is a convex \ function of â > â Proof & á1  á2 > ! p1 Ox9 p2 Ox9 "*-* -*&;& Cn O â1 9 Cn O â2 9 &$&-;&+ *  , Y1  Y2 && *& !$! -&$%  X1  X2  *&

2.1 The capacity±cost function  EFbOX i 9G < nâ i  i ˆ 1 2: IOX i L Yi 9 ˆ Cn O â i 9

76 O2:)9 O2::9

&J&  ,!*& & !-& X "* >! pOx9 ˆ á1 p1 Ox9 ‡ á2 p2 Ox9   & Y >& *& -&$% !$! *& EFbOX9G ˆ x pOx9bOx9 ˆ   á1 x p1 Ox9bOx9 ‡ á2 x p2 Ox9bOx9 ˆ á1 EFbOX1 9G ‡ á2 EFbOX2 9G < nOá1 â1 ‡ á2 â2 9 O&& O2)99   X  Oá1 â1 ‡ á2 â2 9 >& *! IOXL Y9 < Cn Oá1 â1 ‡ á2 â2 9 ! -& IOXL Y9   -;&@ \ ,!- , *& $! $>>+ >! pOx9 O *&& 1)9 IOXL Y9 > á1 IOX1 L Y1 9 ‡ á2 IOX2 L Y2 9 ˆ á1 Cn O â1 9 ‡ á2 Cn O â2 9 O&&
H O2:99 h ! &@ &! *" * *& $$&+ , >& &J O279    > ,& 7 Theorem 2.2 For any   Cn O â9 ˆ nC1 O â9 for all n ˆ 1 2 . . . and all â > â   Proof & X ˆ OX 1  . . .  X n 9 >&  â >& & !-& * -*&;& Cn O â9 *  EFbOX9G < nâ

O2:(9

IOXL Y9 ˆ Cn O â9

O2:39

"*&& Y ˆ OY1  . . .  Y n 9  *& -&$% -*& !$! + *&& 13 IOXL Y9
%% O239 O21! ,!- *&  &  X   & Y1  Y2  . . .  Y n >& *&  -&$% -*& !$! *& EFbOX9G ˆ EFbOX i 9G < nâ >+ O21.9 n  IOXL Y9 ˆ iˆ1 IOX i L Y i 9 O *&&  1(  139 ˆ nC1 O â9 *! h Cn O â9 > nC1 O â9  "& Corollary For a memoryless channel CO â9 ˆ C1 O â9 Osee de®nition O2799 FN.B. This result is not true for channels with memory; see Prob. 21 â    -;&@+  $& *    -!! , â . â  L &&

$$&@  O    -!!  ⠈ â  L && > 279 /& * " %!& * CO â9  -!+ - , !,J-&+ %& â & ! &J& C @ ˆ @fCO â9 D â > â  g *  C

@

ˆ

@fIOX L Y 9g

O2:1)9

"*&& *& @ !  #& ;&  O& &9 & !-& X  "*  >!  EFbOX 9G C @  -& *& capacity , *& -*& , "& &J& â

@

ˆ

fEFbOX 9G D IOX L Y 9 ˆ C

@ g

O2:1:9

*& -&+ CO â9 ˆ C @ ,  â > â @   CO â9 , C @ , â , â @  ," , *& ,- * CO â9  -&%  -;&@ \ , â > â  

2.1 The capacity±cost function

77

 - , â > â @ * CO â9  -!+ -+ -&% , â â < â @ O&& > 2)9 &-&  * &; CO â9 -! >& &J& >+ CO â9 ˆ

@fIOX L Y 9 D EFbOX 9G ˆ âg

â



%  *&& ,- "& && *  +$- CO â9 -!;& %* #  *"  % 21 Example 2.1 O-!&9

 Qˆ

q p

 p  q

bO&  & ;& *& -*& & %&&+ ,  >+ â > â  CO â9 !%*  &$&& *& @ ! &  "*-* ,  - >&  & ;& *& -*& , *& ;&%& $! - ! >& < â   ! >?&-  * &-  #& *&& ;%!& &% $&-& & *& ,"% *!%* &@$& & /& &  ,  !-&   &H!&-& U ˆ OU1  U2  . . .  U k 9 , &$&& &-+ >!&  ;>& "* -  >! ,!- PfU ˆ ?&-     *&& k NN>II , ,  ;& *& -*& !% *& -*& n  & "*  ;&%& - < â & X ˆ OX 1  . . .  X n 9 >& *& -&$% -*& $! Y ˆ OY1  Y2  . . .  Y n 9 ^ ˆ OU ^1 . . .  U ^ k 9 *& &-&;&I & & , U "*-* *& -*& !$!  U "& ! & &$& +  Y O% 279 ^ i 6ˆ Ui g , å ,  i /& ! & *   $&+ % +&  + Pf U ^ > "*&& å   &   ! >& *& >+ *&& 1( IOUL U9 k ^ ^ i 9 ˆ HOU i 9 HOUi j U ^ i 9 ˆ % 2 HOUi j U ^ i9  IOUi L U uˆ1 IOU L U i 9 > % 2 HOå9 >+ I &H!+ O-+  *&& 129 *! ^ > kF1 H 2 Oå9G  >+ *& $-&% *&& "& *;& IOUL U9 O
H O11799 IOUL V9 < IOXL Y9 + >+
H O269 IOXL Y9 < Cn O â9 ˆ nCO â9  >% *&& *&& &H!& "& *;& O $& % & >& 29D k CO â9 < : n 1 H 2 Oå9

O2:139

*&  k=n "*-* "& * - *& rate , *& +&  &$&& *& ! >& , bits per channel use >&%  & >+ !  %+ - ! - +&  *& >! O2139   -&% ,!- , *& > & $>>+ åL *   !$%D  + &&+ * *& & &>+ "& "  - !-& *& "& "& ! - !-& R!;&+
H O2139 + * , *& -*&  >&% !&  * *& ;&%& $! -  < â  , "& "*  &%  +& "* & r . CO â9 *& *& &!% & $>>+ å  >!& >&" >+ å > H 1 F1 CO â9=rG . H 2 1 O1 C @ =r9 . R  PfUi 6ˆ Ui g , å ,  iM * !% &! !&  !& *  -& *& channel coding theorem *& #&+  * &!  *& --&$ ,  -& "*-* "& " &->& ;&  &%& n  O-*&9 code of length n ;& AX   !>& C ˆ fx1  x2  . . .  x M g , A nX ( *& & , *& -&  &J&  r ˆ O1=n9 % MQ, % & >& 2 *& &   ! , bits per O-*& $!9  symbol *& -&  â-admissible , bOx i 9 ˆ njˆ1 bOxij 9 < nâ ,  i "*&& x i ˆ Oxi1  . . .  xin 9  *& &@$ , x i  - $&

decoding rule , *& -& C   $$% f D A Yn ! C [ f?g *& $&- + > NNPII && &-& ,!&L  %J--& " & &%& >&"

-& - >& !&  &%  - !- +&   *& && , % 27  ," & k >&  &%& !-* * k < %2 M *&  " >& $>&  %  distinct -&" x i  &-* , *& 2 k $>& !-& &H!&-&3 && $$% , *& & ,  $>& !-& &H!&-&  *& -& C  -&  encoding rule , *& !-& &H!&-&  >&  &  u ˆ Ou1  . . .  uk 9 *& &   encode u  & , *& -&" x i ; *& &-% !&    x i ;& *& -*&  *& *& & , *& -*& $$&  + ;& , x i L -  y *& &-&;& &-& y   -&" x j O NNPII9 ; *& &-% !& f L *& & & u^ , u  *& *& !H!& !-& &H!&-& u^ O, *&&  &9 * -&$  *& -&" x j  *& & $>>+ , *& +&  %;& * x i   & && >+ POi9 E   *& %;& >+ POi9 E ˆ Pf f Oy9 6ˆ x i g ˆ




f pOyjx i 9 D f Oy9 6ˆ x i g

O2:2?&- ,  "  %;& $ ;& &@ $&M9

2.2 The channel coding theorem Example 2.4 AX ˆ AY ˆ f>+ p , p , 2  !-* & , p  ;&+   O&&
H O 2119 Example 2.6 AX ˆ f! O2139 &- * "& & !  &% , %;& ;!& , â R , CO â9  å . 2d Rne  POi9 E , å ,  å *& ,"% *&& & *   $>&   * Theorem 2.4 Let a   with capacity±cost function CO â9 be given. Then, for any â< > â  and real numbers â . â< , R , CO â< 9, å . 9 M > 2d Rne  O-9 POi9 E , å for all i ˆ 1 2 . . .  M Corollary (the channel coding theorem for DMC's 10 ). For any R , C @ and å . 2d Rne  O>9 POi9 E , å for all i ˆ 1 2 . . .  M Proof of corollary & â< ˆ â

@

 *&&

2.

h

Proof of Theorem 2.4 *!%*! *& $, *# , n   %& &%&  " >& $&-J& & $&-&+ & & *& & Ù ,  $ Ox y9 -% , & -*& $! &H!&-& x ˆ Ox1  . . .  xn 9  & -*& !$! &H!&-& y ˆ O y1  . . .  yn 9 >* , &%* n  + > Ù ˆ A nX 3 A Yn  /& #& Ù    $& $-& >+ &J% pOx y9 ˆ pOx9 pOyjx9L

O2:219

pOx9 ˆ pOx1 9 . . . pOxn 9 "*&& pOx9   $>>+ >!  AX * -*&;& CO â< 9L  pOyjx9 ˆ pO y1 jx1 9 . . . pO yn jxn 9 "*&& pO yjx9 & *& -*&  $>>& & ! " -*& R9  ,+ R , R9 , CO â< 9  &J&  !>& T  Ù  ,"D T ˆ fOx y9 D IOxL y9 > nR9g

O2:229

2.2 The channel coding theorem

)6

"*&& IOxL y9 ˆ %2 F pOyjx9= pOy9G *& & T - >& *!%* ,  *& & , $ "*-* & -& %&*& O&& > 21:9   -& &&  "& &J&  !>& B  A nX D B ˆ fx D bOx9 < âng:

O2:269

&& B  *& & , â >& -&" + "& &J& *& & T   T >+ T  ˆ fOx y9 D Ox y9 2 T  x 2 Bg:

O2:2.9

" & C ˆ fx1  x2  . . .  x M g >& + -& %  > , &%* n /& &J&  &-% !&  ," , y  &-&;& "& &@ & *& & SOy9 ˆ fx D Ox y9 2 T  g  B O"*-* -! >& *!%* ,   NN$*&&II ! y9 , SOy9 - exactly one -&" x i "& & f Oy9 ˆ x i  *&"& &*& >&-!& SOy9 -  -&"  & * & -&" "& & f Oy9 ˆ ? *  "& &>&&+ #&  &11 * &-% !&  &$-&  % 2)

Figure 2.6 *& &-% !& !&  *& $, , *&&

2.

).

Discrete memoryless channels and their capacity±cost functions

, "& !& *& -& C "* *& &-% !& ?! &->& , x i   &  , y  &-&;&  & - --! ,, &*& x i 2 = SOy9  x j 2 SOy9 ,  & j 6ˆ i &-& O&&
H O22& &"& 


POi9 ÄOx i  y9 pOyjx i 9 ‡ ÄOx j  y9 pOyjx i 9 E < y

y

j6ˆ i

ˆ Qi Ox1  . . .  x M 9:

O2:2:9

! %    &*" J  -& fx1  . . .  x M g , "*-* Qi  ;&+   ,  i  !&!+ ,!&+ *"&;& Qi   &@& &+ - $-& ,!-  - >& &;!& &@$-+ O &;& -&+ & &9 &@-&$ , *&  $& $>& -&Q-&+  , >% - $&@ -& "* %& n  *!%& MM  "*  *& $  !-% *& >! POi9 E < Qi  *& J $-&P *& $  * *!%*    $>&  & & Qi ,  $-! -&   $>&  & & *& ;&%& ;!& , Qi  fx1  . . .  x M g %& ;& *& & ,  $>& -&M

*%+12 * ;&%& " ! !  $$-* < , M ˆ 2 Rn  n ! 1M * & #>& $, &-*H!&  -& random coding >&-!& "& -*& *& -& fx1  . . .  x M g NN  II --%   -& $>>+ >! & ! " && *"  "#  &

*& J &$   &->& *& $$$& $>>+ >!  *& & ,  $>& -&   pOx1  . . .  x M 9 ˆ

M 

pOx i 9

iˆ1

"*&& , x i ˆ Oxi1  xi2  . . .  xin 9 pOx i 9 ˆ

n

kˆ1

pOxik 9 * $>>+ 

2.2 The channel coding theorem

)7

>! -&$  *& &@$& & , -*% *& -& NN +II >+ $-#% *& ;! -&" -& &$&&+ --%  *& $>>+ >! pOx9 * -*&;& CO â< 9 /& " ;&" Qi Ox1  . . .  x M 9    ;>&  *&  $& $-& ,  $>& -&L  &@$&-& ;!&  O&&
H O22:99

EOQi 9 ˆ E


y

ˆ E1 ‡





ÄOx i  y9 pOyjx i 9 ‡ E ÄOx j  y9 pOyjx i 9 y

j6ˆ i


j6ˆ i

E2O j9 :

O2:2(9

 "& >! E1 D

E1 ˆ ˆ


x1 :::x M


x i y

ˆ




pOx1 9 . . . pOx M 9


y

ÄOx i  y9 pOyjx i 9

pOx i 9 pOyjx i 9ÄOx i  y9 pOx y9ÄOx y9

O&& O2:2199

xy

ˆ PfOx y9 2 = T g

O&& O2:2)99

ˆ PfOx y9 2 = T  x 2 = Bg

O&& O2:2.99

< PfOx y9 2 = T g ‡ Pfx 2 = Bg:

O2:239

&-&

E1 < PfIOxL y9 , nR9g ‡ PfbOx9 . âng

!

O&& O2:229 O2:269 O2:2399

))

Discrete memoryless channels and their capacity±cost functions IOxL y9 ˆ % ˆ % ˆ

n


pOyjx9 pOy9 n  pO yk jxk 9 pO yk 9 kˆ1

pO yk jxk 9 pO yk 9

%

kˆ1

ˆ

n


IOxk L yk 9:

kˆ1

&-& IOxL y9  *& ! , n &$&& &-+ >!&  ;>& IOxk L yk 9 + &J EFIOxk L yk 9G ˆ IOX L Y 9 ˆ CO â< 9  * &-* IOxk L yk 9 * & CO â< 9 -& R9 , CO â< 9  ," , *& "&# " , %& ! >& O&& $$&@ 9 * 

n!1

PfIOxL y9 , nR9g ˆ & "* & < â<  -& â . â<   ," * 

n!1

PfbOx9 . nâg ˆ % O2239 O26& -!&D

R9 n

O&& O22299 &-& *& >!

2.2 The channel coding theorem
E2O j9 < pOx9 pOy9

):

Oxy92T

% O22(9 O2239  O2669 "& %& EOQi 9 < PfIOxL y9 , nR9g ‡ PfbOX9 . nâg ‡ M :2

R9 n

:

O2:6.9

, M ˆ 2:2d Rne  *&  &  O26.9  < .:2 nO R9 R9  -& R9 . R  ," * , !,J-&+ %& n * & - >& &     && /& *;& &+ >&;& O&& O26& &J&  *&  $& $-& ,  $>& -& *& , O22:9  O2679 "& *;& , M ˆ 2:2d Rne  n %& &!%* EOPE 9 , å=2:

*! *& ;&%& ;!& , PE  , å=2   *&& ! >&  $-! -& Ox1  . . .  x M 9 "* PE Ox1  . . .  x M 9 , å=2 * -& +  ,+ *& --! , *&& 2. >&-!&  + - -&" x i , "*-* bOx i 9 . nâ E POi9 & * *, , *& -&" x i * E . å ! , POi9 > å , O26)9 "& "! *;& PE > å=2  --  , "& E Oi9 &&& *& -&" "* P E > å , *& -& "& >  -& "* 16 > 2d Rne -&" , "*-* POi9

* -& *&&,& J& E , å ,  i --! O>9  O-9 , *&& 2. + -& * , bOx i 9 . nâ *& &-% $*&& SOy9 ˆ fx D Ox y9 2 T  bOx9 < nâg - - x i  *  POi9 E ˆ 1 &-& * &" -& - - + -&" "*-* &  â >&   O9  J&  h

)(

Discrete memoryless channels and their capacity±cost functions Problems 2.1  $!& *& - $&& -$-+B- ,!- , *& ,"% -*&D

 1 < < O9 Q ˆ  < q p  bO - >& &-&;& -&-+  erased *  &-&;&  NNPII9 2.2 & ! -  -*& weakly symmetric , *& -!  ,   @ Q - >& $&  !>& Ci !-* * , &-* i  *& @ Qi , & >+ *& -!   Ci &-* "   $& ! , &;&+ *& "  *&  &  !& , -!  O &@ $& Q9  $%& 7:  "&#+ + &- >!  + &-9 *" * *& -$-+ ,  "&#+ + &- -*&  -*&;& "* &H!$>>& $! 2.3 & ! &J&  strongly symmetric -*& , & r  & "*&  @ Q * *& , !& >&" , r ˆ .D

 q p p p  p q p p    p p q p p p p q "*&& p < 1=r  Or 19 p ‡ q ˆ 1 *& >?&- , * &@&-&  , +!  - $!& CO â9 , * -*& , >+ $! - bOx9 >+ !$$+% *& &  *& ,"% %! & O ! & â  ˆ  2.9 , X   & !-& -*&;% CO â9 & f Ox9 ˆ PfX ˆ xg gOx9 ˆ PfY ˆ xg , x 2 f 1(  > *& $ &- !D

Problems CO â9 ˆ % AOë9 âˆ

2.4

2.5

2.6 2.7 2.8

ëA9Oë9=AOë9 1 q

p

HOq9

)3 O1

q9%Or

19

OA9Oë9=AOë9 ‡ BP 9

 ë %& ;& *& &; ë 2 Fë<  ?&- , * &@&-&   *" * *&&   &&  , %&&+  ! % â  ˆ ?&- , * &@&-&   $;& * CO â9  -!!  ⠈ â     * !$$+ *& &  *& ,"% %! & & Op1  p2  . . .9 >&  &H!&-& , $>>+ ;&- &$&&% *& $! >! , & !-& -*&;% CO â1 9 CO â2 9 . . .  "*&&  n!1 â n ˆ â   *& *&&   !>&H!&-& Op n1  p n2  . . . 9 "*-* -;&%&   $>>+ ;&- p , X   & !-& "* $! >! p *& EFbOX 9G ˆ â   IOX L Y 9 ˆ  â!⇠CO â9 &-& CO â  9 >  â!⇠CO â9 -& CO â9   -&% ,!- , â > â   * $;& *& &H!& -!+ & f Ox9 >&  -&% -;&@ \ ,!- , x > x1  ! & f Ox9  - , x > x2 "*&& x1 , x2  *" * f Ox9  -+ -&% , x1 < x < x2  *-&4&  ,!+  +! - *&  I "* C @ ˆ & AO19 X 3 A X  !$! $*>& A Y O29 O19 O29 : 3 A Y   $>>& pOO y1  y2 9j Ox1  x2 99 ˆ p O y1 jx1 9 p O y2 jx2 9

:
&% !&  $&9 *" * *& -$-+B- ,!- , *& $!- -*&  %;& >+ CO â9 ˆ

@fC1 Ot9 ‡ C2 O â t

t9g:

*! *" * *& %$* , CO â9  >& >+ % >* -& , *& -!;& C1 O â9  C2 O â9  $  *& " -!;& *;% *&  & $& 2.10 O *& -$-+ ,   $& -*& "* & +9 & Z 1  Z 2  . . . >& &-+ >!&  ;>&  #; -* #% ;!&  f 12& !&  &J&  ;& -*& "* AX ˆ AY ˆ f+ *& !& Yi  X i ‡ Z i O  r9 "*&& X i  Yi & *& $!  !$! , *& -*&   & i *& ndimensional capacity  &J&  C O n9@ ˆ @fIOXL Y9g  X %& ;&  n & & !-&  *& capacity  C @ ˆ !$O1=n9C On9@  *" * C O n9@ ˆ n % r HO Z 1  . . .  Z n 9 ˆ n % r HOp9 On 19 H "*&& p   $>>+ ;&- &->% *& -  >! , *& Z i I  H  *& &$+ , *& -* -!& * C @ ˆ % r H $$+ * &!  *& -& "*&& r ˆ 2  *& -*I  $>>& & %;& >+ *& -*- @   q p Qˆ : p q 2.11  *& -*&  - ,!- 
@ $& 27 *" &@$-+ *"  -*&;& "*  $ & >+ *& -% *&&  *  %;&  $&-& &-$ ,  -& C * * $$&& O9 O>9  O-9 , *&& 2. 2.12  $&& *& & ,
@ $& 2)  *" *" *& &-% ,!- 1 -! >&  $;&  -*&;& POi9 E ˆ 6 , i ˆ 1 2 2.13  *& $, , *&& 2. "& %;&  &@$- &-% !& ,  %;& -& C ˆ fx1  . . .  x M g O&& $$ )69 * !& "*& &H!& , ! $!$&   $    &+ !&  $--&  * &@&-& "& * !-&  !+ " *& &-% !& "*-* & >&& * *& &  *&& 2. >! & & ,J-!  +4& *! !$$& *& $>>+ * x i " >&  &  pi  i ˆ 1 2 . . .  M ;& *& &-&;& ;&- y minimum error probability decoding O
9 -*& *& -&" , "*-* *& -  $>>+ pOx i jy9  %& Maximum likelihood decoding O 9 -*& *& & , "*-* pOyjx i 9  %& O9 *" *
 ;& !$    & *  *& $>>+ , &-& & %;& * y  &-&;&    4& >+
  *&-&  * M *& ;&%& & $>>+ PE ˆ iˆ1 pi POi9 E   4& O>9 *" * , pi ˆ 1=M ,  i *&  $&,  &-+ 
 O-9    &J& *& Hamming distance d H Ox y9 >&"&& x  y  *& ! >& , - $&  "*-* *&+ %&& *" *   &-& "+ $-#  -&" x i "*-*  -&  y "* &$&-  * -& O&& > 21:   &- :69 2.14 O!9 & *& ,"% -& , &%* . ;& f& < â "& -! &H!& * *& @ ! ">& - >& < â *  &J& CO â9 ˆ !$fIOX L Y 9 D bOX 9 < â "* $>>+ 1g  $!& CO â9 , *& -*& ,
@ $& 21 22  26 O$$ 77B7)9 & *& -% *&& &  ; , CO â9P    *" * *& & T &J& 
H O2229  , *& , T ˆ fOxL y9 D d H OxL y9 < rg "*&& d H Ox y9  *& Hamming distance >&"&& x  y O& *& ! >& , - $&  "*-* x  y ,,&9  J r  &  , n R9  å *& -;& $>>+ , *& -*& O ! & pOx ˆ &  J& &  & f >&  ,!- $$% Ù  *& & ! >& ;& * *&& &@  && & ù 2 Ù !-* * f Où9 , y ,,   $>&  &J&  $>>+ >!  Ù !-* * EF f Où9G , y & *&   "*& @ ,  $>>& 

1 1  < < < 2 2    < 12 12 < <      Q ˆ  < < 12 12 < :   < < < 1 1   2 2 1 < < < 12 2

O *&&   - - & bOx9 ˆ < ,  x9 O9  $!& C @  O>9   -& , &%* 1 "* & % 2  POi9 E ˆ < ,  i O-9   -& , &%* 2 "* & 12 % 7  POi9 E ˆ < ,  i O9 , fx1  . . .  x M g  + -& , &%* n , * -*& "* POi9 E ˆ < ,  i *" * *& -&I &  , % 721. 2.20 *& >+ &!& -*& * @ ,  $>>&  < ? 1 Qˆ < q p <  1 < p q "*&& < < p < 12 p ‡ q ˆ 1 O ! &  $! - & bOx9 ˆ < ,  x9  -$-+  C @ ˆ 1 p O&& > 21>9 ! & * * -*&  &H!$$& "* noiseless, delayless feedback *  *& &-&;&  >&  - !-& >-#  *& && *& + > *& * &-&;& *& && $ *& ,"% NN-%II &%+ , - !-% *& !$! ,  >+ + &- !-& ;& * -*&D *& &&+ &$& &-* + > !   J+ &-&;& -&-+

:2

Discrete memoryless channels and their capacity±cost functions O9 -!& *& ;&%& ! >& , -*& + > &H!& $& !-& + > , * &%+  !& O>9 % *& &! , $ O9 &% , + R , C @ å . 2d Rne  POi9 E , å , &-* i O&& -+  *&& 2.917  *& &@ @ $>&  "& " #&-*  $, ,  -% *&& ,    "*! - - *    & "+ %&   *& "+ "&#& * *& -*& -% *&& O-+  *&& 2.9  "& ! #&  & &J  &-* $ x1 x2 2 AX  "& &J&
 J Ox1  x2 9 ˆ pO yjx1 9 pO yjx2 9 y2 AY

 J< ˆ

fEOJ OX 1  X 2 99g

"*&& *&  4  #& ;&  &$&& &-+ >!&  ;>& ! % ;!&  AX  + *&  $ H!+ R<  &J& >+ R< ˆ

%2 J < :

*& *&&  >& $;&  *D R< theorem for a    + R , R< *&& &@  -& fx1  x2  . . .  x M g "*  & M ˆ d2 Rn e -&"  M Oi9 , &%* n   $$$& &-% !& !-* * , PE ˆ O iˆ1 P E 9=M && *& ;&%& &-% & $>>+ *& PE , 2 nO R< R9  O * *&&  %& * *& -*& -% *&&  *  %;&  &@$- & & , *"   PE - >& &   ,!- , n   "&#&  * R< , C @   * , & R< , R , C @  &   $+ * PE ! <  $>&  9 2.21 &  -& , &%* n -% + " -&" x1 ˆ Ox11  x12  . . .  x1 n 9  x2 ˆ Ox21  x22  . . .  x2 n 9 ! & * %;&  &-&;& n!$& y *& &-& !$! x1 , pOyjx1 9 . pOyjx2 9  x2 , pOyjx2 9 . pOyjx1 9 & Y1 ˆ fy D pOyjx1 9 . pOyjx2 9g Y2 ˆ fyD pOyjx2 9 . pOyjx1 9g , POi9 E && *& $>>+ , &-& & %;& * x i  & *" * n  POi9 J Ox1 k  x2 k 9 i ˆ 1 2 E < kˆ1

>+ ;&,+% *& ,"% &$D
POi9 pOyjx i 9 E < y2Yi


+ POi9 E *& $>>+ , &-& & %;& * x i   &  *" * POi9 E
&  > 222 ;&  -& "*&& *& ;! -&" & -*& &$&&+ --%  *& $>>+ >! * -*&;& J <  > *& & & : EFPOi9 E G, M 2

R< n

:

2.24 " - $&& *& $, , *& R< -% *&&  2.25 *" * ,   " # R< ˆ 1 %2 1 ‡ 2 pO1 p9  "*&&

p  *& " > & $>>+ !*& & *" * , R< , C @  "*&& C @ ˆ 1 H 2 O p9  *& -$-+ , *&  2.26  $!& R< , *& >+ &!& -*& O&& > 21>9 1 2 C @

Notes 1 O$ 7& "* *& $$&+ * *& ! , *& &&  &-* "  1 O-, > 12>+ @  *;& - " &$+  -!  !  : O$ 7(9 ! % *&  $& %*  & >& 2 ( O$ ) A n && *& & fOa1  a2  . . .  a n 9D a i 2 Ag , && n!$& , A 3 O$ )& !&  - !-& *& !$! , + !-& >+ & +&  *&"& $;& *& ! >& , $>& !-&

:.

Discrete memoryless channels and their capacity±cost functions

!$!  < M * >!& #& *& -% *&& O *&& 2.9 !-* & ;!>& * ! -! -& -& ,&" & !-& & --!&+ && >+  >+ + &- !-& 1< O$ )29 * -+ "*-* & * we can communicate arbitrarily reliably at rates below channel capacity  -& *& channel coding theorem -&  &&  & 2 *& - -  !!+ >&   *& & %&& &! , *&& 2.  &;&+ ! $ 11 O$ )69 * &-% !&   *& >& $>& &L && > 216  21. "&;&   &;&+ &+  +4&   !,J-&+ -&  $   " !  $;& *& -% *&&  12 O$ ).9 ! && > 21( 16 O$ ):9 & * &&% -&" , C - -*%& POi9 E , *& & %  Oi9 -&" -& P E ˆ f pOyjx i 9D y 2 = f 1 Ox i 9g 1. O$ :19 *   , ! &@ $& ,  -*& "*  4& zero-error capacity   #" *  -& &@ , * -*& "* PE ˆ <  & . 12 % 7L && *& $$& >+ * F27G $$ 112B116  ;C4 F71G 17 O$ :29 * &! >&- & & &&%  %* , *& *&&  !&  * * ,&&>-# - -&& *& -$-+ ,   M O&& * F27G $ 12-#L && F27G $$ 6:6B.6)

3 -&& & +& !-&  *& & ,!-

3.1 The rate-distortion function &  ,  !-& * $!-& &;&+ ! ,  &  + > u ,  J& & AU  -& *& source alphabet !$$& * *& &H!&-& , + > $!-& >+ *& !-& - >& && >+  &H!&-& U 1  U 2  . . . , &$&& &-+ >!&  ;>& "* -  >!  ,!- PfU ˆ ug ˆ pOu9 !-*  ,  !-&  -&  discrete memoryless source O 9  *& ! >& pOu9 & -& *& source statistics " !$$& * "& & &H!&    *& !-&I !$! ;&  -*&   -& & /& ! & *   & !-& + > u 2 AU " >& &$!-&  *& &   + > v "*-*   && & , *& J& & AV  -& *& destination alphabet OAV " !!+ >!  "+ - AU   !>&9  !$$& * , &-* $ Ou v9 *&&   &%;& ! >& dOu v9 "*-* &!& *& &  distortion -!& "*& *& !-& + > u  &$!-&  *& & + > v *& ,!- d  -&  distortion measure1 + !$$& * *& ,!- d  &@&&  $ Ou v9 ˆ Ou1  u2  . . .  uk L v1  v2  . . .  v k 9 , A Uk 3 A Vk >+ dOu v9 ˆ

k


dOui  v i 9:

O6:19

iˆ1

  !!+ >!  "+ -;&&  #& AU ˆ f & & >+ *& !-&  & V1  V2  . . .  Vk >& any -&- , k  ;>& #% ;!&  *& & $*>& AV   &J&  *&  &  $& $-&  *& Ui I /& - --!& *& !! ,  IOUL V9 >&"&& *&  ;&- U ˆ OU1  . . .  U k 9  V ˆ OV1  . . .  Vk 9   *& average distortion EOd9 ˆ EFdOU V9G &J& >+
EOd9 ˆ pOu v9dOu v9 uv

ˆ




pOu9 pOvju9dOu v9:

O6:29

uv

O  O629 *& !   &@&& ;& *& r k s k $ Ou v9 ˆ Ou1  . . .  uk  v1  . . .  v k 9 "*&& ui 2 AU  v i 2 AV   pOu v9 ˆ PfU ˆ u V ˆ vg pOvju9 ˆ PfV ˆ vjU ˆ ug9 /& " &J&6 *& ,!- Rk Oä9 "*-*   ,!- , *& !-& - O pOu99 *&  @ D  *& & ! >& ä >+ Rk Oä9 ˆ

fIOUL V9D EOd9 < käg:

O6:69

 O669 *&  4  &@&& ;&  $ OU V9 ˆ OOU1  . . .  Uk 9 OV1  . . .  Vk 99 , k &  ;&- ! % ;!&  A Uk 3 A Vk , "*-* U1  . . .  U k & &$&& "* -  >! ,!- PfU ˆ ug ˆ pOu9 "*&& O pOu99 & *& %;& !-& -  *& ;&%&  EOd9 &J&  O629  < kä -& *& !-& - O pOu99 & J@&  - $!% Rk Oä9 ,  J@& ä "& ! ;+ *& - $>>& pOvju9 * &J& V *&& $>>& - >& *!%* ,   $>>& &J%  -*& "* U  $! V  !$!  *& $&& -&@ * -*&  !!+ -&  k &

3.1 The rate-distortion function

::

test channel  *&  4 O669    #& $-& ;&  & -*& "*& ;&%&   < kä &;& $& + & # >! *& ,!- Rk Oä9 &  &  & * , J@& O pOu99 *& ,!- IOUL V9   -!! ,!- , *& r k s k  $>>& pOvju9 *& !>& , *& & ,  $>>& "*&& EOd9 < kä   - $- &% , r k s k  &
!-& $-&   *& ,!- IOUL V9 -!+ -*&;&   ! ;!&  * &%. *  *& & "& "& NN II *& * NN,II 
H O669 &- -& * *&  ! $>& ;!& , EOd9  %;& >+ k
ä   "*&&
ä  ˆ pOu9
 dOu v9 O6:.9 v

u2 AU

  + >& && , O629 -& EOd9 > Ouv9 pOu v9 v dOu v9 ˆ kä  

*! Rk Oä9  &J& + , ä > ä  + >&;& * , ä1 . ä2  *& & , k & & -*& ,+% EOd9 < ä2   !>& , *& , "*-* EOd9 < ä1    Rk Oä1 9 < Rk Oä2 9 *  Rk Oä9   &-&% ,!- , ä > ä  

*& rate-distortion ,!- , *& !-&  " &J&  ROä9 ˆ , k

1 Rk Oä9: k

O6:79

*& ! >& ROä9 ! !  &$&& *&  ! ! >& , > O! %  %*  & >& 29 &&&  &$&&  !-& + > , "& & "%  &&  ;&%&  , ä *& $&-& & & , * &! *I source coding theorem " >& &  $;&  &- 62 ! >?&-  *& & , * &-   &;&$ &-*H!& , - $!% ROä9 ,  %;&     &!& ! J &!  * *& ,!- Rk Oä9 & -;&@ Theorem 3.1 Rk Oä9 is a convex [ function of ä > ä Proof !$$& á1  á2 > ä @ 

 

! *" * , ä1

Rk Oá1 ä1 ‡ á2 ä2 9 < á1 Rk Oä1 9 ‡ á2 Rk Oä2 9:

O6:)9

  * & pi Oujv9 >& *&  $>>& ,  & -*& -*&;% Rk Oä i 9 i ˆ 1 2 *& , V1  V2 && *& & -*& !$! IOUL Vi 9 ˆ Rk Oä i 9

O6::9

:(

Discrete memoryless sources and their rate-distortion functions EOd i 9 < kä i 

i ˆ 1 2

O6:(9

"*&& d i ˆ dOU Vi 9 && *& ;&%&   *& i* & -*& " &J&  &" & -*& "*  $>>& pOvju9 ˆ á1 p1 Ovju9 ‡ á2 p2 Ovju9 , V && *& !$! , * & -*& *& , O629  O6(9 EFdOUL V9G ˆ á1 EFdOUL V1 9G ‡ á2 EFdOUL V2 9G < Oá1 ä1 ‡ á2 ä2 9 &-& * & -*&   >& , *& --! , Rk Oá1 ä1 ‡ á2 ä2 9   IOUL V9 > Rk Oá1 ä1 ‡ á2 ä2 9  *& *& * -& IOUL V9   -;&@ [ ,!- , *&  $>>& pOvju9 O *&& 1:9 IOUL V9 < á1 IOUL V1 9 ‡ á2 IOUL V2 9 ˆ á1 Rk Oä1 9 ‡ á2 Rk Oä2 9 *&  " &H!& - >&  %;& O6)9   $;& *&& 61 h

*& &@ &! *" * ,    *& - $! , ROä9  -& >+ && * "! $$& , &J O679 Theorem 3.27 For a DMS, Rk Oä9 ˆ kR1 Oä9 for all k and ä > ä

 

Proof & pOvju9 >& *&  $>>& ,  k & & -*& -*&;% Rk Oä9 *& IOUL V9 ˆ Rk Oä9

O6:39

EFdOU V9G < kä:

O6:1+ *&& IOUL V9 >

k


1( "& *;&

IOUi L Vi 9:

O6:119

iˆ1

, "& &J& äi ˆ EFdOUi  Vi 9 "& *;& IOU i L Vi 9 > R1 Oäi 9 EFdOU V9G ˆ

i ˆ 1 2 . . .  k k
iˆ1

äi < kä:

O6:129 O6:169

k  >% O6119  O6129 "& *;& IOUL V9 > iˆ1 R1 Oä i 9 ! -& R1  -;&@ [   k
ä1 ‡


‡ ä k R1 Oäi 9 > k
R1 k iˆ1 > kR1 Oä9

3.1 The rate-distortion function

:3

>+ O6169  *& ,- * R1   &-&% ,!- , ä &-& Rk Oä9 ˆ IOUL V9 > kR1 Oä9  $;& *& $$& &H!+ & pOvju9 >&  &  & & -*& * -*&;& R1 Oä9  &J& pOvju9 ˆ k & +& & -*&  &J& iˆ1 pOv i jui 9   &+  ;&,+ * *& * EOd9 < kä  IOUL V9 ˆ kR1 Oä9 O&& > 669 *! Rk Oä9 < kR1 Oä9  "&  *&& 62  $;& h Corollary ROä9 ˆ R1 Oä9 ˆ Proof * ,"  O679

fIOU L V 9D EOd9 < äg:

&&+ ,

*&&

62  &J O669  h

& ! " $!& ,  %&& &-$ , *& ,!- ROä9 /& &+ #" * ROä9   &-&% -;&@ [ ,!- , ä > ä   *& -;&@+  $&  &&+ O&& $$&@ 9 * ROä9  -!! , ä . ä  ROä9   -!!  ä ˆ ä   >! "& &;& *& $, , * ,-  > 6. !*& & "& * " *" * ROä9 ˆ < ,  !,J-&+ %& ä && , "& &J& ä @ >+
ä @ ˆ  pOu9dOu v9 O6:1.9 v

u

*& ROä9 ˆ < ,, ä > ä @   && * >&;& *  & -*& "*-* $ &;&+ $! u && -+   v , "*-*  pOu9dOu v9 ˆ ä @ " *;& IOU L V 9 ˆ <  EOd9 ˆ ä @  * *" * ROä9 ˆ < , ä > ä @ ;&&+ , ROä9 ˆ


v

pOv9
ä

@

ˆä

u

@

>+ O61.9 *! , ROä9 ˆ ä @  -& ROä9  &-&%  -;&@ [ , ä > ä    - , ä > ä @   ," * ROä9  -+ &-&% , ä  < ä < ä @ O&& > 2)9    * %& ROä9  %;& >+

(
 6( *"&;&   *" * *&&   &&  , %&&+  ! % * ä  ˆ  u *&& &@  & & & + > , "*-* dOu v9 ˆ  6:9 "&;& ,    ! % * &-* " , *& @ D *  & & HO p9 HOä9 * "& >! ! !  >& *& ;!& , ROä9 , < < ä < p  $;& * "& ! $!-&  & -*& "* EOd9 ˆ ä IOU L V 9 ˆ HO p9 HOä9 *& >& "+   *   &J&  NN>-#"II & -*& *   %;& *&  $>> & pOujv9 >+ pOujv9 ˆ ä , u 6ˆ v ˆ 1 ä , u ˆ v *& >-#" & -*&  *"  % 6. &+ !-*  & -*& " *;& EOd9 ˆ ä HOU jV 9 ˆ HOä9 "&;& "& ! #& !& *   $>&  -*& á ˆ PfV ˆ & 1 1 1 P V ˆ ‡2jU ˆ <  P V ˆ 2jU ˆ <   d & & &-D AU ˆ AV ˆ f+ I &H! + O-+  *&& 129 HOU jV 9 < ä %Or 19 ‡ HOä9 &-& ROä9 > % r ä %Or 19 HOä9  *" *& $$& &H!+ ! & < < ä < 1 1=r  &J&  & -*& >+  1 ä , v ˆ u ä pOvju9 ˆ , v 6ˆ u: r 1

*&   $& --! %;& EOd9 ˆ ä IOU L V 9 ˆ HOV 9 % r HF1 ä ä=Or 19 . . .  ä=Or 19G ˆ % r ä %Or 19 HOä9

HOV jU 9 ˆ h

(.

Discrete memoryless sources and their rate-distortion functions 3.2 The source coding theorem

*& & ,!- "*-* " !& ,  $!&+ *& - $ , ;&"  &- 61 *  >&!,! - !-*&&%J--& "*-*  *D ROä9 is the number of bits needed to represent a source symbol, if a distortion ä is allowable *!  !-& + > - >& NN- $&&II  ROä9 >L -& ROä9 &-&&  ä -&& & - $&  $>&  ä -&&  * & NN& *&+II   & & -& NN- $& *&+II

 && "*+ ROä9 %* *;& * %J--& -& *& ,"% ! & OU1  U2  . . .  U k 9 ˆ U &$&& *& J k + > & & >+  -&   & ! !$$& * *&& k + > & NN- $&&II  n > OX 1  X 2  . . .  X n 9 ˆ X  *    &*" $>&  & -;& , X k & + > OV1  . . .  Vk 9 ˆ V !-* * k EFdOU  V i i 9G < kä & *&& --! -&   &>&  & iˆ1 * *& n > X 1  . . .  X n &$&& *& k !-& + > U 1  . . .  Uk "*  ;&%&  < ä *& &*$ >&"&& U X V - >& #&-*&  *"  % 6) " "& #" ,
H O669  O679 * IOUL V9 > Rk Oä9 > kROä9L , *&& 17 * IOUL V9 < IOXL V9L , *& &J , IOXL V9 O&&
H O1.99 * IOXL V9 < HOX9L  , *&& 11 * HOX9 < n >  >% *&& &! "& *;& kROä9 < IOUL V9 < IOXL V9 < HOX9 < n *  O $& % & >& 29 n > ROä9: O6:1)9 k

*&  n=k  O61)9 &$&& *& ! >& , > $& !-& + >  *& >;&  - $& -*& & *! "& &&  &&+ *  & ROä9 > & &&&  &$&&  !-& + > , *& ;&%&  ! >& < ä *& source coding theorem  >& $;& >&" & *   -& &&  & * ROä9 > & &&&  "& ! &J&  !-& -& !-& -& , &%* k   !>& , A Vk  *   & C ˆ fv1  v2  . . .  v M g , destination &H!&-& , &%* k  rate  &J&  >& R ˆ k 1 %2 M  &-* !-& &H!&-& u ˆ Ou1  . . .  uk 9 , &%* k & f Ou9 >&  -&" v i "*-*  NN-&II  u  *& && * dOu f Ou99 < dOu v j 9

j ˆ 1 2 . . .  M:

Figure 3.6 %&& - $& -*& &

O6:1:9

3.2 The source coding theorem

*& average distortion , *& -& C  &J&  >& 1
dOC9 ˆ pOu9dOu f Ou99 k k

(7

O6:1(9

u2 A U

"*&&  O61(9 pOu9 ˆ pOu1 9 pOu2 9 . . . pOuk 9  *& $>>+ * *& J k + > & & >+ *& !-& " >& u1  . . .  uk  , M < 2 n  !-*  !-& -& - >& !&  &;&   - $& -*& & , *&  &$-&  % 6)  ,"  &-* , *& M !-& -&" v i %  - >+ n!$& xOv i 9 ˆ Ox1 Ov i 9 . . .  xn Ov i 99 -& M < 2 n  * " >& $>& *& !-& &H!&-& u ˆ Ou1  . . .  uk 9  *& &$&&& >+ *& n > x ˆ xF f Ou9G  *& & &H!&-& v  #&  >& *& -&" f Ou9 O-& *& $$% v ! xOv9  &  & f Ou9 - >& !H!&+ &-;&& , xF f Ou9G9   -& * *& ;&%&  , *& -*& &  &@-+ dOC9  &J&  O61(9  *& - $&  n=k  ?! d%2 M e=k Example 3.1 O-!&9
;&+*%  *&  &   *& $&;! $$& -& ,
@ $& 61 O$$ :7 (19 &@-&$ * " "& $&-4& *& !-& -  p ˆ q ˆ 12 /& -&  !-& -& , &%* : "* 1) -&"  &+ *& 1) " , *& O: .9  % -& &->&  *& !- O&& $ . ,,9  " *" *&& * &-* , *& 12( >+ ;&- , &%* : ,,&&    & $ ,  & -&" &-&   1 12( 1) 1 dOC9 ˆ ˆ : : 12( (

*& - $&   n=k ˆ .=: ˆ  61.9 %;& 1+ CL T  *& & $+ &$&&& *& >+ &J O61(9 , dOC9 1
dOC9 ˆ pOu9dOu f Ou99 k u ˆ

1
1
pOu9dOu f Ou99 ‡ pOu9dOu f Ou99: k u2S k u2T

O6:2+ * *& !-& &H!&-& " >& $+ &$&&& >+ C *  PfdOu f Ou99 . kä0g " dOu f Ou99 " >& %&& * kä0 ,, dOu v i 9 . kä0 , &-* i ˆ 1 2 . . .  M O&& O61:99 &-& , "& &J&  *&* ,!-  1 , dOu v9 < kä0 ÄOu v9 ˆ O6:229 < , dOu v9 . kä0  *& &-& !  O6219 >&- & *& !&-& ! u pOu9 F1 ÄOu v1 9G . . . F1 ÄOu v M 9G   , "& &J& KOC9 ˆ


u

pOu9

M 

F1

ÄOu v i 9G

O6:269

iˆ1

*& & & O6219 >&- & dOC9 < ä0 ‡ B
KOC9:

O6:2.9

 ;&" , O62.9 ! $, " >& - $&& , "& - J  !-& -& C , &%* k "*   2bkR9c -&" , "*-* KOC9 , Oä9 ä0 9=B /& "  >& >&  J !-*  -& &-+ >! "& " >& >&  &!-& *& &@&-& , !-*  -& &-+ >+ & , random coding  *& " "& " ;&%& KOC9 "* &$&-   -& $>>+ >! ;& *& & ,  $>& !-& -& , &%* k "* 2b kR9c -&"L * ;&%& " >& *"  $$-* <  k ! 1 *! , !,J-&+ %& k *& ;&%& " >& ,Oä ä0 9=B , "*-*  ," *  & & $-! !-& -&  * KOC9 , Oä ä0 9=BL * -& " ,+ *& --! , *&& 6.  " ! #  *& ;&%& KOC9 ;&  !-& -& , &%* k "* M ˆ 2b kR9c -&"  , -!& "& ! $&-,+ *& $>>+ >! "& & ;&%% "* &$&-  *& %* -*-& ! !  >& -&+ &&  *& >! , *&  ;>& V   & -*& * -*&;& ROä9 *! , *& & , *& $, & pOu v9 &&  $>>+ >!  AU 3 AV -*&;% ROä9 *  IOU L V 9 ˆ ROä9 EFdOU L V 9G < ä:

*&

% >!  AU  AV & %;& >+

O6:279 O6:27>9

((

Discrete memoryless sources and their rate-distortion functions
pOu9 ˆ pOu v9 O!-& -9 v

pOv9 ˆ




pOu v9:

u

/& &@& * $>>+ >!  $ Ou v9 ˆ Ou1  . . .  uk  v1  . . .  v k 9 , A Uk 3 A Vk >+ ! % * *& !-&  & -*& & & +& *  >+ &J% pOu9 ˆ

k 

pOui 9

iˆ1

pOvju9 ˆ

k 

pOv i jui 9

O6:2)9

iˆ1

,

"*-* ," pOu v9 ˆ

k 

pOui  v i 9

iˆ1

pOv9 ˆ

k 

pOv i 9:

O6:2)>9

iˆ1

*& $>>+ % &  *& & ,  !-& -& , &%* k "* M -&" * "& "  *& & * %  *& -& C ˆ fv1  . . .  v M g *& $>>+ pOC9 ˆ

M 

pOv i 9

iˆ1

"*&& pOv i 9  %;& >+ O62)>9 O * $>>+ % &   & & &->& >+ +% *& !-& -&  -*&  NN II --%  *& $>>+ >! pOv9:9 &-% *& &J , KOC9 O&& O62699 "& >&%  - $!&  ;&%& EOK9D

3.2 The source coding theorem EOK9 ˆ


v1 :::v M

ˆ




pOv1 9 . . . pOv M 9

u

ˆ

M


pOu9

v1 :::v M iˆ1




pOu9

u


pOu9 pOv9F1

u

v2 A Vk

F1

ÄOu v i 9G

iˆ1

pOv i 9F1




M 

(3

ÄOu v i 9G

ÄOu v9G

M

:

O6:2:9

O *&  &$ >;& &&  *& ,- "*-*  >;! ,  >-!& >+  * , f Ox9   ,!- &J&   J& & A *&  M


f Ox9 ˆ ... f Ox1 9 . . . f OxM 9:9 x2 A

*& & !

x1 2 A

 O62:9 
pOv9F1

xM 2 A




ÄOu v9G ˆ 1

v

pOv9ÄOu v9

v

  O62:9 >&- & EOK9 ˆ


u

 pOu9 1




M pOv9ÄOu v9

:

O6:2(9

v

O  * $ *& && *! >& >&  && &-+ * O62(9 &$&& *& $>>+ *  !-& &H!&-& " >& NN$+ &$&&&II >+  !-& -& v1  . . .  v M -*&  NN II9 *& &@ &$  *& $,  *& &  , *& & !  O62(9  * & &J&  1 , dOu v9 < kä 0  IOuL v9 < kR 0 Ä< Ou v9 ˆ < *&"& "*&& IOu v9 ˆ %2 F pOvju9= pOv9G *& , O6229 Ä< Ou v9 < ÄOu v9  

pOv9Ä< Ou v9 < pOv9ÄOu v9: O6:239 v

v

, Ä< Ou v9 ˆ 1 *& IOu v9 ˆ %2 F pOvju9= pOv9G < kR0   pOv9 > 2 kR 0 pOvju9 &-&

pOv9Ä< Ou v9 > 2 kR 0 pOvju9Ä< Ou v9: O6:6% O6239  O66* &   O66.9 $$-* <  k $$-*& J+  & * -& M ˆ 2b kR9c  R9 . R0 O&& O61399 &@$O 2 kR 0
M9 , &@$F 2 kO R9 R 0 9 1 G $$-*& < ;&+ $+ &- -& * 1 Ä< Ou v9  1 ,, &*& dOu v9 . kä 0  IOuL v9 . kR0  
pOu v9F1 Ä< Ou v9G uv

< PfdOU V9 . kä0g ‡ POIOUL V9 . kR 0g

O6:679

*& $>>&  O6679 >&% #& ;& *& OU V9 $-& "*& $>>+ >!  &->&  O62)9 ! dOU V9 ˆ

k


dOUi  Vi 9

iˆ1

 *& ! , &$&& &-+ >!&  ;>& &-* , "*-* * & EFdOU  V 9G < ä , ä0 >+ O627>9  O6139   >+ *&

Problems

31

"&# " , %& ! >& O&& $$&@ 9 *& J $>>+  O6679 $$-*& <  k -&&  + IOUL V9 ˆ

k


IOUi L Vi 9

iˆ1

O&& &  $ 6:9   ! , &$&& &-+ >!&  ;>& &-* "* & IOU V9 ˆ ROä9 , R0 O&& O6279  O61399   >+ *& $$- , *& "&# " *& &- $>>+  O6679 $$-*& <  k -&&  >%  *&& ,- "& && * *& !$$& >!  EOK9 %;& >+ O66.9 $$-*& & !& *& ,- * ROä9 &&  >& -+ -;&@9 *" * *& k & & -*& &J&  *& & , *& $, ,

*&& 62 * EFdOU9G < kä  IOUL V9 ˆ kR1 Oä9

*& >?&- , * $>&  , +!  $;& * ROä9  -!!  ä ˆ ä    * >+ !$$+% *& &  *& ,"% %! & & Q1  Q2  . . . >&  &H!&-& , -*- -& &->% *&   $>>& , & -*& -*&;% ROä1 9 ROä2 9 . . .  "*&&  n!1 ä n ˆ ä   *& *&&   !>&H!&-& Q n1  Q n2  . . . "*-* -;&%&   -*@ Q *& & -*& -&$%  Q * EFdOU  V 9G ˆ ä   IOU L V 9 ˆ  k!1 ROä n k 9 *! ROä  9 <  ä!ä‡ ROä9   R  -!!  ä ˆ ä    $ (< "& *"& *  !,J-& - , RO ä  *&& &@  & -*& pOvju9 -*&;% ROä9 !-* * *& -*- @ QOu v9 ˆ pOvju9 * *&  & + &+  D & QOðOu9 rOv99 ˆ QOu v9 ,  u 2 U  v 2 V  F HintD , Q< Ou v9 &->&  & -*& -*&;% ROä9 &J& Qi Ou v9 ˆ Q< Oð i Ou9 r i Ov99  *" * Qi &->& -*& -*&;% ROä9 , i ˆ 1 2 6 . . . *& &J&  n & 1 QOu v9 ˆ n 1 iˆ< Qi Ou v9 "*&& n  *& & -  !$& , *& & , ð  rG 3.7  *&  %&& ! *& - $! , ROä  9     $& &

*& ,"% &!  ,& *&$,! *"&;& '&,+ *   -&-  &-* u 2 AU  & BOu9 && *& & , NN>& & &$&& ;& , uII *  BOu9 ˆ fv 2 AV D dOu v9 < dOu v99 ,  v9 2 AV g *& *&& &@  >-#" & -*& pOujv9 -*&;% ROä  9 !-* * pOujv1 9 ˆ pOujv2 9 "*&&;& v1  v2 2 BOu9

$$+ * &!  *& $>& &;&  *&  @

< D ˆ 1 1

, J% ROä < 1 1

 9

, *& !-&

1 1 1 6 6 6

 1 < : 1

3.8 &  J@& !-& p ˆ O p1  p2  . . .  pr 9   @ D "* ^ "*-*  & ,!- ROä9 &  &"  @ D , & , D >+ %  - wi  *& i* " *  ^ j9 ˆ dOi j9 ‡ wi  *" * *& &" & ,!-  dOi  ^ ROä9 ˆ ROä w9 "*&& w ˆ i wi pi  & * &!  *" * *&&   &&  , %&&+  ! % * ä  ˆ !  ROä99 !$$& *&  @ D * *& $$&+ * &-* -!    r $& ! , Od 1  d 2  . . .  d r 9 &J& á i d i ˆ äg O&& > 1(9 ;& * ÖOä9 ˆ @f HOá1  . . .  á n 9D iˆ1 ROä9 > HOU 9 ÖOä9 "*&& HOU9  *& !-& &$+ >+ ;&,+% *& ,"% &$D O9 ,  OU  V 9   & -*& -*&;% ROä9 ROä9 ˆ HOU 9 v pOv9 HOU  jV ˆ v9 O>9  , äOv9 ˆ u pOujv9dOu  v9 *& HOU jV ˆ v9 < ÖOäOv99 O-9 v pOv9ÖOäOv99 < ÖO v pOv9äOv99 < ÖOä9 3.11 O!9 , " D * *& ,!*& $$&+ * &-* row   $& !

Notes

3.12

3.13

3.14 3.15 3.16 3.17

36

 , &;&+ *& " *" * ROä9 ˆ HOU 9 ÖOä9 , ä  < ä < ä @ , *& !-&  + &- Oi9 & "  I "* $! $*>& AOi9 U  !$! $*>& A V   i  ,!- d Ou v9 i ˆ 1 2 *& product * !-& $*>& O29 O19 O29 AO19 @ "* && U 3 A U  & $*>& A V 3 A V    O19 O29 dFOu1  u2 9 Ov1  v2 9G ˆ d Ou1  v1 9 ‡ d Ou2  v2 9 O*+-+ * -&$  " &$&& $& !-&9 *" * *& & ,!- , *& $!- !-&  ROä9 ˆ ‡ fRO19 Ot9 ‡ RO29 Oä t9g "*&& RO19  RO29 & *& & ,!- , !-& 1  2 &$&-;&+ *! *" * *& ROä9 -!;&  >& >+ % >* -& , *& -!;& RO19  RO29  $  *& " -!;& *;% *&  & $& O&! !&  *9 O&,&- -&  *& && &-: 9 , q    ! >&  , AU ˆ AV ˆ f+ dOu v9 ˆ fju vj jq u ‡ vjg , q ˆ 2t 2 ‡ 2t ‡ 1 -& * !-& -& , &%* k ˆ 2 "* M ˆ q -&"D C ˆ fOv1  v2 9D v2  O2t ‡ 19v1 O  q9g *" * *& $*&& , ! t ! *&& -& " - $&&+ -;& *& & , q 2 $ Ou1  u2 9 "*! ;&$ O $*&& , ! t ! Ov1  v2 9  *& & fOu1  u2 9D dOu1  v1 9 ‡ dOu2  v2 9 < tg:9 *" * *& ;&%&  , * -&  16
O2t 6 ‡ 6t ‡ t9=O2t 2 ‡ 2t ‡ 19 '&,+ * dOC9 ˆ 1< 3 , *& !-& -& ,
@ $& 62 O $ (79 *" * *&& 6. &  !& , --! O9  &$-& >+ NNM < 2,b kROä9c IIL  , ä . ä   *  &  !& , O9  !-*%& >! O>9  &$-& >+ NNdOC9 < äII ;& * , < < x y < 1 M > & *" +! -! &@$-+ -*&;& "*  $ & >+ *& !-& -% *&& O *&& 6.9  ä ˆ ä @ 

Notes 1 O $ :79 &-*-+ dOu v9  -&  single-letter  &!&  %!*  ,  &!& * & &J&  -& k!$& Ou v9 ˆ OOu1  . . .  uk 9 Ov1  . . .  v k 99 &-+ *& * >+
H O619 2 O $ :)9 * &@ $&  !&  * O&& F27G $$ 2.)B2.:  % 2  $ 2)29  &->% * *& $&-! !-& *& "& NN. . . *& F!-&G $*>& - , *&& $>& &% 1 & , " && 12  ‡12 & %* *& *;& *& @ *  *"  F
@ $& 62GII 6 O $ :)9 *& &J * ," O
H O669  O6799 - >& %;& ,  >+ + !-&L &&
H O769  O7.9 . O $ ::9 && & 6 *$& 2 7 O $ :(9  ;&" , *&& 62 *& && + "& "*+ *& ,!- Rk Oä9 "&& !-&  *& J $-& *& &  ",  ,  + !-& "* & + &J O669  O679 #& &&  +$-+ Rk Oä9   &-&%

3.

Discrete memoryless sources and their rate-distortion functions

,!- , k , J@& ä O"&;& "& *  !+ !-& "* & +  * >#9 &- O &  $ , ! $!$&9 *& &J %;&  $,+ *& $, , *& -;&& , *& !-&-*& -% *&&  O&& *$& 7 &$&-+
H O71 +   "*  %    k," , *& !-& $>>& & &&D p1 < p2


< pr  &J& Sk ˆ iˆ1 pi  ä k ˆ S k 1 ‡ Or k9 pk   H k ˆ HO p k‡1 =O1 Sk 9 . . .  pr =O1 Sk 99 *& , ä k 1 < ä < ä k  ROä9 ˆ O1 S k 1 9 O H k 1 HOOä S k 1 9=O1 Sk 99 Oä Sk 9=O1 Sk 9%Or k 199 , k ˆ ?&- && &&# $ F1.G $$ 6!&    ;>& "* & <  ;-& ó 2  * -*&  ,& &$-&   % .1 !*& & *&&   NN-II -& "* &-* $! xL   bOx9 ˆ x 2   * &- "& * *" * *& -$-+B- ,!- , * -*&  %;& >+ CO â9 ˆ 12 %O1 ‡ â=ó 2 9:

O.:19

0!  , *& -&& -*& , *$& 2 "& * *" * CO â9 &$&& *& @ ! &  "*-* *& -*& -   &,&& ,  , *& ;&%& $! -  &-&  >& < â

Figure 4.1 *& ! -*&

37

3)

The Gaussian channel and source

&,& &;% , ! O.19 *"&;& "& $!&  %;&  >&, ,  &-$ , *" !-*  -*& %* &  $--& !$$& "& "*     &H!&-& , n & ! >& x1  x2  . . .  xn , & $  *&  T &-  & &H!&   * >+ -;&% *& xi I   -!! ,!- ,  & xOt9 O &J&& &I ! & xOt9 &$&& *& ;%& -  1* 9 & "+   *   J n ,!- ö i Ot9 i ˆ 1 2 . . .  n "*-* & *   *& &; F& xi & &-;&>& , xOt9D T xi ˆ xOt9ö i Ot9dt:

O.:.9

iˆ1


&-!& , * + O.29  iˆ1 x i  &-& "& ! *;& n 1
PT  x2 < n iˆ1 i n

O.:79

"*-*   + * *& $! ;&- x ˆ Oxp . . .  xn 9 - >& >+L  1   ! & "* 
!-& $*&& , ! PT 

*& - $-  * "*& xOt9   & *& &-&;& % " ,& >& , *& , x^Ot9 ˆ xOt9 ‡ zOt9 "*&& zOt9   & # , & $-& -  +$& , &  0* O*& 9 & "*-*  -!& >+ *& *&  % , *& &&-  *& &-&;&  * -&   &>&  & zOt9   "*& ! & $-&L , ! $!$& * & * *&&   ! >& N & T O-, O..99 < x^Ot9ö i Ot9dt *& " > *& & & x^i ˆ xi ‡ zi   ! +D "&   x ˆ Ox1  . . .  xn 9 "*&& x ! ,+ O.79  &-&;& ^x ˆ Ox1 ‡ z1  . . .  xn ‡ zn 9 "*&& z1  . . .  zn & &$&& &

4.1 The Gaussian channel

3:

& ! , -!& *  ?! *& ! && >+ *& ! -*& !-&  *& >&%% , * &- "* & ;-& ó 2 ˆ N < =2  $! - ⠈ PT =n

--%  O.19 *& -$-+ , * -*&  12 %2 O1 ‡ 2PT =nN < 9 > $& + > , " "& &J& *&   NN>"*II2 >+ W ˆ n=2T  >&;& * "& &  % n=T ˆ 2W + > $& &- *& -$-+ >&- & C ˆ W %2 O1 ‡ P=N < W 9 > $& &-

O.:)9

 ! O.)9  *I , ! &@$& , *& -$-+ ,  >  & $"& & ! -*& & * , W  P=N<  *& &! % NN"&>II ! -*& * -$-+ Cˆ

1 P  2 N
 $& &-

O.::9

& ! " &!  *&  --& , * &- *& &; , , ! O.19

%!+ "*
H O269 & ! &J& *& n* -$-+B- ,!- Cn O â9 , *& ! -*& >+   n
2 Cn O â9 ˆ !$ IOXL Y9 D EOX i 9 < nâ  O.:(9 iˆ1

"*&& *& !$& !  #& ;&  $ X ˆ OX 1  . . .  X n 9 Y ˆ OY1  . . .  Yn 9 , n &  ;&- !-* *D X *  -!! &+ ,!- pOx9 n
iˆ1

EOX 2i 9 < nâ

Yi ˆ X i ‡ Z i 

i ˆ 1 2 . . .  n

O.:39 O.:3>9 O.:3-9

"*&& Z 1  Z 2  . . .  Z n & &$&& O, &-* *&  , *& X i I9 & & *& ;& -$-+B- ,!- , *& ! -*&  " &J&  CO â9 ˆ !$ n

/& 

1 Cn O â9: n

&&+ $;& *& ,"% *&& 

O.:1& + & !-& ,+% O.39  O.3>9

*& >+ O.3-9 *& ? &+ , X  Y  %;& >+ pOx y9 ˆ pOx9 gOz9 "*&& z ˆ O y1 * 

x1  . . .  yn

xn 9  gOz9  *& ? &+ , Z 1  . . .  Z n 

1 gOz1  . . .  zn 9 ˆ &@$ O2ðó 2 9 n=2

   z2i  2ó 2

O-,
H O12:99 & Ai ˆ EOX 2i 9 -& X i  Z i & &$&& EOY 2i 9 ˆ EOX 2i 9 ‡ EO Z 2i 9 ˆ Ai ‡ ó 2    >+ *&& 111  n 1=n  n 2 hOY9 < % 2ðe OAi ‡ ó 9 : O.:119 2 iˆ1 n 2 2 " >+ O.3>9 iˆ1 OAi ‡ ó 9 < nO ⠇ ó 9   >+ *& *& &-B %& &- & &H!+ *& $!-  O.119   %& * O ⠇ ó 2 9 n  &-& n hOY9 < % 2ðeO ⠇ ó 2 9: O.:129 2 + *&& 11! hOYjX9 ˆ hOZ9 ˆ On=29% 2ðeó 2 O&&
@ $& 116   > 12:9 *! IOXL Y9 ˆ hOY9

hOZ9

  n â < % 1 ‡ 2  2 ó

O.:169

"*-* $;& * Cn O â9 < On=29%O1 ‡ â=ó 2 9  $;& *& $$&  &H!+ & X 1  X 2  . . .  X n >& &$&& & & *& O.39  O.3>9 " >& J&  Y1  Y2  . . .  Yn " >& &$&& & & O-, > 1239  IOXL Y9 ˆ hOY9 hOZ9 ˆ On=29%O1 ‡ â=ó 2 9

* - $&& *& $, , *&& .1 h Note Another way to de®ne Cn O â9 is to replace condition (4.9a) with X ! & + ,&+

+ ;!&

In Prob. 4.11 it is shown that these two de®nitions are equivalent.

O.:399

4.2 The Gaussian source

33

*& J &!  * &-  *& -% *&& , *& ! -*&   - $&&+ %!  *& -&& -% *&& 2.D

O-*&9 -& , &%* n  ?!  & , M n & ;&- fx1  . . .  x M g ,  -&" x ˆ Ox1  . . .  xn 9   & ;& *& -*&   &-&;&  y ˆ x ‡ z "*&& *& - $& , z & &$&& !  ;>& "* & <  ;-& ó 2  &-% !& , !-*  -&   $$% f , *& & ,  n & ;&- y  *& -& *& & $>>& POi9 E &$&& *& $>>& , &-& & %;& * *& i* -&" " & *  POi9 E ˆ Pf f O y9 6ˆ x i jx i  &S "*&& y ˆ x i ‡ z  >;& Theorem 4.2 Ocoding theorem for Gaussian channels9 Fix â > 2  O-9 POi9 , å for all i ˆ 1 2 . . .  M E Proof --%  *& $&-&% &   $>&  -*&  J&+ >&  ;>& X "* EOX 2 9 < â !-* * IOX L Y 9  >+ -&  CO â9 *& -&$% Y "  >& -&& >! -& IOX L Y 9 ˆ  IOX L FY G9  *& H!4 FY G >&- & J&  J& O&&
H O11:99L  " >& $>&  J  -&&  ;>& FY G !-* * IOX L FY G9 . R *& -&& & +& -*& "*-* *  $! *& ;!& ! & >+ X   !$! *& ;!& ! & >+ FY G "* $! -&  ,+ EOX 2 9 < â *! *  -$-+ %&& * R   *& &@&-& ,  -& ,+% O9 O>9  O-9 ,"  &&+ ,

*&& 2. h 4.2 The Gaussian source

* !-& "*& ,!  &  *& NN-&& & & +& ! !-&II *  !-& $*>& AU *& & ,  & ! >&  *& !-& !$!  && >+  &H!&-& U 1  U 2  . . . , &$&& &-+ >!&    ;>& "* & <  ;-& ó 2  ! >?&-  * &-   - $!& *& & ,!- , * !-& &;&  *& NNH!&&II  -&  "*-* *& & $*>& AV  % *& & , & ! >&  *&  >&"&&  !-& + > u   & + > v  %;& >+

1& & 2 ' 1 % ó , ä < ó 2  2 ROä9 ˆ ä ( < , ä > ó 2 :

O.:1.9

O.:179

  $& O ! , >9  % .2 , -!&  ! !-&  #&+  >& &-!&&    + # , %*&% &@$& & -&  J& ! >& , > & &H!&  &$&&  >+ & ! >& u "* $&,&- J&+   , &&  #" *& &,, >&"&& *& ! >& , > !&  &$&& *& &@$& & !- &  *& &!%   &J&  O.179 ROä9 &  ,- &$&& *&  ! $>& ! >& , > !,J-&  &$&&  ! O;-& ó 2 9  ;>& , *& @ ! $& >& & H!& &  ä * ,- " >& ;&J& >+ *& !-& -% *&& 

*&& .7  >+ *& &! , *$& 7

 &>* O.179 "& J &J& *& k* & ,!- Rk Oä9 , *& ! !-& "* &$&-  *& &H!& & -& >+ O&&
H O6699 Rk Oä9 ˆ , fIOUL V9 D EOkU

Vk2 9 < käg

"*&& *& J !  #& ;&  $ , k &  U ˆ OU 1  . . .  Uk 9  V ˆ OV1  . . .  Vk 9 !-* *D U1  U2  . . .  Uk & &$&&  ;>&

O.:1)9 ;&-

& ! , U  V  %;& >+  -!! &+ ,!- pOuv9

O.1:-9

*& ;& & ,!- ROä9  *& &J& O-,
H O6799 >+ ROä9 ˆ , k

1 Rk Oä9: k

O.:1(9

 *$$&& , -&& & +& !-&  *$& 6  " ! ! * *& J !  O.1(9  &+ -*&;&  k ˆ 1 &,& $;% * ,- *"&;& "& %;&  -& , &@$& , R1 Oä9 "*-*  , -!& *& ;!& , ROä9  "& Theorem 4.3

& 2 ' 1 % ó 2 R1 Oä9 ˆ ä (
ó 2 :

Proof -# ä å . & !-* *D

IOU L V 9 , R1 Oä9 ‡ å U  ! "* EFOU

& 9 O.:13-9 O.:139

* *  $>& ," , &J O.1)9  *& - O.1:9 + O.13-9  O.139  ä> pOu v9Ou v92 u v  ˆ

 pOv9 pOujv9Ou

v92 u v

O.:2+ O.2 12 % ó 2 =äL >! -& *  !& ,  å . & ! "* ;-& ó 2 å *& % O.1:9  J&   R1 Oä9 < IOU L V 9 ˆ hOU 9 hOG9 ˆ 12 % 2ðeó 2 1 2 å9 ˆ 12 %F1 ‡ å=Oó 2 å9G -& *  !& ,  å . &  *&  $ , *& $, , *&& .6  ,& -& *& NN>-#"II & -*& , - $!% ROä9 >&-!&  - >& &$-&   % .6  $& , *& ,- * U  *& !-&  V  *& &M   NN,"II ;& , * & -*& && > .1(9

4.2 The Gaussian source

1-#" & -*& ,  ! !-&

/& - " - $!& ROä9 Theorem 4.4 Rk Oä9  ˆ kR1 Oä9 @ 12 % ó 2 =ä < 

for

all

k,

and

so

ROä9 ˆ R1 Oä9 ˆ

Proof -# å . &  $ ,  ;&- ,+% O.1:9  IOUL V9 , Rk Oä9 ‡ å: O.:279 k

*& >+ *&& 1( IOUL V9 > iˆ1 IOUi L Vi 9 , "& &J& ä i ˆ EFOUi Vi 92 G *& >+ *& &J , R1 Oä9 IOUi L Vi 9 > R1 Oä i 9  >+ k k O.1:>9 iˆ1 ä i < kä !*& & iˆ1 R1 Oä i 9 > kR1 Oä9 > kR1 Oä9 "*&&  k ä ˆ k 1 iˆ1 ä i  -& R1 Oä9  -;&@ [  -+ &-&% >+

*&& .6 &-& Rk Oä9 ‡ å . IOUL V9 >

k


IOUi L Vi 9

iˆ1

>

k


R1 Oä i 9

iˆ1

> kR1 Oä9 > kR1 Oä9: -& *  !& ,  å . ó 2 *& $,      &,  > .2& $&-J& "* d%2 M e >  *& &!% $&+ > &H!&   dOC9  Theorem 4.5 Let ROä9 ˆ @ 12 % ó 2 =ä < denote the rate-distortion function of the Gaussian source with respect to the mean-squared error distortion criterion. Fix ä > 9 dOC9 , ä9 Proof , ä > ó 2  *& >+ -*% *& %& -&" < , &%* 1 "& " -*&;& "*  $ & ,  *& *& * ä , ó 2 & OU  V 9 >& *& >-#" & -*& -*&;% ROä9 &->&  *& $, , *&& .6L && % .6 *& ä1  ä2  * ä , ä1 , ä2 , ä9: &@ $-# ,!- á  â &-* ! % + J&+ * EfFáOU 9 EfFU "*&&

âOV 9G2 g , ä1  áOU 9G2 g ˆ å

O.:2:9 + ;!& !-* O.:2(9 O.:239

Problems  å ‡ 2 åä2 ‡ ä2 , ä9:

1 .229 /& " -& *& -&& & +& !-& áOU 9 "* & $*>& &H!  *& %& , â   ,!- dOu v9 ˆ Ou v92  -& >+ *&& 17 IFáOU 9L âOV 9G < IOU L V 9 ˆ R1 Oä9 , "& && *& & ,!- , * !-& >+ R  ," , O.2(9 * ROä1 9 < ROä9 , R9:

O.:619

*! >+ *& !-& -% *&& , -&& & +& !-& O *&& 6.9 *&& &@  !-& -& , &%* k , *& !-& áOU 9 ,+%D M < 2b kR9c 

O.:629

dOC9 , ä2 :

O.:669

/& " -& !% * -& , *& % !-& U , u ˆ Ou1  . . .  uk 9   >+ ;&- , &%* k  , v i  *& !-& -&" -&  áOu9 ˆ OáOu1 9 . . .  áOuk 99 *& ku v i k ˆ kFu áOu9G ‡ FáOu9 v i Gk   >+ -*"4I &H!+6 EOkU

Vi k2 9 < EFkU

áOU9k2 G ‡ EFkáOU9

‡ 2EFkU

Vi k2 G

áOU9k2 G1=2 EFkáOU9

Vi k2 G1=2 

O.:6.9

"*&& Vi && *& -&" -&  *&  ;&- U ! , O.239 EFkU áOU9k2 G ˆ kå  , O.669 EFkáOU9 Vi k2 9 , kä2  &-& - >% O.6& "* O.629 $;& *& *&& 

h

EFkU

Problems 4.1 &  -*& "* AX ˆ AY ˆ *& & & , "*-* *& !$! Y  *& ! , *& $! X  $ &$&& &  ;>& Z "*-*  % !, + >!&  12 12  !$$& *& $! X  -&  ,+ jX j < â , â   &%& *" * *& -$-+ O&J& %!+ 
H O.(9 O.39 O.1>+ PE , 2 nO R< R9  O *  %& * *&& .2  *  %;&  &@$- & & , *"   PE - >& &   ,!- , n   "&#&  * R< , CO â9  * , & R< , R , CO â9  &  *" * PE ! <  $>&9 4.3  *" * ,  -& fx1  x2 g "* + " -&" , *& &-& -*& *& & && O
!-& -&9  *& &-&;& ;&- y *& & $>>+  %;& >+ P2 Fx1  x2 G ˆ QOkx1 x2 k=2ó 9 p  1 s 2 =2 "*&& QOá9 ˆ O1= 2ð9 á e ds 4.4 &@ *" *  %&& , *& -& fx1  . . .  x M g  &-& >+ *& && -&" &%+  , POi9 E && *& $>>+ , &-% & , x i   & *& POi9 E
 .. > *& ,"% & & , *& ;&%& ;!& EFPOi9 -& &  &&-& &$&&+ E G "*&& *& -&"    --%  *& >! Pfx ˆ ‡ âg ˆ Pfx ˆ âg ˆ 12D  n 1‡ã Oi9 EFP E G , M
: 2

Problems 4.8 " - $&& *& $, , *& R< *&& 4.9 + *" *D O9 

R< O â9 ˆ % 2

O>9 

R< O â9 1 ˆ : CO â9 2

â!‡1

â!&  "& * !+ *& &,,&- , H!4  *& ! -*& && >+ C O rs9 O â9 *& -$-+ , *& ! -*& , "&  * *& $! X ! &   r - ;!&  * *& !$! Y * >&& H!4&  & ,   s ,,&& ;!&  + "& &J& , &-* n C Onrs9 O â9 ˆ !$fIOX 1  . . .   X n L f OY1 9 . . .  f OYn 99g "*&& &-* X i n - ! &   r - ;!& iˆ1 EOX 2i 9 < nâ Yi ˆ X i ‡ Z i  "*&& 2 *& Z i I & &$&& & &  f  + ,!- "*-* ! &   s ,,&& ;!& *& C O rs9 O â9 ˆ !$ n n 1 C Onrs9 O â9 4.10 *" * C Ors9 O â9 ˆ C 1O rs9 O â9 4.11 *" *  rs!1 C O rs9 O â9 ˆ 12 %O1 ‡ â=ó 2 9 F#&-* , $,D  *" * ,  J@& ;!& , r  s!1 C O rs9 O â9 ˆ C O r19 O â9  %;& >+ *& , ! C O r19 O â9 ˆ !$fIOX L Y 9g "*&& *& !$& !  #& ;&   ;>& X ! %   r ,,&& ;!& EOX 2 9 < â  Y ˆ X ‡ Z "* Z  &$&& & & % *& , ! IOX L Y 9 ˆ hOY 9 hO Z9 O-, > 12:9 *" * C O r19 O â9 < 12 %O1 ‡ â=ó 2 9 &@ & X >&  & & & I 1  . . .  I r >&  $ , *& & &  r &;  , &-* i ˆ 1 2 . . .  r -*& xi 2 I i "* jxi j     $>& &J& *& &"  ;>& X 9 >+ PfX 9 ˆ xi g ˆ PfX 2 I i g i ˆ 1 2 . . .  r *& C O r19 O â9 > IOX 9L X 9 ‡ Z9 "  r ! 1  *& $ >&- &  -&%+ J& hOX 9 ‡ Z9 ! hOX ‡ Z9 ˆ 12 %Oó 2 ‡ â9 >+ &>&%!&I   & -;&%&-& *&&  &-& 

r!1

C O r19 O â9 ˆ 12 %O1 ‡ â=ó 2 9:

4.12  J@& r  s *" *  â=ó 2 !1 C O rs9 O â9 ˆ O% r % s9 4.13 *" * , r > s *& C O rs9 O â9 ˆ C Oss9 O â9 F&!  1 !&   ! 2&+ 0G 4.14 , â=ó 2 ˆ á2  *" * C O219 O â9 ˆ á2 1 gO y9% -*Oá ‡ á y9 dy 2 "*&& gO y9 ˆ O2ð9 1=2 e y =2  *" * ,   á C O219 O â9 ˆ á2 =2 á. =. ‡ &;& * *  ;!+ *&  &  *& !H!4& -$-+ CO â9 ˆ 12 %O1 ‡ á2 9 &-& *& ,# *&& NN   %&  >+ $! H!4 &I *!II O>! && > .3>9 && , á < 1 *&  C O219 O â9=CO â9  "+ > & ó 2 %& ,  i ó 2i  â 4.18 *& >?&- , * &@&-&  , +!  &;&  NN,"II & -*& * -*&;& ROä9 , *& ! !-&  &H!& &  -& &- "*  &&&D U   & & ä < ó 2   V ! >& ,!  * EFOU V 92 G ˆ ä  IOU L V 9 ˆ 12 % ó 2 =ä *" *   $>&  #& V ˆ áU ‡ Z "*&& á   -  Z  ! "* ;-& ó 2  >+ -*% ä  ó 2 $$&+L   #& V ˆ âOU ‡ Z9 "*&& â  -  Z  !

*&& " NN,"II & -*& - >& &$-& #& *D

4.19 &  !-& U 1  U2  . . . "*&& *& Ui I & &$&&  ;>& "* -  &+ ,!- pOu9  ;-& ó 2  *" * *& &  ,!-  , * !-& O&& $ 1 ó 2  O&& $, , *&& .. & & >& * OU V9 ! *;&  -!! ? &+  &J% V  < "  9

Problems

1?&- , * &@&-&   *" * , - O.1:-9 "&& &$-& >+ - O.1:-99 *& ;!& , Rk Oä9 "! >& !-*%&  *& ,"% #&-* ,  $,   ! & * k ˆ 1  ä , ó 2  O *& %&&4  %& k  ä  H!& &+   * " >& %;&9 ! ?>   !$$+ *& & *! & R91 Oä9 && *&  ! $>& ;!& , IOU L V 9 !>?&-  O.1:9 O.1:>9 O.1:-99  *" * R91 Oä9 < 12 % ó 2 =ä & OU  V 9 >&  & -*& -*&;% R1 Oä å9  &->&  *& $, , *&& .6 *& å . &  && - ,!- , V ! % + J&+ + ;!& !-* * EFOU V 992 G , ä *& IOUL V 99 < IOUL V 9 ˆ R1 Oä å9   R91 Oä9 < 12 % ó 2 =Oä å9 -& *  !& ,  å . 12 % ó 2 =ä & OU  V 9 >&  & -*& ,+% O.1:9 O.1:>9 O.1:-99 , "*-* IOU L V 9 , R91 Oä9 ‡ å & V 9 ˆ V ‡ G "*&& G   & & &$&& , V *& *& OU  V 9 & -*& J& O.1:9  O.1:-9  EFOU V 992 G < ä ‡ å -& IOU L V 99 < IOUL V 9  ," * R91 Oä9 ‡ å . 12 % ó 2 =Oä ‡ å9 -& *  !& ,  å R91 Oä9 > 12 % ó 2 =ä 4.22 *" * *&& & ,!- áOu9 âOv9 "*-* ,+
H O.2(9 O.239  O.6!&  ;>& "*  -  &+ ,!- pOu9   -  ,,&& &$+ hOU 9 & f Ox9 >&  >+ &%;& &;& ,!- ,  & ;>& x  &J& *& &  ,!- , *& !-& "* &$&-  *&  &!& dOu v9 ˆ f Ou v9  ROä9 ˆ ,fIOU L V 9g *& J ! >&% #& ;&  $ ,  ;>& OU V 9 "*&& U  >!& --%  *& &+ pOu9 OU V 9 *;&  ? &+ ,!-  EF f OU V 9G < ä O -!+  >& -& "* *& $&& , *$& . "& *! &J& Rk Oä9 %!+ "*
H O.1)9  ROä9 ˆ , k k 1 Rk Oä9 "&;&  "! ! ! * Rk Oä9 ˆ k R1 Oä9  * *& &J , * $>&  &+ *&  &9 &J& öOä9 ˆ !$fhOX 9 D EF f OX 9G < äg ;& * ROä9 > RL Oä9 ˆ hOU 9 öOä9 HintD  *" * öOä9  -;&@ \  ä *& , äOv9 ˆ f Ou v9 pOujv9du IOUL V 9 ˆ hOU9 hOU jV 9   hOU jV 9 ˆ qOv9 pOujv9%  < qOv9öFäOv9G dv

1 du dv pOujv9

  < ö qOv9äOv9 dv < öOä9: 4.24 O!9 /* *&  , > 167 &;!& *& * "& >! RL Oä9  *& ,"% -&D

11
9 pOu9 ˆ e juj= A  f Ox9 ˆ jxj: 2A 2 O-9 pOu9 ˆ O1 ‡ u 2 9 2  f Ox9 ˆ jxj: ð 4.25 + -!-%  $$$& & -*& *" * ROä9 ˆ RL Oä9 , *& !-& , > .2.> 4.26 FNoteD *& &! , * $>& - >& !&   % $  *& !+ , *& & ,!- ,  ! !-& "* & + && &%& F16G *$& .G & k NN$&II ! !-& U1  U 2  . . .  Uk  *  *& Ui & &$&& & & &J& ROä9 ˆ , f OUL J !  ;& *& $ OUL V9 "*  ?  k V9g "*&& *& &+  iˆ1 EFOUi Vi 92 G < kä *" * ROä9  %;& $ &-+ >+   k
1 ó2 ROä9 ˆ % 2i  2 ó ‡ iˆ1 äˆ

k
iˆ1

Oó 2  ó 2i 9

"*&& *& $ && ó 2 %& , <  @ i ó 2i  4.27 '&,+ * *& ! -*&  & +&  *& && , *& &J  $ ..

Notes 1 O$ 3)9  0* & *& $&- &+  N< ˆ kT  "*&& k ˆ 4 0  - ˆ 1:6( 3 1< 26 ?!&=85  T  *& &,,&-;& & & $&!& -!+ *& $&- &+  &$&&  *& ,&H!&-+   %;& >+ 0 PO f 9 ˆ hf =Oe hf =kT 19 "E*&4 "*&& f  *& ,&H!&-+   *&4  h ˆ -#  - ˆ ):) 3 1< 6. ?!&&- *& & * zOt9ö i Ot9dt   !  ;>& , ;-& N< =2  !& + , *& % ö i Ot9 *  ,  &&%+ -J&  ,&H!&-& "*&& *& $$@  PO f 9  N<  ; O&& & 29 "&;& * $$@   +$-+ ; !$  ;&+ *%* ,&H!&-& &&L , &@ $& , f , & , - !-% *& !$! ,  ,  !-& ;&  + -*& *!%*! * -*$& "& * #& *& >-# % *"  % 71  ! $%  '!+ + & - !- +& - >& !>! & !& *& & , % 71 *& &-& >-# &$&&  *&  $-&% O"*-* + -!& H!4 ! &-  "&  -% , & -9 $&, &  *& !-& !$! >&,&     ! & * *&& &@ &%& k  n !-* * *& +& $-&& !--&;& >-# , k !-& + > &$&&+  -;& *&  !--&;& >-# , n -*& $! + > O &@ $& k -! >& #&  *&  ! >& , !-& + > & & !% *& ,& & , *& +&  >! ,  +&  k  !-*  & * *9 *& -*&  ! &  --&$ -*& $! + >  -&& &;   &$&  &  -*& !$! + >  && -&$&-&  *& $!1 *& ! $ * *& -*&  NN-&& &II  &-;& >!  ;&+ -&  NN-!! &II -*& - >& ;&"&  NN-&& &II "* ;&+ &  O-, $$ 3)B3:9 *& NN&-&II >-# &$&&  *&  $-&%

Figure 5.1 %&& -

112

!- +& 

The source±channel coding theorem

116

$&, &  *& -*& !$! >&,& &;&+  *& &   ! & * *& &-& $-&& !--&;& >-# , n -*& !$! + > &$&&+  -;& *&  >-# , k & + > *& & &H!&-& V ˆ OV1  V2  . . .  Vk 9 >&% *& +& I & & , *& !-& &H!&-& U ˆ OU1  U2  . . .  Uk 9 /& " $!&  &;&" *& >- --&$ ,  &  *&+ $$+  *& >-# % , % 71  *& ,"% -! "& * &,&  ;&+ ;%!&  %&& -*&  !-& &L *& && *! >& "& *"&;& * "& *;& $;& *& &H!& -% *&&  + , -& $&- -&  "& -& *& -*& /& & %;& *& -*&I $! $*>& AX  *& !$! $*>& AY L "* &-* $! + > x 2 AX *&&  -&  &%;& & ! >& bOx9 *& NN- , &% xII & %&&+ , &-* x ˆ Ox1  . . .  xn 9 2 A nX  bOx9  *& - , &% x  "& & %;& *& -*& - *  , &-* n "& & %;&  - $>>+ >!2 * " !  - $!& *& >! , *& !$! Y ˆ OY1  . . .  Yn 9 -& "& #" *& $! X ˆ OX 1  . . .  X n 9  &-* $;& &%& n  â > + Cn O â9 ˆ !$fIOXL Y9D EFbOX9G < nâg:

O7:19

 O719 *& !$& !  #& ;&  n &  ;&- X ˆ OX 1  . . .  X n 9 #% ;!&  A nX  ,+% EFbOX9G < nâ  Y  *& n &  ;&- #% ;!&  A Yn "*-* &->& *& -*&I !$! , X  !&  *& $!  * -&@ X  -&  n  & â >& test source *& -$-+B- ,!- CO â9 , *& -*&  &J& >+   1 CO â9 ˆ !$ Cn O â9D n ˆ 1 2 . . . : O7:29 n  %&& CO â9   -!! -;&@ \  -+ -&% ,!- , â > â  ˆ , fbOx9D x 2 AX g *& -!- ,- >! CO â9  *& ,"% O  !!   $& % & >& 29 Channel Coding TheoremD @ â< > â   *& , + $& O â C9 å9 "* â . â<  C9 , CO â< 9 å . & AU   & $*>& AV   , &-* $ Ou v9 2 AU 3 AV  &%;& & ! >& dOu v9 *& NNII * &! , *& !-& + > u ;&  *& &  v & %&&+ , &-* $;& &%& k  &-* $ Ou v9 2 A Uk 3 A Vk  dOu v9 &!& *&   , *& !-& &H!&-& u ˆ Ou1  . . .  uk 9 ;&  *& & &H!&-& v ˆ Ov1  . . .  v k 9  "& & %;& *& !-& - *  , &-* k "& & %;& *& >! , U ˆ OU 1  . . .  Uk 9 *&  ;&- * &->& k !--&;& !-& !$!2  &-* $;& &%& k  ä > ä  &J& Rk Oä9 >+ Rk Oä9 ˆ , fIOU V9D EFdOU V9G < käg

O7:69

"*&& *& J !  #& ;&  $ OUL V9 ˆ OU1  . . .  U k L V1  . . .  Vk 9 , k &  ;&- #% ;!&  A Uk 3 A Vk !-* * *& % >! , U  * %;& >+ *& !-& -  EFdOUL V9G < kä *! "* "& & &+ ;+%  O769  *& - >! , V %;& U  * - >!  -&  k  & test channel *& J !  O769    #& $-& ;&  k & ä >& & -*& *& rate-distortion function , *& !-&  &J& >+   1 ROä9 ˆ , Rk Oä9D k ˆ 1 2 . . . : O7:.9 k  %&& ROä9   -!! -;&@ [  -+ &-&% ,!- , ä . ä   *& -!- ,- >! ROä9  *& ,"% *&&  Source Coding TheoremD @ ä . ä   *& , + $ Oä9 R99 "* ä9 . ä  R9 . ROä9 ,  !,J-&+ %& k *&& &@  !-& -& C ˆ fv1  v2  . . .  v M g , &%* k !-* *D O9 M < 2b R9 kc  O>9 dOC9 , ä9 O&- * dOC9 *& ;&%&  , *& !-& -& C  &J& >+ dOC9 ˆ k 1 EFd  OU9G "*&& , &-* u 2 A Uk  d  Ou9 ˆ fdOu v j 9D j ˆ 1 2 . . .  Mg:9 /& " &!  *& >-# % , % 71 >&;& * -& *& &-&  *& &-& *;& >&& $&-J& "& - ;&" U X Y  V   ;&- , U  &J&  >&   ;&-  *& J $-& 

The source±channel coding theorem

117

*& - $>>& , X %;& U , Y %;& X  , V %;& Y & &$&-;&+ %;& >+ *& &-&I &% *& -*& -  *& &-&I &% /& " &J& *&& $ && * " & ! !-* >! *" "& *& +& $&,  The average costD 1 EFbOX9G: n

O7:79

1 EFdOUL V9G: k

O7:)9

k : n

O7::9

∠The average distortionD äˆ The rate of transmissionD

rˆ

*& $*+- %J--& , *&& $ && *! >& >;!D â & ! *" !-* *& +& -  ! O  $&-*& $! >9L ä -& *" &>+ *& +&   *& !-& !$!L  r &!& *" , O ! , !-& + > $& -*& + >9 *& +&   % ,    %;& !-&  -*& "& "! #&  &%  +& "*   â   ä  %& r >! , -!& *&& & -K-% %

*& ,"% *&&  "*-*  *& -& &! , ,  *&+ & ! &@-+ "*     $>& Theorem 5.1 (the source±channel coding theorem). For a given source and channel: O9 The parameters â ä  r must satisfy r
9 Conversely, given numbers â . â   ä . ä   and r , CO â9=ROä9 it is possible to design a system of the type depicted in Fig. 5.1 such that â < â ä < ä and r > r Proof /& * J $;& O9  >&% >+ & #% * *& $, " &$& +  &J O729  O7.9    *& -% *&& 

*! *& J $ , *& *&& O & & -& *& NN-;&&II  NN"&# -;&&II , *& -% *&& 9 " >& &&  >& !&  &@& &+ %&& --! -&

11)

The source±channel coding theorem

*! !$$& "& & %;&  +& , *& +$& &$-&  % 71 ! J >&;  * *& &H!&-& OU X Y V9 ,  ;&-   #; -* OY &$&  U + *!%* X V &$&  X + *!%* Y9   >+ *& $-&% *&& O
H O11799 IOUL V9 < IOXL Y9:

O7:(9

&@ >&;& * -& EFbOX9G ˆ nâ O&& O7799  ," , O719 * IOXL Y9 < Cn O â9  *& *& * , O729 Cn O â9 < nCO â9   IOXL Y9 < nCO â9:

 -& EFdOUL V9G ˆ kä O&& O7)99  ," , > Rk Oä9  , O7.9 * Rk Oä9 > kROä9 &-& IOUL V9 > kROä9:

O7:39 O769 * IOUL V9 O7:1% O7(9 O739  O719 *L >! "* *& *&$ , *& -*& -% *&&  *& !-& -% *&&     *  $;&  , "  "& ! & * â . â   ä . ä    r , CO â9=ROä9 & %;& ! ?>   &%  &-&  &-&  * *& &!% â ä  r ,+ â < â ä < ä r > r  *& &-*- $& & &&- ! >& â<  ä<  ä1  C9  R9 ,+%D â



< â< , â

O7:119

ä



< ä< , ä1 , ä

O7:129

C9 , CO â< 9

O7:169

R9 . ROä< 9

O7:1.9

r , C9=R9:

O7:179

O * * - "+ >& &  *& -& , > 729 ! &-& " >& &%&  " %&  !-& &-&   -*& &-&  *"  % 72 /& " &->& *& !-& &-& , % 72 --%  *& !-& -% *&&  , !,J-&+ %& k < *&& &@  !-& -& C , &%* k < "* M 1 -&" "*&&

The source±channel coding theorem

11:

Figure 5.2 *& %&& &% , *& &-&.

M 1 < 2b R9 k < c

O7:1)9

dOC9 , ä1 :

O7:1:9



  -& &%& m O >& $&-J& &9 "& #& k ˆ k < m *& !-& &-& , % 72 $ *& !-& &H!&-& U , &%* k  m >-# , &%* k <   !$! *& m !-& -&" -&$%  * &H!&-& , !-& >-# *! *& & && ;&- W ˆ OW 1  . . .  W k 9 " "+ >&  &H!&-& , m -&" , *& %;& !-& -& C  $-! W ! &   M 1m < 2 mk < R9 - ;!& * - $&& *& &-$ , *& !-& &-& O&@-&$ , *& $&-J- , m9

 &->& *& -*& &-& "& J &&  &J& *& worst-case distortion , *& !-& -& C  &-* !-& &H!&-& u 2 A Uk <  & d @ Ou9 ˆ @fdOu v i 9D v i 2 Cg *! , *& !-& &  u  *& !-& &-% $-&  &*" %& *+"& d @ OU9 &$&& *& " $>& &!%  *& "-&  , *& -& C  &J&  >& *& ;&%& ;!& , *  4&   $&+ > >D DOC9 ˆ

1 EFd k
&;& && *&& &   2 mk < R9 - WI  >+ O7229  & * + - -&" ;% - $&&+ &->& *& &-&  % 71 "& *;&  &&+ , O72.9 * r ˆ k=n ˆ k < m=nm > r  , O7219 * n⠈ EFbOX9G < nâ  &   *" * *& ;&%&  ä  < ä    * "& ! , -!& &% *& &-&  % 71  %&& &%  &$-&  % 76 *& -*& &-& !&  *& &-& "*& &@&-&  $ & >+ *& -*& -% *&& L  $ *& + -&" Y ˆ OY1  . . .  Yn 9  & , *& -*& -&" + Z ˆ O Z 1  . . .  Z n 9 + *& !-& &-&  % 76 --&$  $! *& -*& -&" Z  !$! *& &H!&-& V ˆ OV1  . . .  Vk 9 , m !-& -&" -&$%  Z , *&&  & O * -&$&-&  *& & %;& >+ *& !-& &-& , % 729 ,  &H!&-& , m !-& -&" -! *;& %;& &  Z & ! ! & *& !-& &-& !$!  J@& NN! &H!&-&II v< ˆ Ov& AV  ;% - $&&+ &->& *& - !- +& , % 71B76 ! & % #   - $!& *& ;&%&  ä ˆ k 1 EFdOUL V9G

  * "& !-&  &"  ;>& B "*-* & ! "*&*&   *& -*& &-&  % 76 !--&& *!  < , &-& !--&&L ZˆX Bˆ 1 , &-& ,L Z 6ˆ X:

Figure 5.3 *& %&& &% , *& &-&.

The source±channel coding theorem

113

*&D EFdOUL V9G ˆ EFdOUL V9jB ˆ % O7279 O72)9  O72:9 "& > ä ˆ k $ &

O7:2:9 1

EFdOUL V9G , ä  h

Discussion /& ! & *& ;&%& - â  J@& O * *& - ,!- bOx9  &-+ 4&  "*-* -& *& -$-+ ,!- CO â9   - C @  *& -*& -$-+9  !+ *& &,, >&"&& *& & r  *& ;&%&  ä , *& - !- +& , % 71 *& & , *& !-&B-*& -% *&& & ! 4&  % 7. & O9 , *& !-&B-*& -% *&&  * , + &4>& +& *&

12
& J& "* & *;% $ && Or äO1 ‡ 1< 7< 99M

Problems 5.1  &@$& && "*&  &%  +& , - !-% >&; ,  !  $-& ;  >+ + &- -*& "*-*D --&$ 1 1+  $% *& $-&   & , R  $& $& &- O*&  $& & & &9  &-% *&  $& >&,&   !$$& *& - &&  ;&%& &H!& & ,   ä  - $&  ;&%& ,   B  $& +  *& -*& /*-* , *& ,"% *&& & , OB ä R9 &  $-$& &4>&P B T (). 2732 .62
+ + &- -*& "* " > & $>>+ p &,&%  % 7. *" * &;&+ $  *& >!+ ,+% ä ˆ p  &4>& O& * , p ˆ < * $ , *& >!+   -!! 9 *" * *& - â . â   $ O>9 , *& !-&B-*& -% *&& - >& & ;& *" * *& H!+ DOC9 &J& ,  !-& -& 
H O71(9  J& $;& * , &-* v 2 A Vk < *& &@$&- EFdOUL v9G  J& *& *" * * -  J& , *& ! !-& , *$& . O
  &9 * $>& !& *& $>& $, ;;&  >!%  NN,-&II - !- +&  -*&;& "*  $ & >+ *& !-&B -*& -% *&&  !$$& "& "  - !-& *& !$! ,  >+ + &- !-& ;&  >+ + &- -*& "*& " & 1 $>>+  1<  O ! & *&&   - -& "* -*& !&9 !*& ! & *   &H!&  *;& ä < + J &%%  !-& -& , *&  "* dOC9  & $&-& -& *& ,"% J%!&D

&& U   & & "*-* $& *!%*  &;-& O $J&9 "*-* !$&  >+  - ë *&  & & Z &$&& , U  &    J+ *& &!  $& *!%*  &;-& O&!9 "*-* !$& >+ *& - ì *& $! X  *& -*& ! ,+ EOX 2 9 < â *" * *&& &@   ä  - ë ì !-* *D O9 ROä9 ˆ CO â9 O>9 EFOU V 92 G ˆ ä "*&& ROä9  *& & ,!- , *& !-&  CO â9  *&

122

The source±channel coding theorem -$-+B- ,!- , *& -*& &-& *" * *& &H!+ k=n < CO â9=ROä9 - >& -*&;& "* &H!+ "* k ˆ n ˆ 1

Notes 1 O$ 1129 -& * -*& "*-* -*&;!+ &  &&& + >  *&  & & & *! >&+ ! $%  2 O$$ 116 11.9  & , *& %;& &J , CO â9  ROä9  >& ! >%!! "& ! ! & * *& -*&  !-& & stationary *  *& -   &$&  "*& "& >&%    *& !-&I !$!  "*& "& >&%  !& *& -*& 6 O$ 11)9 -!+    &-&+  ! & * *& &-&  *& &-& & && - &;-&  &  $;& *& &H!+ k=n < CO â9=ROä9  *  &H!&  * *& &-&  &-& >& !-* * *& &H!&-& OU X Y V9 ,   #; -*L , &@ $&  &-& !-* * X &$&  +  U >!   & -+ %&&&  ;>&  "& "! >& --&$>& . O$$ 11: 11(9 *& $&-! -!- "& !&  *& $, , *& !-&B -*& -% *&&   "*-* *& &-&  &-& "&& ,-&  !-&  -*& &-&  &-& &&;& - &  - &>+ >& %!& * >+ % *& !-&  -*& -% &$&+ "& + & !$ "*  +& & - $&@ * &-&+ O&& > 7:  7( &%9 "&;& , *& $ , O â ä r9 *%   -& &;&+ !-* $& -*&;>& >+ + &   -*&;>& >+  ,-& +& 

6 !;&+ , ;-& $- ,  &

6.1 Introduction  * -*$& "& >&K+ ! 4&  & , *&  $ &!  ,  *&+ "*-* "& *;&  >&& >&  &  & /& * %;&  $, >! & &,& *& &&& && &&"*&& !!+   &@>#  & &   % $$& , & /& -*&  &- ! & &+  %&&4  &@& , *& " $& , ,  *&+ *I channel coding theorem O *&& 2.   -+9  * source coding theorem O *&& 6.9 /& & &-*   &$& &- 6.2 The channel coding theorem /& && *& *&&

, &,&&-& O&& +  *&&

2.9

Associated with each discrete memoryless channel, there is a nonnegative number C (called channel capacity) with the following property. For any å . < and R , C, for large enough n, there exists a code of length n and rate > R (i.e., with at least 2 Rn distinct codewords), and an appropriate decoding algorithm, such that, when the code is used on the given channel, the probability of decoder error is , å /& * " -!-  %!& ! *!%* *& *&&  $% !  "& % $-& "*&& *& *+$*&& - >& "&#&&  *& --! &%*&& *& $ , && " >& *& $*& discrete memoryless channel, a nonnegative number C, for large enough n  there exists a code . . . and . . . decoding algorithm /& *  >&K+ -! ;! - ;&&  *& -% *&&  126

12.

Survey of advanced topics for Part one

· Discrete memoryless channel *& *&& *  >&& $;& , + *& -*& & &*$ *&  $& &@ $& , !-*  -*&  & , "*-* *& $! $*>& AX  *& !$! $*>& AY & >* &H!   J& >& %!$ A  "*&& *& i* -*& !$! Yi  &&  *& i* -*& $! X i >+ *& &H! Yi ˆ X i ‡ Z i  "*&& Z 1  Z 2  . . . ,  &%-  $-& #% ;!&  *& %!$ A *& -*& -$-+ - >& &J& >+   1 @ IOXL Y9  C ˆ !$ O):19 n X n "*&& *& & @ !  #& ;&  n &  ;&- X #% ;!&  A n   Y ˆ OY1  . . .  Yn 9  *&  ;&- * &->& *& -*& !$! , X  *& $! *  Y ˆ X ‡ Z "*&& Z ˆ O Z 1  . . .  Z n 9 - , *& J n - $& , *& & $-&   &+  && * *& & @ !  -*&;& "*& X  !, + >!&  A n    -& IOXL Y9 ˆ HOY9 HOZ9 O&& > 1169 C ˆ % q

, n

1 HO Z 1  Z 2  . . .  Z n 9 n

"*&& q  *& ! >& , && &  A  ! ! * , n

1 1 HO Z 1  . . .  Z n 9 ˆ  HO Z 1  . . .  Z n 9 ˆ H < : n!1 n n

* H!+  -& *& entropy , *& $-& Z 1  Z 2  . . . :

$, , *& -% *&& , * ;& &%- & -*& - >& %;& % *& & , *& $, "& %;& , *&& 2. *& + ,J-!+ & "*& "& +  & * , &-* å . >+    1  P  IOxL y9 C  . å n $$-*& < , n  %& !&+ *&&   *&&  - I asymptotic equipartition property "*-* & * , &-* å . >+    1    P  HOz1  . . .  zn 9 H <  . å n $$-*& < , n  %& -& IOxL y9 ˆ  % q "*  &&&

HOz1  . . .  zn 9 *  ?!

6.2 The channel coding theorem

127

*&& & + *& -&& -*& "* & + , "*-* *& -% *&& * >&& $;&L -! %& F1:G  /,"4 F2:G , &

*& ! -*& , *$& . - >& %&&4&  ," & AX ˆ AY ˆ *& & ! >&  & Yi ˆ X i ‡ Z i  "*&& " Z 1  Z 2  . . .   &;!&  $-&  "*&& *&& +  >&  & # , $! - *&  !--&,! *&&  , * # --! "*& *& Z i I & &$&& O& *& -*&  & +& >!  -&&9  "*& *& Z i I ,  ! $-&L && %& F1:G *$& :  (

 *& -*& "& *;& -!&  , *;& >&& *&  &  *& && *  %& & , ,  K"  & &- +  & , *&  &&% &@& , *& -% *&&  *"&;& *;& >&&  multiterminal -*& *& && !-* &! !&  * * &, --& -*& "* ,&&>-# *  -*&  "*-* *&  & - J ! *& !$! yi -&$%  * $! xi  * O&& F27G $$ 113B12-#   -&& & +& -*& &  -*%&  -$-+ /*  &    #& - !-  & & * -$-+ !-* && O&& *& &%* $$&  $$ 6:6B.6)  F27G   > 22 , f&D x1 < < 1 1

x2 < 1 < 1

y < ? ? 1

O&& NNPII   $&- &!& + >9   ,!$! *&&!$! -*& * -*&  &&  * -$-+ % 6 "&;& , *& " && & $    &$&& ,  &   !&!+ ;& *& -*& *&+ " &,&& "* &-* *& *!   $ H!& D   %;& $ OR1  R2 9 , &   $>& , && 1  J  -& , & R1  , && 2  J  -& , & R2   , *& &-&;&  J  &-% !& !-* * *& - -&-+ &-& >* &%& "*   

12)

Survey of advanced topics for Part one

 & $>>+  &&P *& "& ! !  >& NN&II ,, *& $ OR1  R2 9 &  *& capacity region , *& -*& "*-*  * -&  *"  % )1  & %&& & +& "$! &!$! -*& *& !  &&+ *&  & *&&   -;&@ &%  *& J H! % -& *& -$-+ &% !-* * >+ &>& - !-  >& , + $ , &  *& &%  +  broadcast channel  & "* & $!  & * & !!+ " !$! %   $>&  $;& *& &@&-& ,  -$-+ &% O! /+& F.3G ,  & && --! , *&& !&  -*&9 · A nonnegative number C
;& ,  -&& & +& -*& "& "  *$& 2 * *& --! , -*& -$-+ - >& ;&+ ,J-! !& *& -*& &?+  *%* &%&& , + &+ &- * C  %;& >+ Cˆ

@ IOX L Y 9

X

*& @ ! >&% #& ;&  O& &9 & !-& X  *  ;&  $>& $>>+ >!  *& -*& $! $*>& AX 

* ; $>&   ; >!   &"* && * %*  J $$& -& IOX L Y 9   -;&@ \ ,!- , X   *&-& + - @ !    %> @ !  % * ,-   % $ %& F1:G &;& " &-  *& $>& , J% -*& -$-+ ,    & &-&+   F23G  *! F62G *;& %;& &,J-& ! &- $-&!& , - $!% C  -*& *& *  I *& --! , -$-+  !!+ !-* & ,J-! , *& &J
H O729   %&&  &;& &,,&-;&  -*& "* ;& &%- & *& --! , C  "& *;& &&  &H!;&  *& --! , *& & &$+ , *& & $-& 

Figure 6.1 *& -$-+ &% ,   $&

!$&--& -*&

6.2 The channel coding theorem

12:

#;   &+  - $!& *& &$+ O&& > 12!  %&& ;&+ &  #"  !&  -*& *& -&$% $>&  *  *& --! , *& -$-+ &%   * "% ;!+ !!-*& · For large enough n  -&& & +& -*& *& $>& , && % *& &*$ >&"&& å *& && &-& & $>>+ R *& && &  *&  & n !-* * *&& &@  -& , &%* n * & *& ?> * >&& *& !>?&- ,  & &@& &+ &;& &&-* !%*+ $&#%  * >&& -;&& * , &-* < < R , C *& >& -& , &%* n  & R *  &-& & $>>+ %;& $$@ &+ >+ PE  2

nEO R9



"*&& EOR9 *& -*&I reliability exponent   -;&@ [ ,!- , R * #  &*% #& *& -!;& *"  % )2  >& & $&-& && >+ PE OR n9 *& & $>>+ , *& >& -& , &%* n  & > R  &J& EOr9 ˆ 

n!1

1 % PE OR n9: n

O -!+ >+ * &;& $;& * *   &@ >!  *& ,"% -! "& " %& * &-*-+  >&% !& * !$$& >!  EOr9 & &+ !$$& >!  EOR9 ˆ  !$ n 1 % PE OR n9 &-9 *!%* EOR9   #" &@-+ , + ;   *&& &@ % !$$&  "& >!  EOR9 "*-*  %&& #  *"  % )6 *! *&&   NN-- &II R-  >;& "*-* *& !$$&  "& >!  EOR9 %&& >! , < , R , R- *&&   %$  &@ $& -&   "* " & $>>+ p ˆ :& #" !$$&  "& >!  EOR9 & $&  % ). O& *& %* & !$$% *& "& >!  % ). * -&$  *& R< -% *&&  > 221B22)9

Figure 6.2 *& $$@ & *$& , *& &>+ &@$& ,   

12(

Survey of advanced topics for Part one

Figure 6.3 *& $$@ & *$& , *& #" >!  EOR9

Figure 6.4 *& >& #" >!  EOR9 ,   "* p ˆ :?&- "& *;& -& NN-% *&+II O &;& *& &- $ , * ># 9 - >& ;&"&   &-* ,  -!-;&  $-- -*& -% *&&  "&;& & , *& &@$- -% -*& & " #" - >& !&  -*&;& ;&+ " & $>>&  & ;&+ -&  -*& -$-+ *& -% *&  *& " * >&& "%  -J-& *& NNPE ! < , 

6.2 The channel coding theorem

123

R , CII --! , *& -% *&& , *& #& , -!-;&&   ,  *& *"&;& * --!  -&    -& >&  ! , &,, * >&& &;&  *& $>& , &%  "*&  *&  &  & &%*&% *& NN*&& &@  -&  &-% %* II $ , *& --! & ! &->&  & , *& &,,  *& J $-&   $>&  &- &   &;&+   - , -&   $;& *& -% *&&   &@ $& *& - , linear codes O&& *$& :9 - -& * & %  *& && , *& -% *&& O&& &% %& F1:G &- )29 -& & -& & ;&+ &+  &-& * *" * &-% %*  , % -& &&  >&  $--+ - $&@ *& -% *&& *  >&& $;& , timevarying convolutional codesL && &+ F21G &- )  : &*& , *&& -& , -& *"&;& *  $-- &-% %*  /* &@-+  "& & >+  NN$--II %* P & ! $ *& ;&" "&+ --&$& >+ - $!& -& O&& &% * $-,    F1G *$& 1 R, such that: O9 PE n , 2 nE c O R9 (here PE n is the error probability of the nth code and Ec OR9 is a monotonically decreasing function of R which is positive for all R , C), O>9 The complexity of encoding and decoding Cn is -# % #& *& & *"  % )7 + -*%  & -&  +  #%  $-! # , !& -& O &&B

16
;& &+I &! %&+  *& *& %$ >&"&& ,  *&+ O"*-* *" "*  *! >& $>&  9  -% *&+ O"*-* *" "*   $--  9 ,!&+ "*& &+I *&&  NN$--II     NN-!-;&II  *& && *  &  & ! *"  J *& $$$& & -& *! *& $>& , J%  &@$-+ -!->& $-- &H!&-& , -& , "*-* PE $$-*& < &@$&+ ,  R , C &  $& ,  && &;& $!-& !-*  *&&   " !& *& !>?&- , ,  *&+  -% *&+ O $>>+ $! >* ! , >!&9 · Converses to the coding theorem + -;&&  *& -% *&& *;& >&& -J&  NN"&# -;&&II  NN% -;&&II >! *  &-!&  &% -& *& % -;&&    $+ *& "&# & /& *;& &+ && O$ 1179 *& "&# -;&&  + * , "& & $  - !-& *& !$! , +  >+ + &- !-& ;&  -*&   & &@-&&% *& -*&I -$-+ *& &!% > & $>>+  >!& "+ , 9 *&  k=n &!& *& & ,  L -&+ , k=n &@-&& C *& ROä9 , 1 "*-* ,-& ä *& > & $>>+ "+ , 1

2

nE A O R9



161

R . C

"*&& EA OR9  Arimoto's error exponent "*-* * *& %&& *$& *"  % ))  *& $&- -& ,   "* " & $>>+ p ˆ &"&& -*& -%  !-& -% * "& *;& & $*4& *& !>?&- , !-& -%  rate-distortion theory  >+  "& % & ,J-! &-& *& !>?&-  -&>+ & ;-&  *  &

Figure 6.6 *& %&& *$& ,  I &@$& EA OR9

Figure 6.7 EA OR9 ,   "* p ˆ &K+ -! converses  *& !-& -% *&&  · Discrete memoryless source *&  ! "+  &@& *& !-& -% *&&   $;&  ,  "& - , !-& * *& -&& & +& & *& *&& *  ,- >&& &@&&  * "+   ; ! >& , !-& && $--+ + !-& "*-* - >& &&   +  $-& . . .  x 1  x<  x1  x2  . . . * >&& !& ,! &+ *"&;& *& & ,!- ROä9 - >& --!& &,,&-;&+ , &;&+ ,&" , *&& !-& & O&& &%& F16G   $ 6.9 /& "  &- )2 * H!&    #" >! *& -$-+ &% , -& !&  -*& !-* &  #" >! !-& -% , !$& !& >! & &! *& &$B/, *&&  && -% , -&& !-&  "* &%  &->& *& &$B/, *&&  "& ! $!&  $ !   $  $- , *& !-& -% *&& O "*-* -&+ *$& 11  &&+ &;&M9 & X 1  X 2  . . . >&  &$&& &-+ >!& &H!&-& ,  ;>& #% ;!&   & A *# , * &H!&-&   ,  !-& &;&  *&  %  ,!- *  #& *& & $*>&  A   &J& dOx y9 ˆ 1 , x 6ˆ y ˆ < , x ˆ y *&  "& "  $ (& &&+ $&,&-+ &$&&& !% + >! HOX 9 > $&  $& * !-*  , -!&   $& -&H!&-& , *& !-& -% *&&    *$& 11 "& " %;& &@$- &-*H!& , % * noiseless coding    -& " & ! !  *& &! , &$  /, & OX i  Yi 9 i ˆ 1 2 6 . . .  >&  $&& &-+ >!& &H!&-& , $ , -&&  ;>& "* -  >! ,!- pOx y9 - 

6.3 The source coding theorem

166

 &$& HOX 9 HOX  Y 9 HOX jY 9 &- *& $>&   J &$&& !-& -& , *& X  Y &H!&-& !-* * *& &!%   &%%>& $&-J-+ "& "  -J%! !-*  *& & &$-&  % )( *& ;&- X  &-&  *& -&" f OX9 "*-* - ! & + & , M X ;!&  + gOY9 - ! & + & , M Y ;!& *& rates & &J&  RX ˆ

1 % M X  k

RY ˆ

1 % M Y  k



 *& error probability  ^ 6ˆ X  Y ^ˆ PE ˆ PfX 6 Yg:

$ , & ORX  RY 9    >& admissible ,, , &;&+ å . < *&& &@  -J%! , *&  &$-&  % )( "* PE , å &$  /, O&& F27G $$ .7&  & *& $>& "! &!-&  *& &,&& &-% ,  &H!&-& , &$&&  ;&- "* &$+ HOX  Y 9   *&  >& &% "! >& *& *,$& >!& >+ *& & RX ‡ RY ˆ HOX  Y 9 , *& &-& & *& -&-& *& 4& , *&  >& &% - -&&   *& B/  >& &% ! & &&+ >;& * &  & * , , &@ $& Y  #" $&,&-+  *& &-& *& & % !-&+ >! X  HOX jY 9   &;&+ $  *&  >& &% ! ,+ RX > HOX jY 9   + RY > HOY jX 9 *! *&  >& &% ! >&  !>& , *& &% -&  % )3 *& & #>& ,-  *   * %& & &-&+ /+&  =; F7& $>&

Figure 6.8 -# %

, *& &$B/, *&& 

16.

Survey of advanced topics for Part one

Figure 6.9 *& & ,  >& &

$&, -& , *& -J%!  % )( "*& *& & RX  RY &  *& &% #& NN/B= &%II  % )3  /B= &% X - >& &$!-& &@-+ -& RX > HO Z9 ! -& RY , HOY jX 9 &;>+ *&& " >&   *& &-&$ , Y  /+&  =; *;& && & *&  ! $>& 

*& &$B/,  /+&B=; &! ! >&  &&$ !!& %&&4 , & *&+ · A functon ROä9 /& *;& &+ &&  *$& 6 * &;& ,  -&& & +& !-& !&  ;&+ %&  ! , + &+  $&&   H!& ,J-!  - $!& *& & ,!- ROä9 "&;& *! F62G O&&  '&>   ! F2)G $$&@ : 9 * -;&&  $+ -;&%& %* , - $!% ROä9 ! &-+   *& $>& , - $!% ROä9  * -&  "&+ &%&  ;&  & %&& !-&   &!& *"&;& *& $>& , - $!% ROä9  !-* & ,J-! &*$ *&   $&;& %&& - , !-& , "*-*  ->& , ! , ROä9  #"  *& - , disccrete-time stationary Gaussian !-& &;&  *& &H!& & -& && *& !-&  &&   + ! &H!&-& ,  ;>& . . .  X 1  X <  X 1  X 2  . . . : *& &  ,!- ROä9  %;& $ &-+ >+  1 ð äOè9 ˆ Oè SOù99 dù 2ð ð    1 ð SOù9 ROè9 ˆ @ +
H O7.99  &@ $& *&&  $$&+  #" -&& !-& "* & + , "*-* ROä9 - >& - $!& &@-+ "&;& *&& &@ + &-*H!& , >!% ROä9  "& % &,& *& &&& &&  &%& F16G · For large enough k /& "  &- 2 * ,    *& -*& & $>>+ PE - >& &  $$-* < &@$&+ "* -&% >-# &%* *& %! H!& , *& !-& -% *&&  " H!-#+ & *&  ,  $ ! !-& -& , &%* k  & < ROä9 $$-* äP  * &- - F.6G O&&  %& F1:G &- 369 * *" ,    * , d  Ok9 && *&  ! $>& ;&%&  ,  !-& -& , &%* k  & < ROä9 *&   % k d  Ok9 ˆ ä ‡ < : k

*! *& -;&%&-& , ä  Ok9  ä  !-* & $ * *& -;&%&-& , PE  &  ! , &&-* * >&& &;&  *& $>& , J%  - , &+  $& &>& !-& -& , "*-* *& !-& -% *&& - >& $;& * &&-* * &  !-& -% *&&  , *& , NN + å . <  ä > ä  *&& &@  !-& -& , +$& T "* & < ROä9 ‡ å  ;&%&  < ä ‡ åII !-* *&&  *;& >&& $;& "*&& +$& T  *& & , linear codes O&& *$& :9  tree codes O&& *$& 1& - $&&+ &->& >+ + & , k &+ &$&& -&" x1  x2  . . .  x k  , k &;&+ -&"  & , *& q k & - > iˆ1 á i x i  á i 2 Fq  , "& %& *& -&"   k 3 n @ G G  -&  generator matrix , C & %&&+D De®nition Let C be an On k9 linear code over Fq. A matrix G whose rowspace equals C is called a generator matrix for C. Conversely, if G is a matrix with entries from Fq , its rowspace is called the code generated by G. + $&-,+%  %&& @   &+ $>&  %;& - $- &-$ ,  & &&% -& *& *&& &@ $& * ," O "* q ˆ 2L *&+ & binary -&9 " >& !& , !;& $!$& *!%*! * -*$& Example 7.1 O7 19 & -& C1 "* %&&

@

G1 ˆ F1 1 1 1 1 1G: Example 7.2 O7 69 & -& C2 "* %&& @

 1 1 1 < < G 2 ˆ  < < 1 1 < : 1 1 1 1 1 Example 7.3 O: .9 & -& C6 "* %&& @

 1 < < < < 1 1 < 1 < < 1 < 1  G6 ˆ   < < 1 < 1 1 < : < < < 1 1 1 1

h

h

h

OC1 - + *& " -&" ! G2   *& !H!& 
%&& @ , C2   ,-

 1 1 < < 1 G92 ˆ  < < 1 < 1  < < < 1 1 "*&& , -+ "& *;& --& *& &,  1  &-* " % G92 *& * G2  *& &-% O:19 , C2 >&- &D Ou1  u2  u6 9 ! O u1  u1  u2  u6  u1 ‡ u2 ‡ u6 9:

* &-% , C2 * *& &>& ,&!& * *& ,  + > u1  u2  u6 $$&  *& -&  *& -&"L  %&& *& + > u i " $$&  *& t i * - $& , *& -&" x ˆ uG , *& &,  &+ , *& i* " , G --!  -!  t i  + -& &   "*-* * *&

1.2

Linear codes

$$&+ * *&& &@  &-% !& !-* * *& ,  + > $$&  *& -&    >& systematic /& *;& *! & & * every linear code is systematic -& &@ * *& 
%&& -& , C1  C6 *;& *& , G ˆ FI k AG "*&& I k  *& k 3 k &+ @ &+  &;&+ & -& &?+ * $$&+ O&% C2 9L >! , *& -&  , !&   & +& -*& *& $&, %  -!  $& !  G "  & *& -&I $&, -&  "& - "+ ! & G ˆ FI k AG  * -&  &@ $& >+ &&% *& -!  , G92  *& & O16.279 "& >

 1 < < 1 1 G 20 ˆ  < 1 < < 1  < < 1 < 1 "*-* %&&&  -& "*& $&, -&  *&  &  * , C2  + & +& -*&  !-* ,  "*& , *& %&& @ *&&  *& $&*$ &;& & !&,! @ -& "* &;&+ & -& -& *& paritycheck matrix "*-* "& " &->& , C   On k9 & -& ;& Fq  parity check ) , C   &H! , *& , a1 x1 ‡ a2 x2 ‡


‡ a n x n ˆ $-& , V n OFq 9   && >+ C ? OC NN$&$II9  -& *& dual code , C +   &!  & %&> C ? *  & n  OC9 *  C ? is an On n k9 linear code over Fq $+-*&-# @ , C  " &J&   %&& @ , C ? & &-+D De®nition Let C be an On k9 linear code over Fq. A matrix H with the property that Hx T ˆ 0 iff x 2 C is called a parity-check matrix for C.: , -!& >+ *& %! & >;&  ," * &;&+ On k9 & -& *  !H!& On k9 3 n 
$+-*&-# @ !   & -   !&  %*+ ,,&& -- , , H  &@ $& , G ˆ FI k AG  ," * H - >& #&  H ˆ F AT I n

k G:

O::69

, G   , * ,  H - >& >& >+ J $&, %  -! 

7.2 Syndrome decoding on q-ary symmetric channels

1.6

$& !  $! G  *& , FI k AG  *& $&, % *& ;&& $& !  F A T I n k G(  &@ $& *& $+-*&-# -& , &  * "+ , G1  G2   G6 O$ 16.9 &

 1 1 < < <   1 < 1 < & & &- O>! -, > :11  :2! *& + &  *  &$& +  *& & $& z    *&  & -&"D s ˆ Hy T ˆ HOx ‡ z9 T ˆ Hx T ‡ Hz T ˆ Hz T  -& Hx T ˆ 0 , -!& *& &-&;&   &-+ &&&  zL *& "  #" x ! -& *& #" y  x ˆ y z *& - ,&+ ,-!  *& $>& , J% z

*& + & $;&  & ,  >! z >!  &!%* *  >&-!& ,  J@& s 2 V n k OFq 9 *& & , !  Hz T ˆ s ,   coset 1< , *& -& C *   !>& , V n OFq 9 , *& , C ‡ z< ˆ fx ‡ z< D x 2 Cg:

O::.9

*&& & q n k -& , C -&$%  *& q n k $>& + & sL &-* -& - &@-+ q k && & *! -& *& &-&;& - $!& s *& * &!-& * &-* , z , q n  q k $>&  &+ *& && & , *& -& -&$%  s  &  %!* >&"&& *&& q k -& , z *"&;&   &-&+  #"  &*% & >! *& -*&  * &- "& * ! & * *& -*&   q-ary symmetric channel OH9 *  , X    ;&- &% *& -*& $!  Y    ;&- &% *& -*& !$! *& Y ˆ X ‡ Z "*&& Z ˆ O Z 1  Z 2  . . .  Z n 9    ;&- "*& - $& & &$&& &-+ >!&  ;>& "* -  >! Pf Z ˆ & *& ! >& , 4& - $& , z &;&+ w H Oz9  *& ! >& , & --!%  z , å < 1=q *& %* & , O:)9   &-&% ,!- , w H Oz9   *&  $>>& z  *& & ,  & "&%* O *& -& 1=q , å , 1=Oq 19 && > :69 %!& :1 *" *" *& + & &-& "#  &  $-$& , -!& &$ 2  * %* &$&&  , >&  ! , "#L && *&  >?&- , *$& (  3  *& &% ,  & & -& , *& H , "*-* &$ 2  ->& "&;& , k  n k & >* &;&+     $>&   $& & &$ 2 ;  NN>& #!$II $-&!& "*-* "& " &->& & % -& C2   $+-*&-# @  %;&  $ 1.6    1 1 < < < Hˆ : 1 < 1 1 1  *  $& -& *&& & + ,! $>& + &D &"&& " ;&- x  y  d  Ox y9 ˆ *& ! >& , - $& , "*-* xi 6ˆ yi ˆ w Oy

x9Ocf: $: 1.79:

O * -& J& *& ,  $$&& &H!& ,  &-L && > :.9 *&&   &&% &*$ >&"&& *&  % %& &+ ,  -&   >+  -&- &   H "*-* "& " ;&%&

*! & C ˆ fx1  x2  . . .  x M g >&  -& , &%* n  &-&+ & , !&   H !$$& * "& " C  >& -$>& , -&-%  & $& ,  % "&%* < eL *  , x i  & y ˆ x i ‡ z  &-&;&  w Oz9 < e "& " ! &-&I !$!  >& x^ ˆ x i    &+  && * , &-* -&"  & "* $>>+ 1=M *& *& &-&;&I >& &%+ , %!&% "*-* -&" " &   $-# *& -&" -&  y *  *& & , "*-* d  Ox i  y9   & O&& > 2169 FNoteD -& d  Ox i  y9 ˆ w Oz9  ," * *& + & &-% , & -& &->&  % :1  &H!;&  NNJ *& -& -&"II &-%G   -& * , * %& &- &-% &%+  !& *& -& " >& -$>& , -&-%  $& , "&%* < e ,, *& -& >&"&& &-* $ , -&"  > 2e ‡ 1  O% :69 , d  Ox i  x j 9 > 2e ‡ 1 *  , *&  % $*&& , ! e ! x i  x j & ? *& , x i  &  d  Ox i  y9 < e y - >& -& 

7.3 Hamming geometry and code performance

Figure 7.3 

1.:

% $*&& , ! e ! ?-& -&"

x j *    x i     %& &- &-& "  $&,& x j  x i  ;&&+ , d  Ox i  x j 9 < 2e *  , *&  % $*&& , ! e &&- O% :6>9 *&   -& * , x i  & *&& &@  y * * d  Ox i  y9 < e >!   &  -&  x j     x i   "& & &  &J& *& minimum distance , *& -& C  d

 OC9

ˆ

fd  Ox x99D x x9 2 C x 6ˆ x9g

 "& *;& $;& *& ,"% *&&  Theorem 7.2 A code C ˆ fx1  x2  . . .  x M g is capable of correcting all error patterns of weight < e iff d  OC9 > 2e ‡ 1  &@ $&  -& "* d  ˆ : - -&-  & $& , "&%* < 6L , d  ˆ 22  $& , "&%* < 1&;  * -& d  Ox x99 ˆ w Ox x99  x x9 ! >&  O4&9 -&" , C  & O x 6ˆ x99 *&  ! -& ,  & -&  *&  &   minimum weight w  OC9 "*&& w

 OC9

ˆ

fw Ox9D x 2 C x 6ˆ & &$&& >! *& ! ,  -!   0L   d  OC1 9 ˆ 7  H 2   -& * d  OC2 9 ˆ 2 -& , &@ $& *& *  ,!* -!  , H 2 & &- *& !+ , H 6    $ "& * &;& ! &@ &-   7.4 Hamming codes  &,&&-& "& % $+ *& $+-*&-# @ , C6 D

 < 1 1 1 1 < < H 6 ˆ  1 < 1 1 < 1 < : 1 1 < 1 < < 1 /& " $$+ *&& :6  *& $>& , && % d   + *& & ,"% *& & & , *& *&&  d   *&  & ! >& , -!  , H "*-* !  0 &+ d  6ˆ 1 2 -& *& -!  , H 6 & 4&  - "&;& *&& & + !>& , *&& -!  , H 6 ! %  0 , &@ $& -!  1 2  6 *! d  ˆ 6   C6   singleerror-correcting -& *    -$>& , -&-%  & $& , "&%* <  1 + >&;& * , C  + On n 69 %&&-&-% -& *& n < : -& , *& 6 3 n $+-*&-# @ H * n > (  "! &*& *;&  4& -!  Od  ˆ 19   $ , &- -!  Od  ˆ 29

*& %&& &J ,  >+  % -& ,"

7.5 Syndrome decoding on general q-ary channels

1.3

De®nition Let H be an m 3 O2 m 19 binary matrix such that the columns of H are the 2 m 1 nonzero vectors from V m OF2 9 in some order. Then the n ˆ 2 m 1, k ˆ 2 m 1 m linear code over F2 whose parity-check matrix is H is called a (binary) Hamming code of length 2 m 111 /& #& " & # >!  % -&  -& * + & &-%  &$&-+ &+ , *& & $&  z ˆ ! , w Oz9 ˆ 1 + z i ˆ 1 *& s ˆ c i  *& i* -!  , H &-& *& + & &-+ &J& *& & -  *& %&& &-% %* , % :1 >&- & *& $&- %* , % :. &- &- *  -& C - -&-  $& , "&%* < 1 ,, *&  % $*&& , ! 1 ! *& -&" & ? !   % $*&& , ! 1  V n OF2 9 - n ‡ 1 ;&-  *  %&&-&-% -& - *;&   2 n =On ‡ 19 -&"  $-! , n ˆ 2 m 1 m m *&& - >&   22 1 =2 m ˆ 22 1 m -&" *& &@- ! >&  *&  % -&M *! *&  % -& *;& *& >&!,! %& &$$&+ * *& $*&& , ! 1 ! *& -&" &@-+ J V n OF2 9 "*! ;&$ * & * *&  % -& >&%   &@& &+ &@-!;& - , -& *& perfect codes *& + *& >+ & $&,&- -& & *& &$& -& O&& > :1(9  *& O26 129 + -& O&& &- 3(9 O  && *& -! , $&,&- -&  *$& 129 &;& ; ,  % -& -!% >+  % -& & %;&  > :1:  :13 "&;& *& & &&%  !&,! %&&4 ,  % -&    - , -& "*-* -&- e & , e . 1L * %&&4  *&  $- &&  *$& 3 7.5 Syndrome decoding on general q-ary channels  &- :2 "& &->&  % &-% %* , & -& >&% !&   q+ + &- -*& "&;& ,&" q$! q!$! -*& & "& && >+ H    * &- "& * >&K+ -& *& $>& , &-% & -&  & - $-& -*&

Figure 7.4 + & &-% ,  

% -&

17
&  On k9 & -& ;& Fq "*-*  %%  >& !&   -*& "*& $!  !$! $*>& & >* &H!  Fq  /& ! & * *& -*& &  ;& *  , x ˆ Ox1  . . .  x n 9  *&  & -&" *& &-&;& ;&- y  %;& >+ y ˆ x ‡ Z "*&& Z ˆ O Z 1  . . .  Z n 9    & ;&- "* >! %;& >+ PfZ ˆ zg ˆ pOz9 z 2 V n OFq 9 O& * *& H , &- :2 J * &-$ "* pOz9 %;& >+
H O:799 "  "& &-& C  * -*&P  >&,& *& J &$   - $!& *& + & s ˆ Hy T L * " &,+ *& -& * *& & $& z >&%  ! " *&  #&+ & $& "* *& -& -*-&4& >+ *& + & s  *& & "* *& %& ;!& pOz9 *& * *&  & "&%* &-& *& &-% %*  * %&& ! O-, % :19  %;&  % :7  &@ $& -& !% -& C2   -*& , "*-* *& &%* 7 & $& *;& $>>& %;& >+ *& ,"% >&D z !  " &&- + & $& z !& z   4& -&" *& &-&I & $>>+  *& ?! *& $>>+ * z   4& -&" , C , "& ! & *& -*&   H *& *& $>>+ * z &H!  $-! -&" , "&%* w  FO1 Oq 19åG n w å w O&&
H O:)99 &-& , "& && >+ A i *& ! >& , -&"  C , "&%* i *& & $>>+ , * &&-+ -*& & 

7.6 Weight enumerators and the MacWilliams identities PE ˆ

n


A i å i F1

19åG n

Oq

176

i

iˆ1

ˆ F1

Oq

19åG n FAOä9

1G

O:::9

"*&& ä ˆ å=F1 Oq 19åG  AOz9 ˆ A< ‡ A1 z ‡


‡ A n z n O& * A< ˆ 1 , + -&9 *& %&&% ,!- AOz9  -& *& weight enumerator , C /& * !+  ,!*&  *& &@ &- h 7.6 Weight enumerators and the MacWilliams identities

 &$&D , C   On k9 & -&  "&%* &! &  *& $+  AOz9 ˆ A< ‡ A1 z ‡


‡ A n z n  "*&& A i && *& ! >& , -&"  C ,  % "&%* i k &+ A< ˆ 1  AO19 ˆ q   "& *;& && O
H O::99 AOz9 - >& !&  --!& *& & $>>+ "*& C  !& , & &&- +

*& ,"% *&& *" * AOz9 - >& !&  >! *& & $>>+   !-* & &&% !  &+ "*& *& -&  >&% !&   -&& & +& -*& "*  @ ! #&* &-% !& F& + & &-%  @ ! #&*L && $ 1.)G   $-+ "& * & + >+ -& >! && > :1& " !$$& x<   & *& &-& " &J&+ 

17.

Linear codes

!$! x i , pOyjx< 9 . pOyjx i 9   , Y i ˆ fyD pOyjx i 9 > pOyjx< 9g  ," * PO Qi
>+  < AOã9 1  $ & , x<   & ,  & *& -&" + x j   & &% &-  * ?!

7.6 Weight enumerators and the MacWilliams identities 177  n %;& *" * *& &!% & $>>+ POEj9 < iˆ1 AOi j9 ã i "*&& O j9 A i ˆ *& ! >& , -&" 6ˆ x j "*  % -& i , x j  !  -& *& -&  & AOi j9 ˆ AO :129  *& *&& $;& h Example 7.6 & C1 , * -*$& * + " -&" & ;   $& & ,  , *& "&%* &! & , *& ! -& C ? O&& $ 1.29 Theorem 7.6 Othe MacWilliams identities9 Let AOz9 be the weight enumerator of an On k9 linear code C, and let BOz9 be the weight enumerator of the dual code C ? , that is, BOz9 ˆ

n


B j z j

jˆ
 :2:9  "& &>*  & -;& , x ˆ Ox1  . . .  x m 9   ;&- , + &%* &;& m ˆ 1 ;&  J& F wOx9 &&   % "&%* *  *& ! >& , 4& - $&  $-! , x   - O& x 2 F9 *&  < , x ˆ + &J *& ! >& , - $& xi , x &H!  1 h

7.6 Weight enumerators and the MacWilliams identities

17:

/& " $-&&  $;& *&& :) *& &   - $!& *& !

z wOy9 hy x x9i O::1:9 xx92C y2V

 " "+ , "& --!& *& & ! &  2


O1

z9 wOx

x99

O1 ‡ z9 n

wOx x99

ˆ

 O:1:9 J "& > >+ n


A i O1

z9 i O1 ‡ z9 n i 

iˆ
+ "&%* %* && *$&& ! *& ! -& * + 2 m -&"  *  ;&+  $& "&%*

17(

Linear codes

&! &   >+ *&& :) AOz9 , *&  % -& - >& - $!&  %&& O&& *& ,"% &@ $&   > :23  :6+ *&& :) $ % : 1 z9. O1 ‡ z96 ˆ 1 ‡ :z 6 ‡ :z . ‡ z : : h ( O1 ‡ z9 ‡ :O1

Problems 7.1 & C >&  & -& ;& *& J& F6 ˆ f+ & -& C O9   "&!-& &-*& %&& @ , C O>9   $+-*&-# @ H , C O-9   ! "&%* -& && , &-* , *& ( -& , C O9 & A i Oi ˆ & , ;&- , "&%* i  C  *& A i  O&9 &-& *& ,"% &-&;& ;&-D 111& & "* d  OC9 ˆ 2   !H!&P  $&& *& $, , *&& :. & C >&  q+ & -& >&% !&   H "* @ ! #& * &-% *" * *& &!% & $>>+  >!& >+p AOã9 1 "*&& A  *& -&I "&%* &! &  㠈 2 åF1 Oq 19åG ‡ åOq 29 O *& &J ,  H &&
H O:799

q+ erasure channel   -&& & +& -*& "* $! $*>& A X ˆ Fq  *& J& J& "* q && &  A Y ˆ Fq [ f?g "*&& NNPII   $&- &!& + > , x ˆ Ox1  . . .  x n 9   &  y ˆ O y1  . . .  y n 9  &-&;&  error    --!  *& i* $ , yi 2 Fq >! yi 6ˆ xi   erasure , yi ˆ ? O9 & C >&  On k9 & -& ;& Fq "*  ! -& d   *" * C - -&-  - > , e &  f &!& ,, d  > 2e ‡ f ‡ 1

1)
9 !$$& *& -*&   pure erasure channel *  pO yjx9 ˆ < !& y ˆ x  y ˆ ?  * -& *" * *& - Hx T ˆ 0 , x  >&  -&" - >& !&  > n k & &H!  *& !#" && -&" - $& O-9 $$+ *& &-*H!& !%%&&  $ O>9  &-& *& ,"% -& " , *& O: .9  % -& O$+-*&-# @ H 6  $ 1.)9 "*-* *;& !,,&& &!& >!  &D 19 *" * I   ,  & , C ,, -!  i1  . . .  i k , *& %&& @ & &+ &$&& O-9 " + ,  &  -& C1  C2  C6 , * -*$& *;&P O9 & C >&  binary & -& !-* * &;&+ k&& & !>& , f1 2 . . .  ng   ,  & *" * k ˆ + -& "* * $$&+ O&% *& &&B  -& , &- 3)9G O!9 *&  , ,  & !-&  > :16 - >& !&  &%  &-% %* , & -& * ,,& #&+ , + & &-%    & & -& error trapping  decoding with multipliers12 !$$& * C   On k9 & -&  * "& "*  -&-  $& , e  ,&"& & "* C & I 1  I 2  . . .  I r >& ,   & "* *&  $$&+ * , + e&& & !>& J  f1 2 . . .  ng *&& &@  & & & I i !-* * I i \ J ˆ ö O9 !$$& * y  *& &-&;& -&"  *  - < e &  &-* i ˆ 1 2 . . .  r & x i ˆ *& !H!& -&" , C * %&& "* y  *& $ $&-J& >+ I i *" *  & & , *& x i I  %&&&  *& -!  & -&" O>9 ! % * d  OC9 > 2e ‡ 1 *" * >+ - $% *& x i I  y & >+ & *&  & -&" - >& !H!&+ &J& O-9 $$+ * &-*H!&  *& O: .9  % -& /*  *&  & ! >& , ,  & * " !,J-&  -&- 1 &P !$$& +! "&& $$-*& >+  - !- &%&& "*  +! * * O>+9 -*& --&$ " , &%* n  * *& + # , & $& &;& >&;&  & , *& n ‡ 1 $& O&  %&&-&-% -& ,,  -!  , H   9 *" * ,  J@& ;!& , m , H * *& $$&&  $ O9 *&

1)2

Linear codes n < Oq m 19=Oq 19  * * >!  *$ , n ˆ Oq m 19=Oq 19 *& &!% -&  -&  q-ary Hamming code &+  * n ˆ Oq m 19=Oq 19 k ˆ n m O-9 !- $+-*&-# -& , *& ,"%  % -&D q 6 6 7

n . 16 )

k 2 1< .

7.20 *& >?&- , * $>&   !+ *& $&, -& ,  >+ & -&   ! -*& O&& &- .19 "* &&%+ - EOX 2 9 < â  & ;-& ó 2   $-! "& * >  %!& , *&& :7 , * -*&  & C >&  On k9 >+ & -&L "& ,+  , !&  *&! -*& >+ $$%  &-* NN!  *& &*$ >&"&& d   k , *& >& -&   $-+ "& * &- ! &  binary linear codes  J@& n  d & M L On d9 && *& @ ! $>& ! >& , -&"   & -& "* &%* n   ! -& > d

Problems

1)6

7.21 O *& >&B'* ; >!9 *" * M L On d9 > 2 n =‰1 ‡ …1n † ‡


‡ … dn 1 †Š F HintD , *& >& & -& * ,&"& * * + "  & & ,  -& "! *;&  " , "&%* > d ! *& ! ,  & -& "* & ,  -&    & -&G 7.22 O *&  % >!9 *" * ,  >+ -& , &%* $ n &   % -$>& , -&-% e &  -   2 n = 1 ‡ …1n † ‡


‡ … en † -&" 7.23 O!9 *" * *&  % -& -*&;& *& >! , > :22 7.24 O *& # >!9 & ð i D Ox1  . . .  x n 9 ! xi >& *& $$% * $?&-  >+ ;&-   i* - $& *" * , x %& *!%*  On k9 -& C ð i Ox9 ˆ < * &*& 2 k  2 k 1 ! & * ,-  $;& *
w Ox9 < n
2 k 1 : x2C

-!& * d  OC9 < n
2 k 1 =O2 k 19 7.25 O!9 !$$& n > 2d 2  & C< >& *& -& >& >+ &&-%  -&" , C * & <  *& J n 2d ‡ 2 -&  &&% *&& -& $$+% *& &! , > :2.  C<  --!& * M L On d9 < 2 n

2d‡2

d:

7.26 *& ,"% &! O&  ' & &&>&%9 *" *"  &%*& *&& :7 ,   O9 *" * >! O:169 - >&  $;&  Qi < ã d  Ox< x i 9‡1 , d H Ox<  x i 9   F Hint:  > >+ ,  &$& -& , &%* 2n  *&  &  , & , &%* 2n 1G O>9 &-& *" * *&& :7 - >&  $;&  PE < 12FO1 ‡ ã9 AOã9 ‡ O1 ã9AO ã9G 1 O-9 $$+ *& >! >&  $ O>9  -& C1 , * -*$&  - $& *& &!  * >& 
@ $& :)

*& &@ " $>&  & "* %&&4 , *&& :) *& -/  && *& J &! &&+ !$$& *& & , *& $, , *&& :) "*& q 6ˆ 2 *& &- & "* & -& 7.27 ;& *&& :) , %&& q F Hint: *& $, %;&  *& &@ " "# -& *& H!+ hx yi O&&
H O:1)99  !>+ %&&4&   * & hx yi ˆ ëOx
y9 "*&& ë  + ; *  $* , *& ;& %!$ , Fq  *& - $&@ p*  , !+ "*&& q ˆ p j  p  $ &G 7.28 ;& *& & -/  &&D , C ˆ fx1  . . .  x M g   !>& , V n OFq 9 &J& A i ˆ M 1
*& ! >& , $ Ox x99 , C "* d  Ox x99 ˆ i , *& B j I & &J& >+ *& , ! n 1
A i O1 M iˆ
+  % -& , &%* n ˆ 2 m 1 * "&%* 2 m 1  O>9 &-& *" * ,  >+  % -&

AOz9 ˆ

1 $ O1 ‡ z9 n ‡ nO1 2m

z 2 9O n

19=2

% O1 ‡ z9 :

O-9 &&4&  >+  % -& O&& > :139 7.30  * $>& "& " &J&  ;&%& *&  $ Reed±Muller codes & POm d9 && *& & ,  $+  , &%&& < d  m ;>& ;& F2  & Ov<  v1  . . .  v M 1 9 M ˆ 2 m  &&   ,  *& 2 m >+ ;&- Ox1  x2  . . .  x m 9   & & *& , &-* f 2 POm d9 "& %&  >+ ;&- , &%* 2 m ; *& $$% f ! O f Ov< 9 f Ov1 9 . . .  f Ov M 1 99 *& & ,  ;&- >&  * "+ , $+   POm d9  -& *& d* order Reed±Muller code of length 2 m    Om d9 , * O9 *" *  Om d9   >+ On k9 & -& "* n ˆ 2 m  k ˆ 1 ‡ …1m † ‡


‡ … dm † O>9 *" * *&  ! -& ,  Om d9  2 m d  F Hint:  *" * d  > 2 m d  & f ˆ f Ox1  x2  . . .  x m 9 2 POm d9  -& f O :1:> O&9  $!& *& "&%* &! & ,  Om d9 , d ˆ &  On k9 & -& "* $+-*&-# @ , *& , H ˆ FBI n k G "*&& B  On k9 3 k  I n k   On k9 3 On k9 &+ @ , 1 < t < k *& -& C t "*& $+ -*&-# @  H t ˆ FB t I n k G "*&& B t  *& On k9 3 Ok t9 @ >& >+ &&% *& J t -!  , B  -& a shortened version of C O&&  > :279 O9 *" * C t - ,  -&" , C "*& J t -& & &H!  ;& - >& *!%* ,  2 >! , &%* . " & H >&  m 3 n $+-*&-# @ ,  On k9 >+ & -& C *& b-fold interleaving , C  *& Onb kb9 -& C Ob9 "* mb 3 nb $+ -*&-# @

H

Ob9

  ˆ 

Notes c1

c1

c2 ..

.

cn ..

c1



.

.. c2

.

1)7

    cn

"*&& Fc1  . . .  c n G & *& -!  , H  ># && ! , &%* b  &&-  $& , f >! , &%* b O-9 !- $+-*&-# -& , O9  O21 129 -& * -&-  %& >! , &%* 6L O9  O27 2! , &%* 7 ;&  &,J-& &-% %* , >* -&

Notes 1 O$ 1639 /& $!& *&&  > &!  *L ,  *& !>?&- , &@$- -& *  "* * -% *&&  $ & *& * >&& & 2 O$ 1639 & $&-&+ & block -&  %!* *& , *& & convolutional -& , *$& 1&  $ & $"& "& - -& $! $*>& "* ) 1 !-*  - >& ,! , &@ $&  #*,,  -& F6G *$& : 7 O$ 1.19 !+  
@  $& &  *;&  & " * & & , "  *& @ &  -*%&  *&   
,  /& * ! & *"&;& * , 4& " $$&  *& &!- *&+ & &&& ) O$ 1.29 *&  & - & , *& ,- *  *& >+ -&  $+-*&-# + * &-* -&" *  &;& ! >& , 1I   -& J@& !>& , *& -& $ : O$ 1.29 &&  &&"*&& T  *& $& $& ( O$ 1.69 *&  $-+ , *  O, G  H9  , -!&  --&L *& 
,  &%&   *  &@ $&  ;& *& +& , & * %&&! &H! &$&&& >+ G2 O$ 1.&  % -& -& *&+ &   $& 12 O$ 1)+ &-% ! &  & -& O*& * >+ !!-!& -&9 "& -! *$&  J  & % -& "*-* & &>+ &+   $& &    !& * O; + & &-% , &@ $&9  NN II & -& + "*  &  &!-+   2&  $& &&  *"& "*! !-* ,J-!+ "&;&  &  > *& $&, -& $ & >+ *I *&&    &-&+  !& %& -&   %&&  %& -& &;& ,   & " >& ,J-!   $& &  * &    >-# -& !&  $--& &  ,- cyclic codesL -+-- -& ,  ;&+    *%*+ !-!& !>& , *& & , & -&  * -*$& "& " %;&  %&& !-  -+-- -& -!% >* *& !&+% *& - *&+ O&- (19  *& >- *"& --! !&   $& & -+-- -& O&- (29  &- (6 "& " *" *  % -& - >&  $& &&  -+--&   &- (.  (7 "& " && *" -+-- -& & !&  - > burst errors ! + " >& -!&  *$& 3 "*&& "& " !+ *&   $ , + , -+-- -& +& -;&&D *& E &&B  , + /& >&% ! !& "* *& -!!$$&% &J , *& - , -+-- -& De®nition An On k9 linear code over a ®eld F is said to be a cyclic code if, for every codeword C ˆ OC<  C1  . . .  C n 1 9 the right cyclic shift of C viz. C R ˆ OC n 1  C<  . . .  C n 2 9 is also a codeword.

 "& * && *&& &

+ -+-- -& >! - $&  & -& 1):

1)(

Cyclic codes

*&+ & H!& --&  &@ $& *&& & 11(11 & O: 69 -& ;& GFO29 >! + " & -+--M Example 8.1 , F  + J&  n   &%& > 6 *&& & "+  & ,! -+-- -& , &%* n ;& F !!+ -& *& ,! trivial -+--&D ·  On + *& < 1
+ C1 C2  C6  *& 4& -&" &D C1 ˆ 1& "*& POx9  ;& >+ gOx9 & POx9 ˆ QOx9 gOx9 ‡ ROx9

O(:19

"* &% R , &% g &!-% &-* , *&& $+   x n 1  !% *& ,- * &% R , &% g < n 1 O*& & &H!+ -& gOx9   -&"9 "& > ROx9 ˆ FPOx9G n

FQOx9 gOx9G n :

! FPOx9G n   -&" >+ ! $  FQOx9 gOx9G n   -&" >+

*&& (2 *! -& C  & ROx9   -&"  ! &% R , &% g  gOx9   4& -&" , & &%&&  ROx9 ˆ &- & POx9 ˆ QOx9 gOx9 "*-* *" * gOx9 ;& POx9

h

/& - " &  $;& *&  *&& >! -+-- -&  &>*&  && -&$&-& >&"&& -+-- -& , &%* n  - ; , x n 1 Theorem 8.3 O9 If C is an On k9 cyclic code over F then its generator polynomial is a divisor of x n 1 Futhermore, the vector C ˆ OC<  C1 . . .  C n 1 9 is in the code if and only if the corresponding generating function COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1 is divisible by gOx9 If k denotes the dimension of C then k ˆ n &% gOx9 O>9 Conversely, if gOx9 is a divisor of x n 1 then there is an On k9 cyclic code with gOx9 as its generator polynomial and k ˆ n &% gOx9 namely, the set of all vectors OC<  C1  . . .  C n 1 9 whose generating functions are divisible by gOx9 Proof of (a)  & POx9 ˆ x n 1  &  2O>9L FPOx9G n ˆ + *&& (2 + ;&- , &%* n "*& %&&% ,!-   !$& , gOx9   -&" ;&&+ , COx9 ˆ C< ‡ C1 x ‡


‡ Cn x n 1   -&" *& FCOx9G n ˆ COx9   &  2  $& * gOx9 ;& COx9 +

8.1 Introduction

1:7

*& & >! *& &%&& , gOx9 ," -& COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1   !$& , gOx9 ,  + , COx9 ˆ gOx9IOx9 "*&& &% I < n 1 &% g Proof of (b) !$$& * gOx9   ; , x n 1 *& COx9 ˆ OC<  C1  . . .  C n 1 9   !$& , gOx9 ,  + , COx9 ˆ gOx9IOx9 "*&& &% g ‡ &% I < n 1 *! *& & ,  !-* "   On k9 & -& "*&& k ˆ n &% g  *" * * -&  -+-- "& ! *" * *& %* -+-- *, , + -&"    -&" *! & IOx9 gOx9 >& + -&"L >+ *&& (1  %* -+-- *,  FxIOx9 gOx9G n  ! -& gOx9 ;& x n 1 "& *;& FxIOx9 gOx9G n  gOx9 ˆ FxIOx9 gOx9G  gOx9 ˆ
+ &

"*-* $;& * FxIOx9 gOx9G n   -+-- -&

O>+ &

 1O&99

 1O>99

!$& , gOx9  *& -&  &&  h

*&& (6 *" *&  $-& , *& %&& $+  ,  -+--& -&+ &&  &H!+  $ $+   *& parity-check $+  ,  -+-- -& "*-*  && >+ hOx9  &J& >+ hOx9 ˆ

xn 1 : gOx9

*& ,"% -&  *&& (6 %;& &@$- &-$ , %&&  $+-*&-# -& ,  -+-- -&  &  , gOx9  hOx9 Corollary 1 If C is an On k9 cyclic code with generator polynomial gOx9 ˆ g< ‡ g 1 x ‡


‡ g r x r (with r ˆ n k), and parity-check polynomial hOx9 ˆ h< ‡ h1 x ‡


‡ hk x k , then the following matrices are generator and parity-check matrices for CD

  g< g1 . . . . . . gr < . . . . . . < gOx9  < g   g1 . . . . . . gr < . . . <  <    x gOx9     G1 ˆ  ˆ ..   ..  ..   .  .   .
+ *& &J  &- :. *  &&   % -& /& &;& *& ;&% , *& *& NN- &II  >& (26 h

8.2 Shift-register encoders for cyclic codes

1(1


@ $& (3 "& " * gOx9 ˆ x 6 ‡ x ‡ 1 %&&&  O: .9  % : -& ! --%  >& (1 gOx9 ;&  + x 1 >!  x 1. 1L  ,- x 6 ‡ x ‡ 1jx n 1 , + n "*-*   !$& , : O&& >& (2.9 *! >+ *&& (6O>9 gOx9 %&&&  "*& , + , -+-- -& "* $ && O: .9 O1. 119 O21 1(9 . . . : "&;&  , *&& -& &@-&$ *& J -  ;&- "*& %&&% ,!-  x : 1   *;&  ! "&%* &H!  2 &-& *&& -& &  &&% , & -&-L "& * - *& improper -+-- -& O&& >& (2)9  *& & , * -*$& "*& "& &,&   On k9 -+-- -& "* %&& $+  gOx9 "& * ! &    proper -+-- -& & & , "*-* n  *& smallest $;& &%& !-* * gOx9jx n 1 * &%&   & & -& *& period , gOx9 -&   *& $& , *& &H!&-& Fx i  gOx9G i><  && >& (2: ,  -! , -+-- -& , "*-* *& parity-check $+  * $& & * n9 /* * -;& &>*& "& & &+  -!& ! !+ , -+--&

8.2 Shift-register encoders for cyclic codes " * "& #" >+ *&& (6 * &;&+ -+-- -&  -*-&4& >+  %&& $+  "& - >&%  && "*+ -+-- -& & !-* &&   $& & * >+ & -&  * &- "& * *"  ,- * &;&+ -+-- -& - >& &-& "*   $& J&& -*& -&  shift-register encoder &- *  &-% %* ,  On k9 & -& -+--     !& , $$% *& & , &%*k ,  &H!&-& OI <  I 1  . . .  I k 1 9  *& & , &%*n -&" OC<  C1  . . .  C n 1 9  &H!;&+ , $$% *& ,  $+  IOx9 ˆ OI <  I 1 x ‡


‡ I k 1 x k 1  *& & , -& $+  COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1  , *& -&  -+-- *&& (6 & ! * COx9   -&" ,  + , COx9   !$& , gOx9    "& "  + 1  *&& (6 & "+  -*%&  ,  $+  IOx9   -& $+    $+  !$+ IOx9 >+ gOx9D IOx9 ! IOx9 gOx9:

O(:29

 ! ! * $+  !$-  &+   $& & !%  & *% -& shift-register logic  "& * " #&  >&, !+ , * !>?&-2 %!& (1   >- &$&& ,  -*& -$>& , !$+%

xn ‡ 1 ˆ

Ox ‡ 19 Ox ‡ 192 Ox ‡ 19Ox 2 ‡ x ‡ 19 Ox ‡ 19. Ox ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox ‡ 192 Ox 2 ‡ x ‡ 192 Ox ‡ 19Ox 6 ‡ x ‡ 19Ox 6 ‡ x 2 ‡ 19 Ox ‡ 19( Ox ‡ 19Ox 2 ‡ x ‡ 19Ox ) ‡ x 6 ‡ 19 Ox ‡ 192 Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 192 Ox ‡ 19Ox 1< ‡ x 3 ‡ x ( ‡ x : ‡ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox ‡ 19. Ox 2 ‡ x ‡ 19. Ox ‡ 19Ox 12 ‡ x 11 ‡ x 1< ‡ x 3 ‡ x ( ‡ x : ‡ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox ‡ 192 Ox 6 ‡ x ‡ 192 Ox 6 ‡ x 2 ‡ 192

1 2 6 . 7 ) : ( 3 1< 11 12 16 1.

1Oˆ x n ‡ 19  $"& , &!->& $+  ;& GFO29 , 1 < n < 61

n

Table 8.1 -4 , x n

17 1) 1: 1( 13 2< 21 22 26 2. 27 2) 2: 2( 23 6< 61

Ox ‡ 19Ox 2 ‡ x ‡ 19Ox . ‡ x ‡ 19Ox . ‡ x 6 ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox ‡ 191) Ox ‡ 19Ox ( ‡ x 7 ‡ x . ‡ x 6 ‡ 19Ox ( ‡ x : ‡ x ) ‡ x . ‡ x 2 ‡ x ‡ 19 Ox ‡ 192 Ox 2 ‡ x ‡ 192 Ox ) ‡ x 6 ‡ 192 Ox ‡ 19Ox 1( ‡ x 1: ‡ x 1) ‡


‡ x ‡ 19 Ox ‡ 19. Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19. Ox ‡ 19Ox 2 ‡ x ‡ 19Ox 6 ‡ x ‡ 19Ox 6 ‡ x 2 ‡ 19Ox ) ‡ x . ‡ x 2 ‡ x ‡ 19Ox ) ‡ x 7 ‡ x . ‡ x 2 ‡ 19 Ox ‡ 192 Ox 1< ‡ x 3 ‡ x ( ‡ x : ‡ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ x 2 ‡ x ‡ 192 Ox ‡ 19Ox 11 ‡ x 3 ‡ x : ‡ x ) ‡ x 7 ‡ x ‡ 19Ox 11 ‡ x 1< ‡ x ) ‡ x 7 ‡ x . ‡ x 2 ‡ 19 Ox ‡ 19( Ox 2 ‡ x ‡ 19( Ox ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19Ox 2< ‡ x 17 ‡ x 1< ‡ x 7 ‡ 19 Ox ‡ 192 Ox 12 ‡ x 11 ‡ x 1< ‡ x 3 ‡ x ( ‡ x : ‡ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ x 2 ‡ x ‡ 192 Ox ‡ 19Ox 2 ‡ x ‡ 19Ox ) ‡ x 6 ‡ 19Ox 1( ‡ x 3 ‡ 19 Ox ‡ 19. Ox 6 ‡ x ‡ 19. Ox 6 ‡ x 2 ‡ 19. Ox ‡ 19Ox 2( ‡ x 2: ‡


‡ x ‡ 19 Ox ‡ 192 Ox 2 ‡ x ‡ 192 Ox . ‡ x ‡ 192 Ox . ‡ x 6 ‡ 192 Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 192 Ox ‡ 19Ox 7 ‡ x 2 ‡ 19Ox 7 ‡ x 6 ‡ 19Ox 7 ‡ x 6 ‡ x 2 ‡ x ‡ 19Ox 7 ‡ x . ‡ x 2 ‡ x ‡ 19Ox 7 ‡ x . ‡ x 6 ‡ x ‡ 19Ox 7 ‡ x . ‡ x 6 ‡ x 2 ‡ 19

1(.

Cyclic codes output g0

g1

g2

gr⫺1

gr

input

Figure 8.1 *,&%& --! , !$+% >+ gOx9 ˆ g < ‡ g1 x ‡


‡ g r x r

&;&+ *   &-& ,  On k9 -+-- -& "* %&& $+  gOx9 "* r ˆ n k

 arbitrary $+  IOx9 >+  ®xed $+  gOx9 &,& $;% * *& --! "# "& * >&& &@$  ;! - $&

*& --!  %!& (1  >! >+ -&-% %&*& *&& +$& , - $&D ¯ip-¯ops, adders  constant multipliers *&   $ , *&&  *& K$K$  & & -&  delay && &D

a flip-flop

K$K$   &;-& "*-* - & & && & , *& J& F  $-!&  !  $J& --! %  >!   $ $ , *&    &@& --# "*-* %&&&   % % ONN-#II9 &;&+ t< &-6 /*& *& --# -# *& -& , *& K$K$ & *,& ! , *& K$ K$  *& &- , *& " *!%* *& --! ! *& &@ K$K$  &-*& && *& % $ ! *& &@ -# *& -& , *& *, &%& - >& J& >+ *& --! >&"&& !--&;& K$K$ "*-*  "*&& *& *& " %- && & - &  *& "$! adder "*-* # #& *

an adder

B   &;-& * - $!& *& ! ,  " $! % O *& --!  "+ %&  !-*  "+ * ,& &;&+ -# &-* $!  *& & &-&;& &@-+ & %  * *& &I !$!  "+ ! >%!!9 + "& - &  *&  $& --! && & *& constant multiplier: a constant multiplier

8.2 Shift-register encoders for cyclic codes

1(7

* &;-&  $+ !$&  $! >+ *& - a "*  &+ " &I && , "& - !& "*+ *& --! , %!& (1 - >& !& , $+  !$- /& J "& !  & *& , ! , *& -&,J-& , *& $!- COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1  "*&& COx9 ˆ IOx9 gOx9D C< ˆ I < g<  C1 ˆ I < g1 ‡ I 1 g<  C2 ˆ I < g2 ‡ I 1 g1 ‡ I 2 g<  .. .

O(:69

Cj ˆ I < gj ‡ I 1 g

1

‡


‡ I j g< 

.. . Cn

1

ˆ Ik

1 gr :

*& K$K$ , % (1 & + J& "* & OC<  C1  . . .  C n 1 9 "*&& *& Cj I & &J& >+
H O(69   * --! - >& !&   &-& , *& On k9 -+--& "* %&& $+  gOx9 Comment The preceeding description is for a ``generic'' shift-register circuit, capable of performing arithmetic in an arbitrary ®eld F The required devices (¯ip-¯op, adders, multipliers) are not ``off-the-shelf'' items, except when the ®eld is GFO29 In this case, however, these devices are very simple indeed: ,$,$ &   >& &-& *& ,! K$K$ *! >& 4&   OI <  I 1  I 2  & *"&;&  &%  +& - *,&%& &-& , + -+-- -& "*-*  + %*+ & - $&@ * *& +& - &

Out

In

1

2

3

4

Figure 8.2 +& - &-& , *& O: 69 -+-- -& "* %&& $+  gOx9 ˆ 1 ‡ x 2 ‡ x 6 ‡ x . 

8.2 Shift-register encoders for cyclic codes

*& &   !& *& &! , *&& IOx9   ,  $+  *& IOx9 ! x r IOx9

1(:

(6 + 2 "*-* + * ,

Fx r IOx9G  gOx9

O(:.9

  +& - &-% !& ,  -+-- -& "* %&& $+  gOx9 *& -& - $&  !-*  &-& " >&  NN  gOx9II --! "*&& gOx9  * & -&I %&& $+  %!& (6O9 *" !-*  --! , gOx9 ˆ x r ‡ g r 1 x r 1 ‡


‡ g <   &  && "*+ *& --! , %!& (6O9  >&  $&,  NN  gOx9II --! "& ,-!  *& *,&%& -& Fs<  s1  . . .  s r 1 G "*-* "& - *& state vector , *& -*&   *& -&$% %&&% ,!- SOx9 ˆ s< ‡ s1 ‡


‡ s r 1 x r 1  "*-* "& - *& state polynomial /& >&% "*  &  "*-* &@$ *" *& & -*%&  &$&   $! Lemma 3 If the circuit in Figure 8.3(a) has state polynomial SOx9 and the input is s, then the next state polynomial will be S9Ox9 ˆ Os ‡ xSOx99  gOx9:

⫺g0

In

⫺g1

s0

⫺gr⫺1

s1

sr⫺1

(a)

In

(b)

Figure 8.3 O9 %&& NN  gOx9II *,&%& --! "*&& gOx9 ˆ x r ‡ g r 1 x r 1 ‡


‡ g <  O>9 $&-J- NN  x . ‡ x 6 ‡ x 2 ‡ 1II *, &%& "*&&  * &-   2

1((

Cyclic codes

Proof , *& $&& & ;&-  S ˆ Fs<  . . .  sr 1 G  *& $!  s *& >+ ! !& &->% *& >!% >-# , *& --! *& &@ & ;&- " >& >+ &J S9 ˆ Fs

g < sr 1  s
+ &  1O-9 h Theorem 8.4 If the circuit of Figure 8.3(a) is initialized by setting s< ˆ s1 ˆ


ˆ s r 1 ˆ  !-*%&  -# < *!%* k 1  "! #&   !$!% *& $+-*&-# & *& -&,J-& , Fx r IOx9G  gOx9  *& k* -# >! " *;&  " , r & -# >&,& Fx r IOx9G  gOx9  &+ ! &+   $>&  ; * r-# NN"  &II >+ !% *& --! *"  %!& (.O9  *& --! , %!& (.O9 *& $! > & ,&  *& right & , *& *, &%& *& * *& &,   %!& (6 *& &!  * , *& $! &  a<  a1  . . .  ,& *& t* -# *& *, &%&I & $+     " >& tjˆ< aj x r‡ t j  gOx9 *& * tjˆ< aj x t j  gOx9 O  ;&,+ * &  >& (619 *! , *& &-& $! *& k ,  + > I k 1  I k 2  . . .  I <  *& *, &%& , %!& (. *& -&,J-& , Fx r IOx9G  gOx9 " >& &+ % "* *& k* -#  && - $&& +& - &-& >&  *& *, &%& , %!& (.O9  *"  %!& (7O9 & *  - $&  *& +& &-& , %!& (1  *& &-& , %!& (7O9 *& ,  + > & &  *& &-&  *& reverse & ;4 I k 1  . . .  I <   *& -&" - $& &  &  *& -*&  *& &;&& & ;4 C n 1  C n 2  . . .  C1  C<  Example 8.11 , "& $$+ *& %&& -!- &->&  %!& (7O9  *& $&- -& , gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1 ;& *& J& GFO29 "& %& *& &-& &$-&  %!& (7O>9 , , &@ $& *& ,  &H!&-& OI 2  I 1  I < 9 ˆ O1 1 & O1 1 9 $&- -& gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1

-# < 1 2 6 . 7 )

$!

 -&

!$!

1 1 < < < <
+ J& GFO29  *! >& -& *"  !& *& *,&%& --! , %!& ()O9  >!  +& - &-& ,  -+-- -& "* $+-*&-# $+ 

⫺h1

st⫺1

⫺h2

st⫺2

⫺hk⫺1

⫺hk

st⫺k

(a)

(b)

Figure 8.6  O9 *,&%& --! *  $& & *& k*& & &-! k St ˆ jˆ1 hj S t j  O>9 *& $&- -& St ˆ S t 2 ‡ S t 6  -&$%  hOx9 ˆ 1 ‡ x 2 ‡ x 6 

13.

Cyclic codes

hOx9 ˆ 1 ‡ h1 x ‡


‡ hk x k O&& %!& (:O99 !% *& J k -# *& "-*   *& NN"II $  *& k ,  + > &  !&!+ & !  *& -*&  &  *& k%& &%&

*& *& "-*  $!  *& NN!$II $  !% *& & % r -# *& *,&%& --! --!& *& & % r -&" - $& ; k *& &-! Ci ˆ jˆ1 hj C i j  -& * *& +& - NNhOx9 &-&II , %!& (: ,,& , *& +& - NN gOx9 &-&II , %!& (7  "  $ "+D  *& gOx9 &-& *& ,  + > OI <  I 1  . . .  I k 1 9 --!$+ -&" - $& Cr  C r‡1  . . .  C n 1   *& -&" + > & &  *& -*&  *& &;&& & C n 1  C n 2  . . .  C<  "*&&  *& hOx9 &-& *& ,  + > --!$+ -&" - $& C<  C1  . . .  C k 1  *& -&" + > & &  *& -*&  *& ! & C<  C1  . . .  C n 1 Example 8.12 , "& $$+ *& %&& -!- , %!& (:O9  *& $-! O: 69 >+ -+-- -& "* gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1 

first k ticks: down last r ticks: up

⫺h1

⫺h2

⫺hk⫺1

⫺hk

Out

In (a)

first 3 ticks: down last 4 ticks: up

Out

In (b)

Figure 8.7 O9 +& - *,&%& &-& ,  On k9 -+-- -& "* $+ -*&-# $+  hOx9 ˆ 1 ‡ h1 Ox9 ‡


‡ hk x k  O>9 $&- -& hOx9 ˆ 1 ‡ x 2 ‡ x 6 

8.3 Cyclic Hamming codes

137

hOx9 ˆ x 6 ‡ x 2 ‡ 1 "& %& *& &-& *"  %!& (:O>9 *! , &@ $& , *& ,  + > & OI <  I 1  I 2 9 ˆ O1 1 9  $& & *& &-! Ci ˆ C i 2 ‡ C i 6  $!-& *& -&" O1 1 &  $& "*& k , r & k , n=2 h

8.3 Cyclic Hamming codes4  &- :. "& &J&  >+  % -& , &%* 2 m 1  >& + & -& "*& $+-*&-# @ *  -!  *& 2 m 1 4& >+ ;&- , &%* m arranged in any order  &@ $& *& ,"% $+-*&-# @ &J&  O: .9  % -& O-, *& @ H 6  &- :.9D

 < 1 1 1 1 < < H ˆ  1 < 1 1 < 1 < : 1 1 < 1 < < 1

*&& & , -!& O2 m 19M "+  & *&& -!   *!%* + & , *&& &% $!-&  $&,&- %&&-&-% -&  & &% & >&& * *& ,   $& & $  ,- "& * &&  * &- *   $>&  -*&  &% * $!-&  cyclic ;& , *&  % -& "*-* &    $& *, &%&  $& & , >* *& &-&  *& &-&  &->& *& $$$& &% "& ! ! & *& &&  >& ,  "* *& J& J& GFO2 m 9  "*-* &-* && &  &$&&& >+  >+ ;&- , &%* m O  $$&@  "& %;&  ! + , *& &&& ,-9

 >  -+--  % -& , &%* 2 m 1 >&% "*  $ ;& m  á 2 GFO2 9  !&   &J&  & -& C  ," ;&- C ˆ OC<  C1  . . .  C n 1 97 "* - $&  GFO29   C ,  + , C < ‡ C 1 á ‡


‡ Cn 1 á n

1

ˆ + & -& &J& >+ *& 1 3 n $+-*&-# @ H ˆ F1

á

á2

...

á n 1 G:

O(:39

/& *  $;& * *& -& &J& >+ O((9  O(39   -+- % -&  " *&  $ *%  -& >! * &J 

13)

Cyclic codes Table 8.2 *& $"& , á  GFO(9 "*&& á6 ˆ á ‡ 1 i

ái

< 1 2 6 . 7 )

+  &$&&   m> -!  ;&-  &@ $& "* m ˆ 6 , á   $ ;&   GFO26 9 ,+% á6 ˆ á ‡ 1 *& >& (2 *" *& $"& , á &$&&&  *&& & ;&- *! *& 1 3 : GFO(9 @ H  O(39 &J& *&  & >+ -&  *& ,"% 6 3 : >+ @ H9D

 < < 1 < 1 1 1 H9 ˆ  < 1 < 1 1 1 < : 1 < < 1 < 1 1

*& ,"% *&&

 *&

 &! , * &-

Theorem 8.6 The code de®ned above (i.e., by (8.8) or (8.9)) is an On n m9 binary cyclic code with generator polynomial gOx9 the minimal polynomial of á Furthermore, its minimum distance is 6 so that it is a Hamming code Proof   -& * *& -&   >+ & -& , &%* n /& &&  *" *   -+--  * gOx9  *& %&& $+   *" *   -+-- & * , "& !$+ &H! O((9 >+ á  !& *& ,- * á n ˆ 1 "& %& Cn

1

‡ C< á ‡


‡ C n 2 á n

2

ˆ + *&& (6 gOx9  *& -&I %&& $+ 

 - $&& *& $, "& &&  *" * *& -&I  ! -& d   6 -& *& -&  & d  ˆ w   /& !& *& &J O((9L , *&& "&&  -&" , "&%* 1 *& á i ˆ < ,  & i "*-*   $>&  +  " , "&%* 2 -! &@ ,  + , á i ‡ á j ˆ < ,  & i  j "* < < i , j < n 1 ;% >+ á i * >&- & 1 ‡ á j i ˆ &  >!  $& *,&%& &-& ,  % -& >&  *& %&& $+  -& *& %&& $+  ,   % -& ! >&  $ ;& $+   &   $& &  -+--  % -& , &%* 2 m 1   &-&+  *;&  $ ;& $+  , &%&& m  >& (6 "&  & $ ;& $+   , &%&& m , 1 < m < 12   %;& ;!& , m *&& "  %&& >& + $ ;& $+  , &%&& m >! *& &  >& (6 & -*&  *;& *& ,&"& $>& 4& -&,J-& "*-* " &  *,&%& &-& "* *& ,&"& $>& 2 & &  $ * *& &-%  $-+ ,  % -& *"&;&  *& ,-    $>&  >!  $& decoders , *&  /& !& !-*  &-& , *& O: .9  % -& "* %&& $+  gOx9 ˆ x 6 ‡ x ‡ 1  %!& (( *& &-& - , *&&  $ " *, &%& --!   AND %& *& !$$& *, &%&   NN  gOx9II --!  *& "& *, &%&   NN  x n ‡ 1II --!

*&  %& !$!  1 , *& !$$& *, &%& - *& $& 1< . . . & %&& $+  ,   % -& m

 ;& $+  , &%&& m

1 2 6 . 7 ) : ( 3 1< 11 12

x‡1 x2 ‡ x ‡ 1 x6 ‡ x ‡ 1 x. ‡ x ‡ 1 x7 ‡ x2 ‡ 1 x) ‡ x ‡ 1 x: ‡ x ‡ 1 x( ‡ x: ‡ x2 ‡ x ‡ 1 x3 ‡ x. ‡ 1 x 1< ‡ x 6 ‡ 1 x 11 ‡ x 2 ‡ 1 x 12 ‡ x ) ‡ x . ‡ x ‡ 1

mod g(x) circuit

Switch A: closed for first n ticks open for next n ticks

AND gate: 10…0 recognizer Switch B: open for first n ticks Switch B: closed for next n ticks mod (xn ⫹ 1) circuit

Figure 8.8 &-% --! , *& O: .9  % -& "* %&& $+  gOx9 ˆ x 6 ‡ x ‡ 1

OR<  R1  . . .  Rn 1 9 -& ROx9 ˆ COx9 ‡ EOx9  COx9  gOx9 ˆ & x t EOx9  gOx9  *& "& *, &%& " - x t ROx9  x n ‡ 1 & *& t* %* -+-- *, ORn t  . . .  Rn t 1 9 , *& &-&;& " *& " *, &%& & -&-& ; "-*   *& AND %& "*-* !$!  1 ,  + , *& !$$& *, &%& - 1< . . . & -&-& *& ,& Oe 19  n ,!*& -# *& &-&;& " " *;& - $&&  --! ?!&+ ! *& "& *, &%&  " $$& "*  & -&-&  *& "& *, &%&  &- (7 "& " && *"  %&&4& *& --! , %!& ((  &  >!  &-& ,  -+-- burst&-&-% -& *& !& +% *&+ , >!& -&- " >& $&&&  &- (. O -&+ *& dual -& , -+--  % -& &  &&%   $ *&+ & -;&& >&K+  >& (.19

8.4 Burst-error correction  + -*& , $--  $-& & "*& *&+ --! &  --!  bursts *+-+  >! , & --! "*& ,  & & *& -*& & &;&+ -&& ,  >&,  &  *& &!     * &- "& " && *" -+-- -& - >& !&  &&-  -&- & >! /& >&% "*  $!&+ *& - &J ,  & >! ,  -&" C   &   &-&;&  R ˆ C ‡ E *& *& & ;&- E  -&  burst of length b , *& 4& - $& , E & -J&  b -&-!;& - $&  &@ $& E ˆ O  E  &% "* *&  4& + > *& >!I location  *& &@ , *& J 4& + >  *& >!  &@ $& *& ;&- E ˆ O! &-$ , w ˆ <  1 *&&,& *&&  *%  $;&  "& ! & w > 2

8.4 Burst-error correction

2! &-$ , E *& $& " -  , EI 4& - $&   *& - $& , E not -!&  *& $& " ,  -+-- ! , &%% ?! ,& *&  4& - $&  *& $&  -!% ! ?! >&,& *& J 4& - $&  *& $& *& & , -& -&$%  * ! , ! &-$  &@ $& , E ˆ O+ % , &%* b "*-* >&% "*  1 *! *&& & 2 b 1 $>& $&     , n
2 b 1 4& >! , &%* < b

% 1  * ! >&  --! , *& 4& >! "& %& n2 b 1 ‡ 1  && h

*& &@ " *&&  %;& !&,! >!  *& 4& , -& "*-* -&- >! &   $-+ "& * -  -& -$>& , -&-%  >!& $& , &%* < b  burst-b-error-correcting code Theorem 8.9 OThe Hamming bound for burst-error correction9 If 1 < b < On ‡ 19=2 a binary burst-b-error-correcting code has at most 2 n = On2 b 1 ‡ 19 codewords Proof + *&& (( *&& & n2 b 1 ‡ 1 >!& $& , &%* < b , *&& & M -&" *&& & *& MOn2 b 1 ‡ 19 " "*-* ,,& ,  -&" >+  >! , &%* < b *&& " !  >& -   MOn2 b 1 ‡ 19 < 2 n  h Corollary (the Abramson bounds). If 1 < b < On ‡ 19=2 a binary linear On k9 linear burst-b-error-correcting code must satisfy n < 2r where r ˆ n

b‡1

1 Ostrong Abramson bound9

k is the code's redundancy. An alternative formulation is

r > d%2 On ‡ 19e ‡ Ob

19 Oweak Abramson bound9:

Proof   & On k9 -& *&& & M ˆ 2 k -&"   >+

*&& (3 2 k < 2 n =On2 b 1 ‡ 19 &%% * "& %& n < 2 r b‡1 2 b‡1  -& n ! >&  &%& * >! - >&  $;&  n < 2 r b‡1 1 *& % >  >! &%% this  >  >!  r "& %& *& "&# >  >! h

8.4 Burst-error correction

2+ *& $%&*& $-$& *&& ! >& " - -&" "*-* %&&  *& J n 2b -& *&& " -&" - *& >& &$&&& -*& -+  ,"D 2b

n 2b )****************+,****************- )***********+,***********X ˆ        A A A A A A

Y ˆ       B B B B B B "*&& NNII && %&& &  *& AI  BI & >+ ! *& *& " Zˆ        A A A B B B ,,& ,

>* X  Y >+  >! , &%* < b a --

h

Corollary (the Reiger bound). If < < b < n=2 a binary On k9 linear burstb-error-correcting code must satisfy r > 2b where r ˆ n

k is the code's redundancy

Proof *& ! >& , -&"   On k9 >+ & -&  2 k  "*-* >+ *&& (1< ! >& < 2 n 2b  *  &H!;&  *& & & , *& -+ h /& & "   $  -!  && , &@ $& , >!& -&-% -&  &-* -& *& -& " >& -+--  && &*& *& % >   &%& >! O"*-* $$+   & -&  ?! -+-- -&9  * -! "*& "& + *  $-! >! O&*& *& >  >!  *& &%& >!9  tight "& & * *&& &@  -& "*& &!-+  &H!  *& ;!& , *& >! ,  !-* -& &@ "& " + * *& >!  loose Example 8.13 *& On 19 >+ &$& -& "* gOx9 ˆ x n 1 ‡ x n 2 ‡


‡ x ‡ 1 "*&& n   - -&-  & $& , "&%* < On 19=2     >!OOn 19=29&-&-% -& -& r ˆ n 1 *& &%& >!  %* h

2& -&" , &%* n   -+-- -& "* gOx9 ˆ 1   O;+9  b ˆ < >!&-&-% -&  -& r ˆ <  *& &%& >!  % %* h Example 8.15 + >+  % -& "* n ˆ 2 m 1 r ˆ m  gOx9 ˆ  $ ;& $+  , &%&& m   b ˆ 1 >!&-&-% -& O + & ;&- , "&%* 1  $ ,-  >! , &%* 19 *& % >  >!  %* ,  *&& -& h Example 8.16{ + -+--  % -& , "*-* *& -&" ,  "&%* *;& >&& & ;&   b ˆ 2 >!&-&-% -& -&  Abramson code *&& -& & -+-- -& "* %&& $+  , *& , gOx9 ˆ Ox ‡ 19 pOx9 "*&& pOx9   $ ;& $+  O&& >& (779 *&  & >  -&  *& O: 69 -+-- -& "* gOx9 ˆ Ox ‡ 19Ox 6 ‡ x ‡ 19  $+-*&-# @   1 á á2 á6 á. á7 á) Hˆ  1 1 1 1 1 1 1 "*&& á   $ ;&  ,+% *& &H! á6 ‡ á ‡ 1 ˆ <  O(9

 ;&,+ * * -&  &&  b ˆ 2 >!&-&-% -& "& &&  -*&-# * *& + & ,  >! , &%* < b < 2 & -  b ˆ < O*& 4& & $&9 + &  <   b ˆ 1  >! "* á*& i &-$ O1 i9 * + &   b ˆ 2  >! &-$ O11 i9 * 1 i + & á< Oá‡19  *&& 1 ‡ 2n + & &  -   *& -&  &&  b ˆ 2 >!&-&-% -& & J+ * , gOx9 ˆ Ox ‡ 19 pOx9 "*&& pOx9   $ ;& $+  , &%&& m *& n ˆ 2 m 1 r ˆ m ‡ 1  b ˆ 2  * *& % >  >!  %* O m ˆ 6 & *& O: 69 b ˆ 2 -& *& &%& >!  %*  >! ,  %& ;!& , m *& &%& >!  &9 h Example 8.17) *& O17 39 >+ -+-- -& "* %&& $+  gOx9 ˆ Ox . ‡ x ‡ 19Ox 2 ‡ x ‡ 19 ˆ x ) ‡ x 7 ‡ x . ‡ x 6 ‡ 1 ! !  >&  b ˆ 6 >!&-&-% -& , "*-* *&&,& >* *& %

>   &%& >! & %*  $;& * *   "& !& *& $+-*&-# @   1 á á2 . . . á1. Hˆ  1 ù ù2 . . . ù1. "*&& á   $ ;&   GFO1)9 ,+% á. ‡ á ‡ 1 ˆ &-!& , *& NN&"&& *& $& 1 11  1& , *& , ù t L  %!* >&"&& *&  "& #  Os t9  6 , *& $&  1 *& s ˆ i t ˆ i  Os t9  6 ˆ ! &-$ 11  &@ $& *& + & á1 * s ˆ 11 t ˆ ! &-$ O11 :9 & *& & $&  O& + *&& " , * -& "*-* "& - $+   6 3 : +D A< B< C
! >+ &% *& >;& + >+ -!   -& *& interleaving , A B  CD A< B< C< A1 B1 C1 A2 B2 C2 A6 B6 C6 A. B. C. A7 B7 C7 A) B) C) : & ! !$$& * * % -&"   & ;&  >!+ -*&  !,,&  >! , &%* ) -& >+ ID A< B< C< A1 B1 C1 A2 B2 C2 A6       B7 C7 A) B) C) :   $>&  -&- * >! , &  $+ >+ NN&&&;%II * % -&"   - $& -&" >&-!& ,& &&&;%  & , *& *&& -&" " *;& !,,&&  >! %& * "D A< B< C
& &&;% , *&& -&" , *& O: 69 >  -&  -& *& depth6 interleaving , *& % -&    O21 39 & -&L  *& %! & >;& *" *    ,-  b ˆ ) >!&-&-% -& & %&&+ , + $;& &%& j *& &$* j &&;% , *& O: 69 >  -&   O: j 6 j9 b ˆ 2 j >!&-&-% -& & * &-* & , *&& -& J& *& &%& >! "* &H!+ O-& r ˆ . j  b ˆ 2 j9  *   -& &&   , &,J-&-+ &! "*&  -&  &&;& h

%*," %&&4 , *& %! & %;& 
@ $& (1( &  *& ,"%  $ *&&  Theorem 8.11 If C is an On k9 linear burst-b-error correcting code, then the depth-j interleaving of C is a Onj kj9 burst-bj-error-correcting code

2;! >!   &;&*&& !& * , "& &&;&  cyclic -&  &$* j *& &!% -&   -+-- *& &@ *&& $& * ! Theorem 8.12 If C is an On k9 cyclic code with generator polynomial gOx9 then the depth-j interleaving of C is an Onj kj9 cyclic code with generator polynomial gOx j 9



C
+ C< C1


C j 1  &+ &  "*& $, "& &;&  >& ()7 *" *"  - $!& *& %* -+-*, ,  &&;& -&"D O(:1  -& "* %&& $+  gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1  !% *& &&;% &-*H!& "& - $!-&  J& , + , -+-- >!&-&-% -& ;4 *& O: j 6 j9 b ˆ 2 j -& "* %&& $+  g j Ox9 ˆ x . j ‡ x 6 j ‡ x 2 j ‡ 1  + % "* *& -+-- O17 139 b ˆ 6 -& ,
@ $& (1: "& > *& J& , + ;4 *& O17 j 3 j9 b ˆ 6 j -+-- -& "* %&& $+  gOx9 ˆ x ) j ‡ x 7 j ‡ x . j ‡ x 6 j ‡ 1 & * &;&+ -&  &-* , *& , & && *& &%& >! O+  *&& (1+ -+-- n ˆ 17 >! &-&-% -& , b ˆ 1 2 . . .  :  &-* !-* ;!& , b "&  *& "& >!  *& &&& &!-+ !% *& O"&#9 >  >! OrA 9  *& &%& >! OrR 9 + -*-&  ! ! *  &;&+ -& *&&   -+-- -& "*& &!-+  @OrA  rR 9  "& *;& & *& %&& $+  gOx9 , !-*  -&  &-* -& *& -& b ˆ 1 2 6 "& *;& &+ -

8.4 Burst-error correction b 1 2 6 . 7 ) :

rA rR gOx9 . 7

2 x. ‡ x ‡ 1 . Ox . ‡ x ‡ 19Ox ‡ 19

) ) Ox . ‡ x ‡ 19Ox 2 ‡ x ‡ 19 : ( Ox . ‡ x ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19 ( 1< Ox . ‡ x ‡ 19Ox 2 ‡ x ‡ 19Ox . ‡ x 6 ‡ 19 ˆ x 1< ‡ x 7 ‡ 1 3 12 Ox . ‡ x ‡ 19Ox . ‡ x 6 ‡ x 2 ‡ x ‡ 19 Ox . ‡ x 6 ‡ 19 ˆ x 12 ‡ x 3 ‡ x ) ‡ x6 ‡ 1 1. 1< 1. x ‡ x 16 ‡


‡ x ‡ 1

2  -& O
@ $& (1)9
@ $& (1: >& (7( O6 19 &&;& 3 7 >&

()
& , "+ , *& *& -+-- >!& -&-% -& "& *;& !& *&   $ ,,&&-&  *& %!&& >+  detect + & >! * &  %  correct && -& & strong >!&-&-% -&  *& && , *& ,"% &J De®nition An On k9 cyclic code with generator polynomial gOx9 is said to be a strong burst-b-error-correcting code, if, whenever Z1 and Z2 are burst-error vectors with the same syndrome, and with burst descriptions O$&1  -1 9 and O$&2  -2 9 such that &%*O$&1 9 ‡ &%*O$&2 9 < 2b then Z1 ˆ Z2 

21
!&-&-% -& & -$>& ,  !&! >!& -&-  &&- Theorem 8.13 If C is a strong burst-b-error-correcting code, then for any pair of nonnegative integers b1 and b2 such that b1 < b2 and b1 ‡ b2 ˆ 2b it is possible to design a decoder for C that will correct all bursts of length < b1  while at the same time detecting all burst of length < b2  Proof * ," , *&& :.  *& &J , % >! -&-  && * & E && *& & , & >! , &%* < b1   & F && *& & , & >! , &%* < b2  *& , Z1 2 E  Z2 2 F  "& #" * Z1  Z2 *;& >! &-$ !-* * &%*O$&1 9 < b1  &%*O$&2 9 < b2  ! b1 ‡ b2 ˆ 2b   >+ *& &J ,  % >!b&-&-% -& Z1  Z2 *;& ,,&& + & *! >+ *&& :. C * *& ;&& -$>+ h Example 8.21{  >!b&-&-% -& -!%  , *& -&  >& (6 & not %  &@ $& -& *& O: 69 b ˆ 2

>  -&  $+-*&-# @ - >& #&  >&   1 á á2 á6 á. á7 á) Hˆ  1 1 1 1 1 1 1 "*&& á6 ‡ á ‡ 1 ˆ <  GFO(9  "& + 
@ $& (1) * -& -&-  >! , &%* < 2L >!  - -&-  >! , &%* < 1 "*& &&-%  >! , &%* < 6 -& &% *& & $& 7 O111 m and ®nally that g 1 Ox9 and g2 Ox9 are relatively prime. Then gOx9 ˆ g 1 Ox9 g 2 Ox9 is the generator polynomial for an On k9 cyclic code which is a (strong) burst-b-error-correcting code, whereD n ˆ - On1  n2 9 kˆn bˆ

&%O g 1 9

&%O g 2 9

Ob1  m On1 ‡ 19=29:

Proof + *&& (6O9 g 1 Ox9jx n1 1  g 2 Ox9jx n2 1 *! -& >* x n1 1  x n2 1 ;& x n 1 "*&& n ˆ - On1  n2 9  -& g 1 Ox9  g2 Ox9 & &;&+ $ &  ," * gOx9 ˆ g 1 Ox9 g 2 Ox9jx n 1   >+ *&& (6O>9 gOx9 %&&&  On k9 -+-- -& "*&& k ˆ n &%O g9 ˆ n &%O g 1 9 &%O g 2 9 * !-*  &+ *& *& , *& $, & "* *& & >! *& >!&-&-% -$>+ , *& -& /& " ! & * *& -& %&&& >+ g 1 Ox9  %  *" * *& &!% %& -&   % *& $, ,  NN"&#II g1 Ox9  &,  >& ()3

*! & Z1  Z2 >& " & ;&- , &%* n "* *&  & + & &;&  *& -+-- -& "* %&& $+  gOx9  "* >! &-$ OP1  i9  OP2  j9 !-* * &%*OP1 9 ‡ &%*OP2 9 < 2b /& &&  *" * Z1 ˆ Z2  &I && *& %&&% ,!- , *& " >! patterns >+ P1 Ox9  P2 Ox9  ," * *& %&&% ,!- , *& & ;&- Z1  Z2 & Fx i P1 Ox9G n  Fx j P2 Ox9G n &$&-;&+   *& & & +  & & Fx i P1 Ox9G n  gOx9  Fx j P2 Ox9G n  gOx9 ! -& gOx9jx n 1 &  1O&9  $& * *& " + & -  ,- >& "&  S1 Ox9 ˆ x i P1 Ox9  gOx9 S2 Ox9 ˆ x j P2 Ox9  gOx9: -& Z1  Z2 & ! &  *;& *&  & + & "& *&&,& *;& x i P1 Ox9  x j P2 Ox9 O  gOx99: -& gOx9 ˆ g1 Ox9 g 2 Ox9 *&  x i P1 Ox9  x j P2 Ox9 O  g 1 Ox99:

O(:119

212

Cyclic codes

! &%*OP1 9 ‡ &%*OP2 9 < 2b >+ ! $  2b < 2b1 >+ *& &J  , b *! -& g1 Ox9 %&&&  % >!b1 &-&-% -&  ," * , *& ;&"$ , *& -+-- -& %&&& >+ g 1 Ox9 *& & ;&- Z1  Z2 & &- & x i P1 Ox9 ˆ x j P2 Ox9 O  x n1

19:

! &%*OP1 9 ‡ &%*OP2 9 < 2b < n1 ‡ 1   >+ *&& P1 ˆ P2  iˆ j

(: O(:129

O  n1 9:

O(:169

 ;&" , * O(119  $& Ox i

x j 9P1 Ox9 
% xi

xj 
% * ,- "* O(169  !% *& ,- * n ˆ - On1  n2 9 "& *;& i  j O  n9L >! -& i  j >* &  *& %& b and period n<  Then gOx9 is the generator polynomial for an On n 2b ‡ 1 m9 cyclic code which is a strong burst-b corrector, where n ˆ - O2b 1 n< 9 This code is called a Fire code, in honor of Philip Fire, its discoverer Proof * ,"  &&+ , *&& (1. >+ #% g 1 Ox9 ˆ Ox 2b 1 19  g 2 Ox9 ˆ f Ox9 O 
@ $& (22 "& " * *& g 1 Ox9 -&   % >!b -&-9 h Example 8.23 --%  *& -+  *&& (1. *& >+ -+--& "* gOx9 ˆ Ox 6 ‡ 19Ox 6 ‡ x ‡ 19 ˆ x ) ‡ x . ‡ x ‡ 1 %&&&  O21 179 % b ˆ 2 >!&-&-% & -& *!   $>& 

8.4 Burst-error correction

216

&;&  &-& , * -& * -&-  >! , &%* < 2L   -&-  >! , &%* < 1  &&-  >! , &%* < 6L   &&-  >! , &%* < . & * * -& && &*& *& %

>  >!  *& &%& >!L  *& *& * -& *  && *&& >! & $$&+ &;& % >! -&-  & * , "& #& g 1 Ox9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1 "*-* %&&& *& O"&#9 O: 69 b ˆ 2

>  -&  g 2 Ox9 ˆ x 2 ‡ x ‡ 1 *&& (1.  $& * gOx9 ˆ g 1 Ox9 g 2 Ox9 %&&&  O"&#9 O21 179 b ˆ 2 >!&-&-% -& ! g 1 Ox9 g 2 Ox9 ˆ x ) ‡ x . ‡ x ‡ 1  "& *;& &+ && * * $+   %M h

Example 8.24 *& $+  P67 Ox9 ˆ x 67 ‡ x 26 ‡ x ( ‡ x 2 ‡ 1   $  ;& >+ $+  , &%&& 67   >+ *& -+  *&& (1. gOx9 ˆ Ox 16 ‡ 19P67 Ox9 ˆ x .( ‡ x 6) ‡ x 67 ‡ x 26 ‡ x 21 ‡ x 17 ‡ x 16 ‡ x ( ‡ x 2 ‡ 1 %&&&  % -+-- O16O267 19 16O267 19 .(9 ˆ O..)):)73(::1 ..)):)73(:269 b ˆ : & -& * $-! & -&  H!& , ! >&-!&  * !&  NN*&&II ;& ,   + ,  # ;& /*   *&& -+-- -&P  %&& , gOx9 * &%&& r  %&&&  On n r9 -+-- -& *& , + n< < n gOx9  %&&&  On<  n< r9 *&& -+-- -& * -& - ,  ;&- C , &%* n< "*& %&&% ,!- COx9   !$& , gOx9  *& -& , *&  & -& n< ˆ 172772  *& -& !&  -!+  O172772 1727! &-&-% -& *&  &-&  ;&+ -&;;&  + & $  -&- >! , &%* < .   -& *& -&  %  *&&>+ % *& >+  &&-  >! , &%* < 1& >& (7 %  & !&,! >+ & -&  &-* , *&& -& gOx9 ˆ Ox 2b 1 ‡ 19Pb Ox9 "*&& Pb Ox9   $ ;& $+  , &%&& b *& &!-+  *! r ˆ 6b 1 *!%* *& &!-+ &H!& >+ *& "&# >   &%& >!  +$-+ -&>+  & * * * NN&@II &!-+  $$&+ *& $-& * ! >& $  & * *& -& >& % >! -&-

b

O67 2:9 6 O1&% *& & $& , *& &-& - $!& *& & & + & SOx9 &J& >+ SOx9 ˆ ROx9  gOx9 *& *& ;&- >& >+ !>-% SOx9 , ^ ˆ ROx9 COx9

SOx9

ROx9 ;4 O(:1)9

^  %!&&  >&  -&" *  >&-!& COx9  gOx9 ˆ ROx9  gOx9 SOx9  gOx9 ˆ +  & $&  E * !-*  !& , + -+-- -& &I " -& *& $&- -& ,  -+-- burst&-&-% -& !$$& *& * C   >!b&-&-% -&  * *& + & SOx9 J& *& " - SO+ *& >;& -! *& &-& - ,&+ & * COx9  &J&  O(1)9  *& -!  & -&" &-%  *&&,& &+ , O(1:9  J& ,!&+ *"&;& * + *$$& , *& & ;&-   >! -&  $ !  -&   $ other * + #% ;%& , *& NN-+--&II , *& -& &&I *"D , *& & ;&- EOx9   4& >! , &%* < b *& EOx9 *  !H!& >! &-$ , *& ,

21)

Cyclic codes

OPOx9 i< 9 "* PO!  -&  $ & -&-&  &&+ , i< 6ˆ 9 *& ! >& , -+-- *, &H!&   *  *& !H!& &%& j<  *& %& < < j< < n 1 !-* * i< ‡ j<  < O  n9 "*-*  j< ˆ O i< 9  n: , " R j< Ox9 && *& j< * -+-- *, , ROx9 "& *;& R j< Ox9 ˆ C j< Ox9 ‡ E j< Ox9 "*&& C j< Ox9  Ej< Ox9 & *& j< * -+-- *, , COx9  EOx9 &$&-;&+ " C j< Ox9   -&" -& *& -&  -+--  E j< Ox9 ˆ POx9 >+ *& &J , j<    , S j< Ox9 && *& & & + & , R j< Ox9 "& *;& (a) P E(x):

1XXXX

S(x) ⫽ [xi0P(x)]n mod g(x).

i0 (b) P Ej0(x):

1XXXX

Sj0(x) ⫽ P(x).

0

Figure 8.9 O9 *& >!& ;&- EOx9 ˆ Fx i< POx9G n O>9 ,& *,% EOx9 j< !  *& %* "*&& j< ˆ On i< 9  n *& & $&  NN$$&II  - & NN$$&II   - >& -&-& && *& >!& pattern " >& %;& >+ S j  Ox9  *& >!& location " >& %;& >+ *& , ! i< ˆ O j< 9  n "*&& j<  *& ! >& , *, &H!&  $ *& &  * $ "& -! &%  ,+  $& &-& !% *&& & "&;& >&,& %  "& "*  & * *& --! , *& !--&;& S j Ox9I - >& -&>+  $J& >+ !% *& ,"% &! Theorem 8.15 OMeggitt's lemma9 For j > < de®ne Sj Ox9 ˆ Fx j ROx9G n  gOx9 i.e., Sj Ox9 is the remainder syndrome of the jth cyclic shift of ROx9 Then for j > + &

 1O&9

Sj Ox9 ˆ Fx j ROx9G  gOx9 -& gOx9jx n

1 *&

FxSj Ox9G  gOx9 ˆ FxOFx j ROx9G  gOx99G  gOx9 ˆ Fx j‡1 ROx9G  gOx9 ˆ S j‡1 Ox9:

>+ &

 1O9 h

Example 8.25 /& !& *&& & !% *& O: 69 b ˆ 2 >  -& "* gOx9 ˆ x . ‡ x 6 ‡ x 2 ‡ 1 !$$& *& &-&;& ;&-  R ˆ

21(

Cyclic codes

F1!& $&  !$!  & 3 O *& %*  "# , *&& & no &L && >& (:79

*& %* &->&  %!& (1< & &,   $& & >+ *,&%& %- /& !& *  %!& (11  (12 , " ,,&& >!&-&-% -+-- -& *& O: 69 b ˆ 2 >  -& ,
@ $& (1)  *& O17 39 b ˆ 6 -& ,
@ $& (1:  *& &-% --!  %!& (11  (12 *&& & *&&  - $&D   gOx9 *, &%&  x n 1 "& *, &%&   NN1+ *&& (.9  *& "& *, " - ROx9 &@ "-*  opened  "-*   closed *! "% *& !$$& *, &%&  - !-&  *& "& *, &%& ; *& 1& &!&  *& % $ "* *& & >! -&-&  %&& --! #& *&&  %!& (11  (12 & -& NN,! $&--#!II &-& *&+ &H!& &@-+ 2n --# -#  -&- + >! , &%* < b "* *"& - $&@+ *  OOn9 *&+ &  "&$& !&  $--& "&;& , -+ *&& -+-- >! &-&-% -& O&%
@ $& (2.9  J- , * --! + >& &H!&Q&& >& (:: Problems 8.1 *" * *& ! >& , k & !>$-& ,  n & ;&- $-& ;& *& J& GFOq9  &@-+ Oq n 19Oq n 1 Oq k 19Oq k

1

19 . . . Oq n k‡1 19  19 . . . Oq 19

 !& * , !  ;&,+ * *&& & &@-+ 11(11 >+ O: 69 & -& 8.2 -!& *& ,"%D O9 19 x 1<  Ox 12 19L O-9 Ox 17 19  x 1< 1 8.3  *& NN II $& -;& E - !;&P *  & *& && OP  Q9  R ˆ P  OQ  R9 P  Q ˆ Q  P !&  %&&P

Problems

221

8.4 *" * , m 2 f9 ;& &  1 $ O>9 O-9 ;& &  1 $ O-9 O9 ;& &  1 $ O9 O&9 ;& &  1 $ O&9 8.6 ;&  ,  $, * , C   -+-- -&  , COx9 2 C *& ,  i > 1 x i COx9  x n 1 2 C HintD & &  1O9G 8.7  *& G1  H 1 -& , *& O( .9 -+-- -& ;& GFO69 "* %&& $+  gOx9 ˆ Ox 2 ‡ 19Ox 2 ‡ x ‡ 19 O&& + 1  *& & (69 8.8  *& G2  H 2 -& , *& O( .9 -+-- -& ;& GFO69 "* %&& $+  gOx9 ˆ Ox 2 19Ox 2 ‡ x ‡ 19 O&& + 2  *& & (69 8.9 O9 *" * *& ,"% -& & %&& $+-*&-# -& ,  On k9 -+-- -& "* $+-*&-# $+  hOx9 O NN&;&&II ~ $+-*&-# $+  hOx99D ~ n 1 G6 ˆ Fx j hOx9G jˆ< H 6 ˆ Fx i‡ k

O-! 9

~ r 1 x i‡ k hOx9G iˆ
& --& *& @ H 1 , + 1  *&& (6 O9 , *& + & S , *& ;&- R ˆ OR<  R1  . . .  Rn 1 9  --!&  S T ˆ H 1 R T  *" * *& %&&% ,!- ROx9 ˆ R< ‡ R1 x ‡


‡ Rn 1 x n 1  SOx9 ˆ S< ‡ S1 x ‡


‡ S r 1 x r 1 & && >+ SOx9 ˆ

FROx9hOx9G  x n

FROx9hOx9G  x k xk

O"*-*   ?! -! + "+ , +% * OS<  S1  . . .  S r 1 9 & *& -&,J-& , x k  x k‡1  . . .  x n 1  *& $!- ROx9hOx99 O>9 % *& &! , $ O9 J *& + & , *& ;&- R ˆ F1&-!& , *& 1) -&"  *& -& + F11&K+ improper -+-- -& & *& , "*-* *& $& , *& %&& $+  gOx9  & * n  * $>&  "& " ;&%& -+-- -& , "*-* *& parity-check

Problems

226

$+  * $& & * n *! & C >&  On k9 -+-- -&  & x n 1 ˆ gOx9hOx9 "*&& gOx9  *& %&& $+   hOx9  *& $+-*&-# $+  , C !$$& ,!*& * *& $& , hOx9  n< , n  * C<  *& On<  k9 -+-- -& "* %&& $+  gOx9 ˆ Ox n< 19=hOx9 O9 /*  *& $+-*&-# $+  , C< P O>9 , d <  *&  ! -& , C<  "*  *&  ! -& , CP O-9 *" * C ˆ On=n< 9C<  "*&& NN jC< II && *& -& , &%* n >& , C< >+ &$&% &-* -&" j  & 8.28  &-* j  *& %& 1 < j < 1) J *& &%*  &   ! -& , *& O$$&9 >+ -+-- -& "* %&& $+  Ox ‡ 19 j  F HintD *& &! , *& $&;! $>& " >& *&$,!G 8.29 , gOx9 * $& n  &%&& r *& , + m > 1 gOx9jx mn 1   >+

*&& (6O>9 gOx9 %&&&  Onm nm r9 -+-- -& O m > 2 * -&   $$&9 O9  *" *  ;&- C ˆ FC<  C1  . . .  C nm 1 G   * -& ,  + , nm 1 i  n  < O  gOx99 iˆ< Ci x O>9 *" * *& -  $ O9  &H!;&  +% * FC<  C1  . . .  C n 1 G ‡ FCn  C n‡1  . . .  C2 n 1 G ‡


‡ FCO m

19 n 

COm

19 n‡1 

. . .  C mn 1 G

  *& On n r9 $$& -+-- -& %&&& >+ gOx9 O-9 " + " , "&%* 2 &  * -&P 8.30  %!& (6 "& &&  NN  gOx9II --! ,  monic $+  gOx9
@$ *"  ,+ * --! ,   - $+  gOx9 ˆ g < ‡ g 1 x ‡


‡ gr x r  "* gr 6ˆ & St Ox9 ˆ

t


aj x r‡ t

j

 gOx9:

jˆ
9 , n && *& $& +! ,!  $ O9 J  $+-*&-# @ , *& -+-- -& , &%* n %&&& >+ gOx9 O-9 & * -& *;& + " , "&%* 2P ,  &@$ "*+  ,  &@$-+   !-* " O9 "  +& - *,&%& &-& , * -&

22.

Cyclic codes

8.33 &% *&& ,,&& *,&%& &-& , *& O17 :9 >+ -+-- -& "* gOx9 ˆ x ( ‡ x : ‡ x ) ‡ x . ‡ 1 8.34 * $>& --& *& O61 2)9 -+--  % -& "* gOx9 ˆ x 7 ‡ x 2 ‡ 1 O9   +& - O& , *& , FAjI 7 G9 $+-*&-# @ , * -& O>9 &%  &-% --! , * -& O-9 & +! &-& , $ O>9  &-& *& &-&;& " R ˆ F111111111111&  + 1  *&& (6 O9 *" * n ˆ 2 k 1 O>9 &->&  &,J-& *,&%& &-& , C k 

Problems

8.42 8.43 8.44 8.45 8.46 8.47 8.48 8.49 8.50 8.51

8.52

227

O-9 *" * *& & , 4& -&" , C  &@-+ &H!  *& & , -+-- *, , *& J " , G1  !& * &! , C. !% *& $ ;& $+  x . ‡ x ‡ 1 , >& (6 O9  *& "&%* &! & , C k    , ! , *& ! >& , ordinary O&  -+--9 >! , &%* b ;&  "&& $*>& O-, *&& ((9 , 1 < b < n &&4& *&& ((   q&& $*>&
@& *&& ((  -;& *& -& b . On ‡ 19=2 &&4& *&& (3   -+   q&& $*>& &&4& *&& (1<   -+  -;& *& -& ,  q&& $*>& /*+  *& -& b ˆ <  -;&&  *&& (3P /*+  +! !$$& *& " ;& , *& >  >! & -& NN%II  NN"&#IIP

& *& >  >! O-+  *&& (39 &;& %* , *& >+ &$& -&P  *& &%& >! &;& %* , *& >  -&P   & >!&-&-% $$-   !,J-&  -&- >! * --! +  -& $&- - *&   $ &@ $& , *  "*  -& phased burst-error correction  $*& >!& -&- *& >-# &%* n   !$& , b  >! , &%* b + --! +  - "*-* & !$& , b -& -$>& , -&-%  $*& >! , &%* b  -&  phased >!b&-&-% -&  &@ $&  $*& >!6&-&-% -& , &%* 12 ! >& >&  -&- + >! , &%* 6 "*-* --!  - & , -&-% t $*& >! , &%* b *& r > 2tb &  O2+ & -&  "*-* *& $+-*&-# & &->& >+ *& ,"% +D

 C< C. C( C12 C1)  C1: C1 C7 C3 C16     C1. C1( C2 C) C1< : C11 C17 C13 C6 C:   %;& -&" FC<  C1  . . .  C13 G *& - $& --!$+% *& !$$& &, 6 3 . $ , *& + -+ *& ,   *& - $& , % *& %* -!   > " , *& + & $+ $ *& $+ -*&-# !& & * *& ! , *& - $&  &-* "  -!  , *& +  4& *! , &@ $& C< ‡ C. ‡ C( ‡ C12 ‡ C1) ˆ < OJ " ! 9  C. ‡ C1 ‡ C1( ‡ C17 ˆ < O&- -!  ! 9 O9 *" * * -& - -&-  $*& >! , &%* . O>9 *" * *& -&$% 6 3 . + -&  not -$>& , -&-%  $*& >! , &%* 6 O-9 &&4&  *" *  b 3 Ob ‡ 19 + -& - -&-  $*& >! , &%* b ,  + , b ‡ 1   $ & ! >&

22)

Cyclic codes

8.53 O! -&9 O9 & gOx9 ˆ Ox b 19 pb Ox9 "*&& pb Ox9   &!->& $+  , &%&& b  $& n1  *" * gOx9 %&&&  On n 2b9 -+--& "* n ˆ - On1  b9 "*-*  -$>& , -&-%  b-phased bursts , &%* b O&& >& (719 O>9 & *& &H!&-& , Onj On 2b9 j9 -+-- -& >& >+  &&;% *& -&  $ O9  & bj && *& -&$% O+ & !$*&9 >!&-&-% -$>+ *" *  j!1 bj =2bj ˆ 12  * *&& -&+ $-+ && *& &%& >! 8.54 &  >+ -+-- b ˆ 6 >!&-&-% -& , &%* 61 O9 & *& >   *& &%& >!  & & *&  ! &&& &!-+ O>9  +! *# *&&  !-*  -& "* *& &!-+ $&-&  $ O9P
@$ ,!+ 8.55 *" * , gOx9  *& %&& $+  ,  On k9 -+-- -&  , Ox ‡ 19    ; , gOx9 *& g9Ox9 ˆ Ox ‡ 19 gOx9  *& %&& $+  ,  On k 19 -& "*-* &H! *& % -& !  " ,  "&%* 8.56 &-& *& ,"% + -&" , *& O: 69 >  -& ,
@ $& (1)D O9 F+ *& '& & &&  *&& O&& > 369 &-& *& -!  , B9 & & *& , B - >& &+ &$&&   *& -& & -&-  & $& , "&%* < t  ;&,+ *& >! k > n mt  *&  & >&;& * *& % $+-*&-# @ H ;&"&   @ "* && , GFO29 *& * GFO2 m 9 *  & mt 3 n

 >+ *& &! , &- :1 * & * *& ! -& *  & < mt   *& -& &, *  & > n mt h

*& -& &->&  *&& 31 & -& BCH codes  * , *& ;& & +*!*!  -H!&%*&  *&& -& &  $   !-* >&-!& , *&& 31 &, O*& -& - *;& *%*& &  %&  ! -&9 >! *& >&-!& *&& & &,J-& &-%  &$&-+ &-% %*  , *&   *&

26.

BCH, Reed±Solomon, and related codes

&@ &- "& " && * , "& -*& &@-+ *& %* &% Oá<  á1  . . .  á n 1 9  -& %-+ >&- & -+-- -&   >+ *& &! , *$& ( *& &-% ! -+ >&- &  $&  + * NN-+--II ;&" ,  -& " " !  &J& ! & & , *& -&I  & *&  &- 36B37 "& " ,!+ &->& & ;& , &&# $I , ! &-% %* ,  -&

9.2 BCH codes as cyclic codes &- *& &J ,  t&-&-%  -& , &%* n ˆ n 1 j 2 m 1D C ˆ OC<  . . .  C n 1 9   -&" ,, iˆ< C i á i ˆ < , j ˆ 1 6 . . .  2t 1 O&H!;&+ , j ˆ 1 2 6 . . .  2t9 "*&& Oá<  á1  . . .  á n 1 9    , n - 4& && & , GFO2 m 9 , *&   -*& $$&+ *& -& >&- &  -+-- -&  *&&>+ *&  *&  $& & -*&+ ;>& , -+-- -& *&& NN-+--II  & *& , *& , O1 á . . .  á n 1 9 "*&& n   ; , 2 m 1  á   && & , GFO2 m 9 , & n /* &$&-  !-*   *& &J >&- &D C ˆ OC<  C1  . . .  C n 1 9   -&" ,, n 1


C i á ij ˆ &- &  cyclic code  *& && , *$& (  && * *   & COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1 >& *& %&&% ,!- , *& -&" CL *& O3:9 >&- & COá j 9 ˆ + *&& (1 C R Ox9 ˆ xCOx9 Ox n 19 "*-* & * R C Ox9 ˆ xCOx9 ‡ MOx9Ox n 19 ,  & $+  MOx9 *! , j ˆ 1 2 . . .  2t C R Oá j 9 ˆ á j COá j 9 ‡ MOá j 9Oá jn

19:

! COá j 9 ˆ < >+ O3(9  á jn 1 ˆ < -& á n ˆ 1  ," * C R Oá j 9 ˆ < , j ˆ 1 2 . . .  2t  * C R    *&  -& &J& >+ O3:9 "*-* & * *& -&  -+--  " ," , *&& (6 * &;&+  -&  -*-&4& >+

9.2 BCH codes as cyclic codes

267

 %&& $+  gOx9 ! *" - "& - $!& gOx9P --%  *& &J gOx9  *& & &%&& $+   *& -& & *& & &%&& $+  ,+% gOá9 ˆ gOá6 9 ˆ


ˆ gOá2 t 1 9 ˆ & A ˆ fá á6  . . .  á2 t 1 g , GFO2 m 9 &-& , A  &J&  >& *& & ,  GFO29-?!%& , && &  A & i A ˆ fâ2 D â 2 A i > &- & V^ j ˆ V Oá j 9

O3:179

 Vi ˆ

1 ^ V Oá i 9: n

O3:1)9

*&& & + &&%  !&,! &*$ >&"&& *&  &   ,&H!&-+  -& ,  %;& ;&- & , *&  *  NN$*& *,II  *&  &   -&$   NN & *,II  *& ,&H!&-+    *& ,"% &&  "& !$+ *& i* - $& , V >+ á ìi & , "& &J&  &" ;&- Vì  Vì ˆ OV<  V1 á ì  . . .  V n 1 á ìO n

19

9

O3:1:9

*&    ^ ì ˆ O V^ì  V^ì‡1  . . .  V^ì‡ n 1 9 V

O3:1(9

"*&&  O31(9 *& !>-$ & #&  n /& &;& *& $, , O31(9  >& 31+ O3129 W i ˆ 1n W ^ Ox9 ˆ W ^< ‡ W ^1 x ‡


‡ W ^ n m 1 x n m 1  -& W ^ Ox9   4& $+ W   , &%&& < n m 1  ," * W i ˆ < , at most n m 1

9.3 Decoding BCH codes, Part one: the key equation

263

;!& , i   W i 6ˆ < , at least m ‡ 1 ;!& , i *! "OV9 ˆ "OW9 > m ‡ 1 h /& &   &+  !-& *& #&+ &H! >! "& &&  ,&" & &J /* *& ;&- V J@& "& &J&  support set I  ,"D I ˆ fi D < < i < n

1  V i 6ˆ & 31(9 Corollary 1 For each i 2 I, we have Vi ˆ

ái

ùV Oá i 9 : ó V9 Oá i 9

O3:279

Proof , "& ,,&&& *& #&+ &H! O3269 "& %& ó V Ox9 V^9Ox9 ‡ ó V9 Ox9 V^Ox9 ˆ ùV Ox9O nx n 1 9 ‡ ù9V Ox9O1

x n 9:

O3:2)9

& * , x ˆ á i "* i 2 I , O32& &-;&& , ó V Ox9  ùV Ox9 *& &@ -+ + * , *& J ,&" ,&H!&-+  -& , V & #" *& & - >& &-;&& , ó V Ox9 & ;   $& &-!  *& & & , *& -+ "& !$$& * *& -&,J-& , ó V Ox9 & %;& >+

9.3 Decoding BCH codes, Part one: the key equation

2.1

ó V Ox9 ˆ 1 ‡ ó 1 x ‡


‡ ó d x d :

Corollary 2 For all indices j, we have d


V^ j ˆ

ó i V^ j i 

O3:2(9

iˆ1

where all subscripts are to be interpreted mod n Proof *& #&+ &H!  $& * ó V Ox9 V^Ox9 
& #&  n  "& *;& &J& ó < ˆ 1 ! " &H! O36 , aOx9  bOx9 &H!  dOx9 &  &H! , *& , uOx9aOx9 ‡ vOx9bOx9 ˆ dOx9:

O3:6(9

*& %* ;;& ,! &H!&-& , $+ D Ou i Ox99 Ov i Ox99 Ori Ox99 Oq i Ox99 *&  - & u 1 Ox9 ˆ 1

v 1 Ox9 ˆ & *& quotient  remainder &$&-;&+ "*& ri 2 Ox9  ;& >+ ri 1 Ox9D ri 2 Ox9 ˆ q i Ox9ri 1 Ox9 ‡ ri Ox9

&% ri , &% ri 1 :

O3:.+ u i Ox9 ˆ u i 2 Ox9

q i Ox9u i 1 Ox9

O3:.19

v i Ox9 ˆ v i 2 Ox9

q i Ox9v i 1 Ox9:

O3:.29

-& *& &%&& , *& & & ri & -+ &-&% *&& " >&   4& &L -  r n Ox9  ! ! * r n Ox9  *& %- , aOx9  bOx9  ,!*& & * *& && &H! &@$&% *& %-   & - > , *& % " $+  O-,
H O36(99 

9.4 Euclid's algorithm for polynomials

2.7

Table 9.2 $&& ,
!-I %* 

  


v i ri 1 v i 1 ri ˆ O 19 i a u i ri 1 u i 1 ri ˆ O 19 i‡1 b u i v i 1 u i 1 v i ˆ O 19 i‡1 u i a ‡ v i b ˆ ri &%Ou i 9 ‡ &%Ori 19 ˆ &%Ob9 &%Ov i 9 ‡ &%Ori 19 ˆ &%Oa9

+ !-  iL && > 313O9 Example 9.3 & F ˆ GFO29 aOx9 ˆ x (  bOx9 ˆ x ) ‡ x . ‡ x 2 ‡ x ‡ 1 *& >&*; ,
!-I %*  %;&  >& 36

*& i ˆ . & , >& 36 *" * %-OaOx9 bOx99 ˆ 1 O"*-*  >;! +"+9  "* $&+  , >& 32 +& *& &H! Ox 7 ‡ x . ‡ x 6 ‡ x 2 9aOx9 ‡ Ox : ‡ x ) ‡ x 6 ‡ x ‡ 19bOx9 ˆ 1 * &@ $&  -!& 
@ $& 3. h /& " ,-! ! &  $&+   >& 32 "*-* - >& &"&  v i Ox9bOx9  ri Ox9 O  aOx99: % $&+   *& ,- * &% ri

1

. &% ri  "& %& *& & &

&% v i ‡ &% ri , &% a:

*&  &! , * &- O *&&  O3.79 /& >&% "*  & 

O3:..9 O3:.79

379   # , -;&&  O3..9

Lemma 2 Suppose Euclid's algorithm, as described above, is applied to the two polynomials aOx9 and bOx9 Given two integers ì > < and í > < with ì ‡ í ˆ &% a 1, there exists a unique index j, < < j < n, such that:

2.)

BCH, Reed±Solomon, and related codes

Table 9.3  &@ $& ,
!-I %*  i

ui 1 < 1 2 6 . 7

vi

ri

qi

1 < x( ) . 2 < 1 x ‡x ‡x ‡x‡1 1 x2 ‡ 1 x6 ‡ x ‡ 1 x6 ‡ 1 x7 ‡ x6 ‡ x2 x2 x. ‡ x ‡ 1 x) ‡ x. ‡ x6 ‡ x2 ‡ 1 x‡1 x7 ‡ x. ‡ x6 ‡ x2 x: ‡ x) ‡ x6 ‡ x ‡ 1 1 ) x ‡ x. ‡ x2 ‡ x ‡ 1 x(
+ &H!% &% r j

1

> í ‡ 1

&% r j < í:

O3:.(9 O3:.39

*& >+ $&+  "&  *;& &% vj < ì

O3:7 ì ‡ 1:

O3:719


H! O3.39  O37 ì ‡ 1 > &% v ‡ 1  &% r j 1 > í ‡ 1 ˆ &% r ‡ 1 &-& , *&&   &@ !-* * O37.9  O3779 *  ! >& !H!& " &"& $&+  
H O3729  ,"D u j a ‡ vj b ˆ r j 

O3:7)9

ua ‡ vb ˆ r

O3:7:9

"*&& u   & !$&-J& $+  !$+ O37)9 >+ v  O37:9 >+ vj D u j va ‡ vj vb ˆ r j v O3:7(9 uvj a ‡ vvj b ˆ rvj :

O3:739

%&*& O37(9  O3739  $+ r j v  rvj O  a9 ! >+ O3.:9  O3769 &%Or j v9 ˆ &% r j ‡ &% v < í ‡ ì , &% a  + >+ O3.)9  O3769 &%Orvj 9 ˆ &% r ‡ &% vj < í ‡ ì , &% a  ," * r j v ˆ rvj  * ,- - >& "* O37(9  O3739  $& * u j v ˆ uvj  ! -& $&+  %!&& * u j  vj & &;&+ $ & * & * uOx9 ˆ ëOx9u j Ox9 vOx9 ˆ ëOx9vj Ox9 ,  & $+  ëOx9 *&
H! O37:9 >&- & ëu j a ‡ ëvj b ˆ rL - $% * "*
H O37(9 "& --!& * rOx9 ˆ ëOx9r j Ox9 h

*& &! , *&& 37 " >& !& -+  ! ,*- % -! , &-% %*  ,   &&B  -&  ,-& *&& -! "& " !-& *& %* - $-&!& NNEuclidOaOx9 bOx9 ì í9II De®nition If OaOx9 bOx99 is a pair of nonzero polynomials with &% aOx9 > &% bOx9, and if Oì í9 is a pair of nonnegative integers such that ì ‡ í ˆ &% aOx9 1, EuclidOaOx9 bOx9 ì í9 is the procedure that returns the unique pair of polynomials Ovj Ox9 r j Ox99 with &% vj Ox9 < ì and &% r j Ox9 < í, when Euclid's algorithm is applied to the pair OaOx9 bOx99

2.(

BCH, Reed±Solomon, and related codes

*& ,"% *&&

!

4& *& &! , * &-

Theorem 9.6 Suppose vOx9 and rOx9 are nonzero polynomials satisfying vOx9bOx9  rOx9 O  aOx99

O3:)+ O3)69  O3).9 &% vj Ox9 < &% vOx9 < ì  &% r j Ox9 < &% rOx9 < í h Example 9.4 & aOx9 ˆ x (  bOx9 ˆ x ) ‡ x . ‡ x 2 ‡ x ‡ 1 F ˆ GFO29  
@ $& 36 % >& 32 "& - >!& *& !$! , Euclid , *& &%* $>& $ Oì í9D Oì í9

EuclidOx 8  x 6 ‡ x 4 ‡ x 2 ‡ x ‡ 1 ì í9

O& ó Ox9 ˆ x 2 ‡ 1 ùOx9 ˆ x 6 ‡ x ‡ 1 h

 * $ *& $$- , *&& 3.  *& $>& , ;% *& #&+ &H! ,  -& *! >& $$&  + &;& "& $&  !  *& &@ &- 9.5 Decoding BCH codes, Part two: the algorithms & ! &-$!& *&  &-% $>&  "*-* "& >& & $+  *& & , &- 36 /& & %;&  &-&;& ;&- R ˆ OR<  R1  . . .  Rn 1 9 "*-*   + ;& ,  !#" -&" C ˆ OC<  C1  . . .  C n 1 9 , *& t&-&-%  -& &J& >+ O3:9 & R ˆ C ‡ E "*&& E  *& & $& ! %   &-;& C , R *& J &$  *& &-% $-&   - $!& *& syndrome polynomial SOx9 &J& >+ n

SOx9 ˆ S1 ‡ S2 x ‡


‡ S2 t x 2 t 1 

O3:)79

1 ij iˆ< Ri á 

, j ˆ 1 2 . . .  2t /& "  *& & , &- 36 "*&& S j ˆ * SOx9 ˆ V^Ox9  x 2 t  "*&& V^Ox9  *& %&&% ,!- , *& !& , , *& ;&- V &J&  O36)9  * *& #&+ &H! O36:9 >&- & ó Ox9SOx9  ùOx9 O  x 2 t 9

O3:))9

"*&& ó Ox9  *& &- $+   ùOx9  *& &&;! $+ 

*& &@ &$  *& &-% $-&   !&
!-I %*    $-! *& $-&!& EuclidOaOx9 bOx9 ì í9 &J&  &- 3.  ;& *& #&+ &H! , ó Ox9  ùOx9 *  $>& -& , *& ! >& , & * -!+ --!&  < t *& >+ O32+ &  1 %-Oó Ox9 ùOx99 ˆ 1 *! *& *+$*&& , *&& 37 & & "* aOx9 ˆ x 2 t  bOx9 ˆ SOx9 vOx9 ˆ ó Ox9 rOx9 ˆ ùOx9 ì ˆ t í ˆ t 1  * , *& $-&!& EuclidOx 2 t  SOx9 t t 19  -& 

27
& && & >+ *& ,- * ó O& 31 / ``Frequency-Domain'' BCH Decoding Algorithm / { for (j ˆ 1 to 2t)  Sj ˆ in ˆ1< Ri á ij L SOx9 ˆ S1 ‡ S2 x ‡


‡ S2 t x 2 t 1 L if (SOx9 ˆˆ&"&& *& " -& , -&  *  -& & *& &;& !+  *&   , ,  characters *& * bits  * &- "& " &J&  !+ &&B  -&

*! & F >& + J& "*-* -  && & á , & n) , r   J@& &%& >&"&& 1  n *& & ,  ;&- C ˆ OC<  C1  . . .  C n 1 9 "* - $&  F !-* * n 1


C i á ij ˆ &%%  *& -& & -&  codewords *& ,"% *&& %;& *& >- ,- >!  -& Theorem 9.7 The code de®ned by (9.67) is an On n r9 cyclic code over F  with generator polynomial gOx9 ˆ rjˆ1 Ox á j 9, and minimum distance d  ˆ r ‡ 1 Proof & C ˆ OC<  C1  . . .  C n 1 9 >&  >+ ;&- , &%* n ;& F  & COx9 ˆ C< ‡ C1 x ‡


‡ C n 1 x n 1 >& *& -&$% %&&% ,!- *& O3):9 + * C   -&" ,  + , COá j 9 ˆ + *&& (6O>9 *& -&   On n r9 -+-- -& "* %&& $+  gOx9  $;& *& & >! d   >&;& * O3):9 + * , C^ ˆ O C^<  C^1  . . .  C^ n 1 9  *&  ,  -&" *& C^1 ˆ C^2 ˆ


ˆ C^ r ˆ < O-,
H O31199

*! >+ *&  %! & O *&& 369 *& "&%* , + 4& -&"  > r ‡ 1  *& *& * *& %&& $+  gOx9 ˆ x r ‡ g r 1 x r 1 ‡


‡ g <  "*& ;&"&   -&" * "&%* < r ‡ 1 *! d  ˆ r ‡ 1  && h Example 9.6 & *& O: 69 &&B  -& ;& GFO(9 , á  

9.6 Reed±Solomon codes

277

$ ;&   GFO(9 ,+% á6 ˆ á ‡ 1 *& %&& $+  , *& -&  gOx9 ˆ Ox á9Ox á2 9Ox á6 9Ox á. 9 ˆ x . ‡ á6 x 6 ‡ x 2 ‡ áx ‡ á6  , gOx9  ;&"&   -&"   Fá6  á 1 á6  1 ! -!-;&L *"&;& ,  -& *&&   &,J-& interpolation algorithm "*-*  -&+ &&  *& %%& &$ , ! , ! &- +

*& &@ *&& $& * ! Theorem 9.10 Consider the On de®ned by (9.67), where k ˆ n between the codewords C ˆ OC<  all polynomials POx9 ˆ P< ‡ P1 x over F, given by

k9 Reed±Solomon code over the ®eld F r There is a one-to-one correspondence C1  . . .  C n 1 9 of this code, and the set of ‡


‡ Pk 1 x k 1 of degree k 1 or less

Ci ˆ á

iO r‡19

Thus apart from the scaling factors á codeword are the values of a certain Ok

POá i 9:

iO r‡19

, the components of a given RS 19st-degree polynomial

Proof & C ˆ FC1  . . .  C n 1 G >&  J@& -&" /& &J&  NN"&II ;& , C -& D ˆ FD1  . . .  D n 1 G >+ Di ˆ á

iO r‡19

Ci

, i ˆ + O3):9 "& *;& C^1 ˆ C^2 ˆ


ˆ C^ r ˆ + O31)9 "& *;& Di ˆ POá i 9 , i ˆ % * "* O3)39 "& > C i ˆ á iO r‡19 POá i 9 "*-*  "* "& "& h

*& ,"% &@ $& !& *&&

31& 
@ $& 3) -- %  *&& 33 *&&   !H!& -&" C !-* * C1 ˆ á6  C. ˆ á  C) ˆ á.  & ! -!- * -&" /& >&% >+ >&;% * , I ˆ f1 . )g *&& 33 %!&&  &&-& *& &@&-& ,  6 3 : %&& @ , C , *& , G1.)

.< 1  1 ˆ / < 
+ J& GFO29 O>& 32:9Q*& *&& >- - $& OK$K$ &  !$&9 " +$-+  >& NN,,*&*&,II &  *!%* *& &% , *&& - $& ;& *&  $ J& GFO2 m 9    $  &&% $-   >&+ *& -$& , * >#  "& " --!& ! -! ,  &-& "* %!& 36 "*-* *"  +& *,&%& &-& , *& O: 69  -& ;& GFO(9 "* gOx9 ˆ x . ‡ á6 x 6 ‡ x 2 ‡ áx ‡ á6 O&&
@ $& 3)  3:9 /& ! "  *& $>& , decoding  -& "*-* ! !  >& H!&    *& &-% ,  -&  ;&" , *&  + , *& &J O- $& O3:99 "* O3):99 * *!  >& !$% & ! >&% >+ , + % *&  &-% $>&  /& & %;&  &-&;& ;&- R ˆ OR<  R1  . . .  Rn 1 9 "*-*   + ;& , 

9.6 Reed±Solomon codes

273

first 3 ticks closed last 4 ticks open

a3

a

1

a3

first 3 ticks down last 4 ticks up

Out In

Figure 9.3 +& - *,&%& &-& , *& O: 69  -& ;& GFO(9 "* gOx9 ˆ x . ‡ á6 x 6 ‡ x 2 ‡ áx ‡ á6 

!#" -&" C ˆ OC<  C1  . . .  C n 1 9 , *& On k9  -& &J& >+ O3):9 & R ˆ C ‡ E "*&& E  *& & $& -& >+ *&& 3: d  ˆ r ‡ 1 "& - *$&  -&-+ &,+ C !& "OE9 < br=2c   , *& & , *& -! "& * & t ˆ br=2c  ! & * "OE9 < t

*& J &$  *& &-% $-&   - $!& *& syndrome polynomial SOx9 ˆ S1 ‡ S2 x ‡


‡ S r x r 1 

O3::19

n 1 "*&& S j ˆ iˆ< Ri á ij  , j ˆ 1 2 . . .  r + *& &! , &- 36 , "& &J& *& NN"& & $&II >+ V ˆ OE<  E1 á E2 á2  . . .  E n 1 á n 1 9 *& SOx9 ˆ V^Ox9  x r   *& #&+ &H! O3269 &!-& >&- &

 x r

ó Ox9SOx9  ùOx9 O  x r 9 "*&& ó Ox9  *& - $+   ùOx9  *& &;! $+  , *& ;&- V

 * $ *& &-% $>&    &@-+ *&  &   " ,  -&  &->&  &- 37  $-! , *& $-&!& EuclidOx r  SOx9 t t 19  -&  " &! *& $ , $+  OvOx9 rOx99 "*&& vOx9 ˆ ëó Ox9  rOx9 ˆ ëùOx9 ,  & 4& -  ë

*& J &$  *& &-% %*   !& ó Ox9  ùOx9  && & *& & $& E ˆ OE<  E1  . . .  E n 1 9  *&-& *& % -&"

2)
&-!&  -& & >+  * E i ˆ <  1 ,  i *! , *&&   &  $ i & E i 6ˆ &- & *& inner channel9   &%  -& ,   * &@ $& *& !& -*&     "* 1) $!L *& &! , * &- !%%& * "& &% *&& $!  !$!  && & , GFO1)9 *& *  ,! & ;&- ;& GFO29  & ! " -& !%  O17 119  -& ;& GFO1)9  &!-& *& &  *& !& -*&  !&  %!& 3:

*&  &-&  %!& 3: #& 11 ,  + > á ˆ Oá<  . . .  á1< 9 , GFO1)9 O"*-* & &+ .. > , *& % !-&9  $!-&   -&" C ˆ OC<  C1  . . .  C1. 9 *& !& -*& *& %>& C    &-&;&  R ˆ OR<  . . .  R1. 9 *&  &-& *& $!-&  & & ⠈ Oâ<  . . .  â1< 9 , á "*-* " >& &H!  á , *& !& -*& * -!&  & * " + > & *! , E Oˆ + , &-& &  %!& 3) *& 17 $>>+ , &-& &  %!& 3:   & * kˆ6 17 k E917 k ˆ & H!$+  ;& GFO1)9

*& $&-&% &@ $& !& >* *& %&& & , --&  *& & "*+  -& &  !&,!  --&& +&  Any -& - !- +& - >& &%&   + !& -*&   %!& 3( "&;& , * $ , ;&"  >& !&,! "& ! >& >&  &%  !& -& -$>& , -&-%  , *& & -!& >+ *& !& -*& "*-*  #&+  >&  ;&+ - $&@ >& -&  & & -!& >+ & &-& ,!& /*& *& & &-& , *  "*& Ov1  . . .  v k 9 6ˆ Ou1  . . .  u k 9  %!& 3( *& + > v1  . . .  v k !!+ >& $--+  && >-&  u1  . . .  u k  * & * &  *& !& -*& &  --!  >! , &%* k  "& *;& &+ && *  -& & "& !&  >!& -&- *  *& & "*+  -& &  "&$& !&  !& -&  --&& +&  9.7 Decoding when erasures are present /& *;& && *    -& - -&- !$& &  * &- "& " && * *&+ -  -&- *& - , -*& K" -& erasures  &!&   $+  -*& + > "*-*  &-&;& &%>+  &@ $& -& *&
%* " BLOCK , *& * &&  -*%& , O  A "& %& BLACKL *   error  *& * $ "&;& , *&  & " !,,&  erasure  *& * $ *& &!  BLCK "*&& NNII  *& erasure symbol  $--& &!& & H!& -  *&+ - >& &@$&-&  --! "*& *& -*& & >&- &

9.7 Decoding when erasures are present

2):

!!!+ &;&& ,  *  &  &@ $& , +! & +%  #  *& $   "K+% ?& $& ;&*& +! -;&  erased ! && "  #& "* +! & +%  +L *&+ "  $+  >& >&  !& +!  * &- "& " &  &*% >! &!& -&- /& " && *  $-$&  &!&  + *,  *  -&-   & O *&& 3119L  "& " && *"  ,+ *&    &-% %*   &  -&- >* &!&  &

 &  -*& "*-* - $!-& &!&  "&  & "&  $+ &%& *& !&+% + > & F  F ˆ F [ fg "*&& NNII   >;&  $&- &!& + > *& + "& transmitted + > & *& && & , F >! + && &  F - >& received *&  *&&- &! >!  !&! &!&& -&- ," O $& "* *&& :29 Theorem 9.11 Let C be a code over the alphabet F with minimum distance d. Then C is capable of correcting any pattern of e< erasures and e1 errors if e< ‡ 2e1 < d 1 Proof  $;& *& *&&  "& J !-& *& extended Hamming distance d H Ox y9 >&"&& + >  FD & ' < , x ˆ y d H Ox y9 ˆ 1 , x 6ˆ y  &*& x  y  NN  II (1 , x 6ˆ y  & , x  y  NN  II: 2

*! , &@ $& , F ˆ f& *& O: 69 -+-- -& , "


@ $& (2 "* -&

C< ˆ + *& %&   $ &-&-% -& O *&&    O11 )9 + -& ;& GFO69L && > 3).B3):9 /& >&% "*  4% ! >&*&&- ,-  *& 26 & ;&- $-& ;& GFO29 "*-* "& - V26    % $*&& , ! 6 -       26 26 26 ˆ 2& #&  *& -&"   O26 129 >+ -+-- -&M  -%*&&- &  *& "& &&  -!-  >+ -+-O26 129 *&&&-&-% -& & & "* d  > : /& >& *& -!-  -& $$&& , *& J& GFO211 9 -& 211 1 ˆ 2! "& "  && * &@$- ,-4  *& & , * &- /& - " &J& *& + -& De®nition The O26 129 Golay code is the binary cyclic code whose generator polynomial is gOx9, as de®ned in (9.85) or (9.88). "  OM9 "& *;&    *" * *& -&I

*& J &$  * &-  *& &+

 !

"&%*  > :

Lemma 3 Each nonzero Golay codeword has weight > 7 Proof  ;&" , *& !-!& , *& & B , 4& , gOx9 O-,
H O3(799 "& && * , &;&+ -&" C COâ9 ˆ COâ2 9 ˆ COâ6 9 ˆ COâ. 9 ˆ : Lemma 5 If C is a Golay codeword of even weight w, then w  < O  .9 Proof & *& %&&% ,!- , C >& && >+ COx9 & COx9 ˆ x e1 ‡ x e2 ‡


‡ x e w 

O3:319

"*&& < < e1 , e2 ,


, e w < 22 -& C >&%  *& + -& COâ9 ˆ &-!& w  &;&   - $! #& $-&  GFO299 *! ~  COx9 COx9

22


ìb x b

O  x 26

19

bˆ1

"*&& ì b  *& ! >& , && $ Oi j9 "* e i , ! O33)9 &-* ì b  &;&D ìb  < " , e i

O  29

e j  b *&  e j

+ -& *&& & wOw

b ˆ 1 2 . . .  22:

e i  26

ì b ˆ ì26 b 

O3:3(9

b O  269 *!

b ˆ 1 2 . . .  11:

19 &   *& ! 22


e j  b O  269 +

ì b ˆ wOw

O3:339

 *& %* & , O33:9

19:

O3:1+ appending an overall parity check & >+ &J%  2.* - $& C26  ,"D C26 ˆ C< ‡ C1 ‡


‡ C22 :3 , &;&+ + -&"  &@&&  * "+ *& &!% -&   >+ & O>!  %& -+--9 O2. 129 -& "*-*  -& *& O2. 129 extended Golay code     $& & *&  $;& *& ,"% *&&  O&& >& 3739

Theorem 9.14 The number of codewords of weight i in the O2. 129 extended Golay code is < unless i ˆ + O&& >&  373 3)29 /& --!& "*  & & # >! *&  $& & , *& + -& -& *& O26 129 -&  -+--   -& * "& -! &%  11 %& *,&%& &-& ,  O&& &- (2   >& 3)- &-% %*    &+L "& -! &+ ,+ *&  &-% %*  %!& 31  32  -&- &;&+ $& , "  ,&"& & >! *& -&  NN--&+II -$>& , -&-% *&&M !&+ *"&;& *& -&    &!%*  * *& + & NN>& #!$II %* -!&  &- :2  !!+ $-- O&& > 3)69

2(2

BCH, Reed±Solomon, and related codes Problems

9.1 /& "  &- 31 * *& ,!- fOV9 ˆ V6 #& *& @ H 2 ,
H O329  *& $+-*&-# @ ,  "&-&-% -& ;&%& "*&*&   *& ,"% -& fI "#D O9 fOV9 ˆ TV "*&& T   & ,  , V m  O>9 fOV9 ˆ a< ‡ a1 V ‡ a2 V2  "*&& V  &%&   && & , GFO29 O-9 fOV9 ˆ V 1  "*&& V 2 GFO2 m 9 9.2 !$$& F   J& J& "* q && & O9 , a   >+ && & , F &J& *& Oq 19&%&& $+  f a Ox9 ˆ Ox a9 q 1 1  *& ;!& , f a Ox9 , &-* , *& q && & , F O>9 % *& &! , $ O9  *&"& *" * &;&+ ,!- f D F ! F - >& &$&&&   $+  , &%&&   q 1 9.3 O * $>& %;&  %&&4 , *& '& & &&  *&&  "*-*  &&&  *& $, , *&& 319 & Pi Ox9 >&  - $+  , &%&& i , i ˆ & - && & *" * . 0 P< Ox1 9 . . . P< Ox n 9 6 7 P1 Ox n 9 7  n 1  n 6 P1 Ox1 9 . . . 6 7 &6 Ox j xi 9: 7ˆ . . 6 7 iˆ1 jˆi‡1 .. .. / 1 Pn 1 Ox1 9

...

Pn 1 Ox n 9

F HintD , xi ˆ x j  *& &, & ;*&G *& '& & &&  *&&  *& $&- -& Pi Ox9 ˆ x i  9.4 &&   $&!-& % ,  %* , - $!% *&  & , *& t&-&-%  -& , &%* 2 m 1D {

}

S ˆ f1 6 . . .  2t 1g; k ˆ 2 m 1; while (S is not empty) { u< ˆ least element in SL u ˆ u< ; do { delete u from SL k ˆ k 1; u ˆ 2u  2 m 1; } while (u 6ˆ u< 9 }

O9 *" * "*& *& %* & & *& &%& k  &H!  *&  & , *& t&-&-%  -& , &%* 2 m 1

Problems

9.5

9.6

9.7

9.8 9.9

9.10 9.11

O>9 & *& %*  - $!& *&  & , *& t&-&-%  -& , &%* )6 , 1 < t < 61 O9 ;& * *&  & , *& "&-&-%  -& , &%* n ˆ 2 m 1  n 2m ,  m > 6 O>9 & %&&+ *" * , + J@& t > 1 *&  & , *& t& -&-%  -& , &%* n ˆ 2 m 1  n mt ,  !,J-&+ %& m O-9 /*  *&  & ;!& , m< !-* * *& three&-&-%  -& , &%* n ˆ 2 m 1 *  & n 6m ,  n > m< P O9  &-* t  *& %& 1 < t < : - $!& *&  & , *& t& -&-%  -& , &%* 17 O>9  &-* , *& -&  $ O9 --!& *& %&& $+  ! %  $ ;&  á  GFO1)9 * J& *& &H! á. ˆ á ‡ 1 O,
@ $& 319 
@ $& 31 "& - $!& *& %&& $+  , *& *&&& -&-%  -& , &%* 17 !& *& ! $ * *& $ ;&  á , GFO1)9 J& á. ˆ á ‡ 1 , "& * -*&  $ ;&  ,+% á. ˆ á6 ‡ 1 & "* "! *& %&& $+  *;& !& !  >&P ;& *& ;&&  , !
H O3129 & *& J& GFO:9  &$&&& >+ *& & , &%& f9 % 6  *& &&& $ ;& )*  , !+ J *&  , *& ;&- V1 ˆ O1 2 6 . 7 )9  V2 ˆ O1 6 2 ) . 79 O-9 /*+  +! !$$& *&  , V2   !-* NN $&II * * , V1 P ;& * *&  , *& $*&*,& ;&- Vì  &H! O31:9  %;& >+ *& , ! O31(9 O&&4&  -&9 & gOx9 >&  $+  "* -&,J-&  GFOq9 "*-* ;& x n 1 !*& ! & * á   n*  , !+   & &@& J& , GFOq9  * gOá i 9 ˆ & *& -+-- -& "* %&& $+  gOx9 *" * *&  ! -& , C  > d F HintD & *&  %! &G  *& -;&& , *& NN %! &II O *&& 369 !&P *   , V   ;&- , "&%* w &  &-&+ ," * *&  V^ ! *;& w 1  & -&-!;& 4& - $&P , +! "&  yes %;&  $, , +! "&  no %;&  &@$- -!&&@ $& ;& * %-O V^Ox9 1 x n 9 ˆ i=2 I O1 á i x9 "*&& V^Ox9   &J&  O31.9  I   &J&  O3139 ^ & #" *& & - >& *" * , + d -&-!;& - $& , V --!& $;& "&  #" ó V Ox9 O-, + 329

*& J& GFO1)9  &$&&&  >& 31 -   $ ;& 7*  , !+  &+ á6 "*-* , *& & & , * $>& "& * && >+ â & V ˆ O1 â.  â7  &  $"& && ;&  J& F , ì  í & &%;& &%&  Oì í9 Pade approximation  AOx9    ,!- pOx9=qOx9 !-* * O9

qOx9AOx9  pOx9 O  x ì‡í‡1 9

O>9

&% qOx9 < ì &% pOx9 < í:

% *&& 37 *" * , &-* Oì í9 *&&  O$ ,  - ,-9  !H!& $ O p< Ox9 q< Ox99 !-* * , O9  O>9  * *& pOx9 ˆ ë p< Ox9  qOx9 ˆ ëq< Ox9 ,  4& - ë *& $ O p< Ox9 q< Ox99  -& the Oì í9 Pade approximant  AOx9 &,&%  >& 36 - $!& *& &C $$@   AOx9 ˆ 1 ‡ x ‡ x 2 ‡ x . ‡ x ) ‡


;& GFO29 "* ì ‡ í ˆ : 9.22 /* *&  & &!$ 
@ $& 37 &-& *& ,"% + -&" , *& O17 79 *&&&-&-%  -&D

Problems

2(7

R ˆ FR<  . . .  R1. G ˆ F11& , Pr  * "+   & -& ;& Fq m "* &%* n  & r ‡ 1   ! -& n r 9.35 *& &!$ >&% *&  &   > 36. -&$%  &-* f 2 Pr & C ˆ OC<  C9<  C1  C91  . . .  C n 1  C9n 1 9 2 GFOq m 92 n >& &J& >+ C i ˆ f Oá i 9 C9i ˆ á i f Oá i 9 *" * *& & , ;&- *! >&   & -& ;& GFOq m 9 "* &%* 2n   & r ‡ 1 !-* * "* &-* 4& -&" *&& &  & n r - $ OC i  C9i 9 9.36 & ö D GFOq m 9 ! GFOq9 m >&  && & $$% , GFOq m 9 

Problems

2(:

GFOq9 m  #& *& -& ;& GFOq m 9 &J&  > 367  #&    -& ;& GFOq9 >+ $$% *& -&" C  OöOC< 9 öOC9< 9 . . .  öOC n 1 9 öOC9n 1 99 *" * *& &!% GFOq9 & -& * &%* 2mn   & mOr ‡ 19  * "* &-* 4& -&"  % *& n !>;&- OöOC i 9 öOC9i 99 *&& &  & n r - & 9.37 O * $>&   ! , $-& &$& $$&-&9 & fx1  . . .  x M g >&  & , M - ;&- , VnOF 2 9  & w i ˆ w H Oxi 9 && *&  % "&%* , xi  & p ˆ Ow1 ‡


‡ w M 9=nM ;& * % M < nH 2 O p9 "*&& H 2  *& >+ &$+ ,!- F HintD & X ˆ OX 1  X 2  . . .  X n 9 >&   ;&- "*-*  &H!+ #&+  ! & + , *& ;!& xi   & p j && *& ,- , *& M ;&- * *;&  1  *& j* -& " ;&,+ *& ,"% % , &H!&  &H!&D % M ˆ HOX9
 367 $&-4&  q ˆ 2 n ˆ 2 m   Oá<  á1 . . .  á n 1 9 + &% , *& && & , GFO2 m 9 & r=2 m ˆ r  *" * *& &!% -& *;& O9 &%* ˆ m2 m‡1  O9 & ˆ 12 r ‡ 21m  $ % O9 d n > O1 r9 H 2 1 12 ‡ %22O1m r9 

*&& -& & *& Justesen codes && >;& F HintD  $;& O9 !& *& &! , > 36:G 9.39 + *" * , + < < R < 12 *&&   J& &H!&-& , 0!&& -& ;& GFO29 "* &%* n i   & k i    ! -& d i !-* *D 

i!1



i!1



i!1

n i ˆ 1

k i =n i ˆ R

!$ d i =n i > H 2 1 O1=29
O1 ˆ & 9.42   %;& ;!& , d *" + $ , &%;& &%& Oe<  e1 9 & *&& !-* * e< ‡ 2e1 < d 1P 9.43 , m  n & $;& &%& !-* * m ‡ n  &;& *" *

2((

BCH, Reed±Solomon, and related codes 2

3 4 5 m n m‡ n ‡ ˆ : 2 2 2

O&& *& & #  &&+ $&-&&%
H O3(.99 9.44 & *& O: .9  % -& ,
@ $& :6 *& -& * d  ˆ 6   >+ *&& 311   -$>& , -&-% + $& , e< &!&  e1 & $;& e< ‡ 2e1 < 2 &% *   &-& O, $>&9 &-* , *& "  $ O9 O>9  O-9 O9 F1 1 1 < <  9 F<  1 1 1 < 1G O-9 F< 1  1 <  1G O9 , R    + -*& ;&- , &%* : -% &@-+ & &!& "*  *& $>>+ *  " >& !H!&+ &-& >+ *&  &-% %* !-&  *& $, , *&& 311P 9.45 *" * , ó 1 Ox9  *& &- $+   ùOx9  *& & &!& &;! $+  ,  &&!& &-% O&& &H!  O3::9  O3(199 *& %-Oó 1  ù9 ˆ 1 9.46 ;&%& *& $>>+ , &-& & ,  &&B  &-& !& *& ,"% --! -& O9 r &!&  1 & O>9 r 1 &!&  1 & O-9 r ‡ 1 &!&  & 9.47 & *& O17 :9  -& ;& GFO1)9 O%&&& >+  $ ;&  ,+% á. ˆ á ‡ 19  &-& *& ,"% &-&;& " R ˆ Fá16 1  á1< á12 á)  á7 á16  á1 á( á: á2 á3 G: 9.48 % *& !%%&  *& & , &- 3: &-& *& ,"% + ;&- , *& O17 79  -& "* %&& $+  gOx9 ˆ x 1< ‡ x ( ‡ x 7 ‡ x . ‡ x 2 ‡ x 2 ‡ 1 O-,
@ $& 319 R ˆ F1 1  < < <  < < < 1  1 < 1G: O9 & *&  &  - $& O>9 & *& ,&H!&-+  - $& 9.49 &  On k9 & -& "* d  ˆ d , *&& &  & *&& 311 %!&& * *& -&  -$>& , -&-% + $& , !$  d 1 &!& *" * * &! - >&  $;& >+ *"% * *&&   & & & , d &!& * *& -& I -$>& , -&-% 9.50  &- (. "& -&& -& -$>& , -&-% %& >! , &  ! ! *   ;&+ !-* &&  -&- %& >! , erasures ,& % * $>&  +! " %&& * *   O9 , C   On k9 & -& "*-*  -$>& , -&-% + &!& >! , &%* b  & *" * n k > b O, *& &%& >! -+  *&& (19 *" * + On k9 -+-- -&  -$>& , -&-% + &!& >! , &%* n k  & O-9 & *& O: 69 -+-- -& ,
@ $& (2 &- *& ,"%

Problems

9.51

9.52

9.53

9.54

2(3

-&" "*-* *;& !,,&& &!& >! , &%* .D O1<      "* && & x  y *! >% R ˆ O1 x 1 y 1 < 19 O9 & *& ,- * &;&+ -&"  *& O: .9  % -& J& *& &H! HC T ˆ  *&& &H!  *& *&& && & &$&&% *& &!& /& -! *& ;& *&& &H! , *& && & *! -&-% *&& &!& & *&& 311 + %!&& * *& -&  -$>& , -&-% " &!& /*I "% *&&P  * $>&  "& -&  &;& $$-*  -&-% &!&  & "*-* ;;& *& & , NN%!&%II *& ;!& , *& &!& O9 ! & J * *& -&  >+ & *& J& F  *&& 311  GFO29 !$$& "& *;&  -& C "*  ! -& d  * "& *;& &-&;&  + -&" -% e< &!&  e1 & "* e< ‡ 2e1 < d 1 !$$& * "& -*%&  *& && $  & &% 
@ $& 33  * d  ˆ .   >+

*&& 31&  +! & #&  ;&%& *& : . ˆ 67 $>& &!& $& , 4& ,!   && & "*-*  *& & -&->&  $-! $&& J *& number , "&%* ,! &!& $& * & -&->&

* $>& --& *& $>>+ *   + &&-& ;&- , GFOq9 n " >& &-& >+  &-& ,  &&B  -&

23
9 & *& , ! , $ O9  - $!& *&   ,  J@& ;!& , t  q ! 1 , *& $>>+ *   + &&-& " , &%* q 1 " , "*  % -& t  & ,  & -&"   t& -&-%  -& , &%* q 1 ;& GFOq9 O ! & * *& -&I &!-+  r ˆ 2t9 & C >&  On k9 >+ -+-- -& "* %&& $+  gOx9  $+-*&-# $+  hOx9 O9 *" * , hO19 6ˆ 9 *" * , *&&   $ Oè1  è2 9 ,  , hOx9 !-* * è1 è2 ˆ 1 *& &;&+ -&"I "&%*  ;>& >+ ,! F HintD *   %&& 4 , *& &!  &  7G  *& &@ "& $;& * *& O26 129 + -& * d  > : *" *  ,- d  ˆ : , * -&  *  " "+D O9 + &@ % *& %&& $+  gOx9 O>9 + *"% * any O26 129 >+ & -& ! *;& d  < : *" * *&&   O3+ & -& "* d  ˆ 7 &  $&,&- "&-&-% -& , &%* 3& , 12> + & , 1 2  "&%* -&$%  &> & *" * *& ! >& ,  "&%* + & -&$%  "> &  rO3< r9  & $  && & rG

*& O26 129 >+ + -& &J&  &- 3( "*& - >& "* + & >& #!$ &-% * *& $$&+ * &;&+ & $& , "&%* < 6 " >& -&-& O9 &->&  & "* *& &-& "  , *& & $& * "&%* . O>9 /* , *& & $& * "&%* 7P O-9 &&4& *& &! , $ O9  O>9  &-* &%& t  *& %& . < t < 26 "* " *& &-&  , *& & $& * "&%* tP ;& *&& 31.

* $>& --& *& ! >& , -&" , "&%* (  *& O2. 129 &@&& + -& O9 *" * *& ! >& , -&" , "&%* (   4&  *  " "+D O19 + &@ % *& %&& $+  gOx9 , *& % O26 129 + -&L O29 + *"% * any O2. 129 >+ & -& ! *;& d  < ( O>9 ;& 1< * *& -& - &@-+ :73 " , "&%* ( *" * , + !>& fi1  . . .  i7 g , J;& && & , f&% !&  -&-  & $& , "&%* *&&  &9 O9 *" * *& -& - &&-  & $& , "&%* . O>9 ;& * *& "&%* &! & , *& -&  1 ‡ :73x ( ‡ 27:)x 12 ‡ :73x 1) ‡ x 2.  , &-* e  *& %& . < e < 2. - $!& *& ! >& , & $& , "&%* e * *& -& " &&- O-9 " ! & *& &-& + & $  -&- & $& , "&%* two  &  &$& $ O>9 O9 " ! & *& &-& + & $  -&- & $& , "&%* one  &  &$& $ O>9 O&9 + ! & *& &-&  !&   detect-only mode & , *& + &  4& *& &-&;& "  --&$&  -&- >! *&"&   &?&-& &$& $ O>9 9.62  * $>&  "& " >&K+ -& &-& , *& O26 129  O2. 129 + -& O9 &%  *,&%& &-& , *& O26 129 + -& O>9 + ,+% +! &%  $ O9  *&"& - & !$ "*  &% ,  &-& , *& O2. 129 &@&& + -& 9.63 -! *& 4&  - $&@+ ,  + & >& #!$ &-& , *& O2. 129 + -& In Probs. 9.64±9.67 we will investigate the ternary Golay code. Observe that in the vector space GFO611 9 a Hamming sphere of radius 2 contains     11 11 1‡2 ‡. ˆ 2.6 ˆ 67 1 2 vectors. This suggests that it might be possible to perfectly pack GFO611 9 with :23 ˆ 6) spheres of radius 2 The ternary Golay code does this. It is an O11 )9 linear code over GFO69 whose codewords, when taken as sphere centers, produce such a packing. The code is de®ned as follows. Since 67 1 ˆ 11
22, it follows that GFO67 9 contains a primitive 11th root of unity, which we shall call â Since over GFO69 the factorization of x 11 1 is x 11 1 ˆ Ox 19 gOx9 g~Ox9, where gOx9 ˆ x 7 ‡ x . x 6 ‡ x 2 1( and g~Ox9 ˆ x 7 x 6 ‡ x 2 x 1, we may assume that â is a zero of gOx9 The O11 )9 ternary Golay code is then de®ned to be the cyclic code with generator polynomial gOx9 To show that the spheres of radius 2 around the 729 codewords are disjoint, we must show that the minimum Hamming distance between codewords is > 7, that is, every nonzero codeword has weight > 7 The following problems contain a proof of this fact1< 9.64 *" * *& -&I  ! "&%*  > . F HintD & *&  %! &

*&& 36G 9.65 *" * , C< ‡ C1 ‡


‡ C1< ˆ & >+ 6 F HintD && &  . &- 3(G 9.66 ,  *& *& * C< ‡ C1 ‡


‡ C1< ˆ á 6ˆ &&+ -;&& *  ,- -H!&%*& * -$& &  +*!*! *& -& "&& &-*&& &*!*! -H!&%*&  BCH codes    $  & & >& * + *& -&  *& &-% %*  "&& -;&& >+ *&& &+ "&  *& *+ , *& &-% %*  && $$ 67.B677 6 O$ 26:9 , *& -*-&- , F  J& "& ! & * *& -*-&- &  ;& n . O$ 2..9 + -;& *& &%&& , *& 4& $+   1 *  &  * >- ,- #& &%Oab9 ˆ &%Oa9 ‡ &%Ob9 &%Oa ‡ b9 < @O&%Oa9 &%Ob99 &- " * &;& , & , a  b  *& 4& $+  7 O$ 2.:9 * $,  !&  0  *&& ) O$ 27.9    >!   $$- *& J& F " >& GFO2 m 9 ,  & m > 1 "&;& *& !&+% *&+ %& *!%* &H!+ "& , + J& J&    "& * #&  !&-&+ &-  F : O$ 2:19 & , -!&  & , *& && - $& , C "&& -!+ & :1:O>9 1< O$ 23& !& , + ,,&& $ , ;&"  * !-+ &- "& $&& *&& $$-*& "*-* "& *;& -& *& polynomial matrix $$-* *& scalar matrix $$-*  *& shiftregister $$-* O *&& *& $$-*& *& state-diagram $$-* *& trellis $$-*  *& tree $$-* " >& %;& &  *& -*$&2 9 ·The polynomial matrix approach &- , *$& : *  >+ On k9 & >-#6 -& - >& -*-&4& >+  k 3 n %&& @ G ˆ O g ij 9 ;& F2  On k9 convolutional code O9   -*-&4& >+  k 3 n %&& @ GL *& ,,&&-&  * ,  -;! -& *& && g ij & polynomials ;& F2  &@ $& *& @ G ˆ Fx 2 ‡ 1 x 2 ‡ x ‡ 1G  *& %&& @ ,  O2 19  "*-* "& >&  1 , ,!!& &,&&-&  +  Gˆ

1 < < 1

x‡1 x



 *& %&& @ ,  O6 29  "*-* "& -  2 /& " &J& *&&  $ ! >& * & -& "*  D

*& memoryD 236

23.

Convolutional codes Mˆ

@ F&%O g ij 9G:

O1-# -& - >& ;&"&    "* M ˆ  ! >& $$&  *& -&,J-& ,  k!$& , $+  I ˆ OI < Ox9 . . .  I k 1 Ox99 *& *& NN-&"II C ˆ OC< Ox9 . . .  C n 1 Ox99 "*-*   n!$& , $+   &J& >+ C ˆ I
G

O1& &-& ;
H O1& &-& ;
H O1 $& & , * -&$&-& & !+  *& - @ $$-*  I ·The scalar matrix approach *&  ! >"& O-9 &$&&  ,  $+  -&" C ˆ OC< Ox9 . . .  C n 1 Ox99  >& >+ &&;% *& $+ I -&,J-& *! , *& j* $+   Cj Ox9 ˆ C j< ‡ C j1 x‡ . . .  *& - &$&& , C  C ˆ OC-# 1 1 < 1 1 1

 1 < 1

1 1 1

Example 10.6 , "& #& L ˆ 2   %&& @

1 < 1 < < 1 < < G2 ˆ   1
 I j >!   *& $&-&% " > I j 1  I j 2  *& &-& ! & & >& " $! >    *& -!& >L *  "*+ *& -&    *;& & + M ˆ 2  , &;&+ $! > *&& & " !$! >L *! *& -&I &  1=2 + & * *& &-& , % 1& , & >+ ! % *& j* !$! & , &-* , *& *, &%& Example 10.7 *,&%& &-& ,  2  !&  % 1& &D @& &$&& *& ,! &L   , & &  *& -&$%   $! , NN+   &%&   -&$%  NN1II >+  *& &%&  *& >&   &%& &$&& *& &-&I !$!   ;& , & &  *&  &@ $& -& *& &%& %% , & cO1&*; "*& *& *, &%&  % 1+ "&% *!%* *& & %  "& + ;& *&  & &%& +  &L * #&  ,J-!  #&&$ -# , "*&& "& *;& >&&   >&- & &&>&  ,+ *& & % >+ -!% *&  & ,  & &*$ *& >& "+   *   *;&  ,,&& -$+ , *& & % , &-* -#   ? *& %&*&   trellis diagram  *"  % 1& -& *& depth  *& &$* j $$&-& ,  & " >& && >+  !>-$ j &$* j &  ?& >+  &%&   &$* O j ‡ 19 & ,, *&&   &%& ?% *& " &  *& & %  O  &@-&$ --! , j ˆ <  1 >&-!&  j ˆ < *& *, &%& , % 1+ -% *& $$$& $* *!%* *& &  &@ $& *& !$! & O111 >+ > &-* , *& ). -&"  R "&;& *&&   !-*  $& &* "*-* #& ;%& , *& ,- * *& ). -&" -&$  *& ). $* , a<  a(  *& & , % 1&"&&  % 1&  *& -&$% " > , R  &@ $& *& b6 ! c. &%& %& *& >& d H O+ S *& & , & fa b c dg &@ , s t 2 S  *&&   &%& %% , s  t  *& & %  "& &J&

Figure 10.8 *& ;& , *& & , % 1&"&& a<  a( 

6& <  1 --%  "*&*& *&  , s  t -&   <   1 $! , *&&   !-* &%& BOs t9   &J& O&& % 1I %*  %;&  % 1 %* , % 1&  * *& - $&@+ , *& %*   &@$&+ -&% ,!- , M O*!%*   + &  L9     +  $-- &* , &-% I "* &;&+   ;!& , M && "*

Figure 10.12 '&>I %*  $$&  *& & , % 1& *&   &;& , -& "* k ˆ 1 " "& -& & %&& -*&  &- !&;&  " $!  I O>! && > 1 216 *   %&&    @ ! #&* &-& ! -&  -&" C i ˆ OC i<  . . .  C i n 1 9 , "*-* *& $>>+ PfRjCi g ˆ

n 1 

pORj jCij 9

jˆ
& "*&& R ˆ OR<  . . .  Rn 1 9  &-&;&  *& pO yjx9I & *& -*&I  $>>& -& *& %*    -+ -&% ,!- ,  %! & *&  &-& + &H!;&+ # ,  -&" C i , "*-* LOC i  R9 ˆ

n 1


% pORj jCij 9

O1& /* * &  * *& '&> %* , % 1#& !$  L ‡ M n> ;&- , *&&   &%& %% , & s  & t  *& & %  && >+ C s t *& n> &-& !$! -&$%  *  *  *& >&  *& s ! t &%& O&& % 1 &-*H!&  *& $>& , &! &%  !-* $*  $$+ *& &!  > $&, -& & & , $&-J- -;!  -&  !! "& " !& *& --&$ "*  1  -! %&&4 & & % *& & % , % 1& , 1I  *& >& , *& &%& - $% *& $* O &@ $& *& $* acbaacddba * "&%* 129 /& "! $&*$ #&  -! *& ! >& , $* , a  a , "&%* i ,  & J@& &%& i >! * ! >& " "+ >& &*& <  J& >&-!& , *& $ , "&%* <  a & "+ ! * $>&   >&;& * &;&+ $* , a  a - >& !H!&+ &- $&   &H!&-& , $* , a  a "*  & && &!  a O &@ $& *& $* acbaacddba - >& &- $&  *& $* acba aa  acddba9 & ! -  $* O*& * *& ; $* aa9 >&%%  &%  a "*  & && &!  fundamental path  , &-* i && >+ Ai *& ! >& , ,! & $* , "&%* i +   & & - --!& A< ˆ A1 ˆ A2 ˆ A6 ˆ A. ˆ && & ;& * & * *&&   && -&$&-& >&"&& *& & , ,! & $*  *& % & %  *& & ,  $*

6&%   -;&& >##&&$% &;-& , &! &% $* , a<  a1 >+ "&%* , , "& &J& *& >& ,  $* P  >& *& $!- , *& >& ,  &%& *& -&+ *& &@$&  *& >& , P " &H! *& "&%* , P  &@ $& *& >& , *& $* P ˆ a< cbcdba1  x :    *& "&%* , P  : &-% * ! %  *& - $! , *& ! >& Ai O"*&& " Ai &H! *& ! >& , $* , a<  a1 , "&%* i9 "& &J& *& path weight enumerator ( ,  1  *& %&&% ,!- AOx9 ˆ A< ‡ A1 x ‡ A2 x 2 ‡




O1& *&  % "&%* , *& &-& !$! -&$%  *& &%&  *& &@$& , z &->& *&  % "&%* , *& -&$% $! *& &@$& , y  "+ 1 -&$ %  *& ,- * *& &%* , &-* &%&  1 -& % "& &J& *& >& ,  $*  >& *& $!- , *& >& ,  &%&  &@ $& *& $* P ˆ a< cbcdba1 * >& x : y ) z 6  * & * *&  % "&%* , *& -&$% &-& !$! O11>+  &-* ! >& *& %*+

Figure 10.15

& &>&+ >&& ;& , % 1&,& -!% &-*H!& , >!% *& & $>>+ "& * &->& "* "& & >+  NN&II !$$& "& & !%  $-! On k9 -;! -&   $-! -&& & +& -*& O>! && > 1+ & $+9 & , -&- $* &% & &$& >+  & , $* "*-* & &&+ >&" *& -&- $* &@-&$ , *& & $ *&& -&- $* &% & "& - error events O % 1& & &;& -&$   ,! & $*  *& &-&I & %  *& ;! &-& & $>>& "& * " -! &  $>>& &&  & &;& & >;! *% "& -! +   - $!& *& $>>+ * *&& &  & &;& ! -& "& *;& ! & L  >& ;&+ %& * $>>+ " >& & < !& *& -*&  && O&& > 1>+ * *& J & &;& >&%  &$* & * & *  - -*& x- !& POy l jxl 9 > POy l j0 l 9 -& *&"& *& $* >&

Figure 10.16  & & $*

10.3 Path enumerators and error bounds

611

Figure 10.17 &$* < & &;&

, x- >+ &$-% *& & &;& E "* Ol " >& $&,&&  x-  + &% &- "* * %;&  *& $, , *&& :7 O-,
H O:1699 *& $>>+ * E " >& $&,&&  0 l >+   &-&  >!& >+ ã wH O E9 "*&& ã  %;& >+
H O:(9  wH OE9  *&  % "&%* , E *! *& J & $>>+ PE1  >!& >+
PE1 < ã wH O E9 O1&;& *"&;& * *& & &;& & &-  *& ,! & $* *!%* *& & % L  AOx9 O
H O1>+ * *& &-&I $* &$ , *& -&- $*  &$* >+ * *& &-&  ,, *& -&- $*  &$* j %;& *    *& -&- $*  &$* j 1 * -%   &"* !!   "& " &J& *& error event probability PE  *& $>>+ * *& &-&  ,, *& -&- $*   %;& &$* j *  ?! *& $>>+ * *&&   & &;& *%% >&" *& -&- $*  &$* j O&& % 1! -&$%  O1 " NN--&+II >& %* *! *& bit error probability Pe "  %&& >& & * *& & &;& $>>+ % &%    * &% 
H O1!  Pe O&& > 1+ & ã7 & 
@ % *& &J , AOx9  AOx y z9 "& && *   >&-!& 7  *&  & "&%* , + ,! & $*   %&& -;! -& *& free distance d f  &J&  >& *&  & "&%* , + ,! & $*    *  *" O&& > 1& *&   $ %& &!& ,  -;!  -&I >+  - > &  !-* &,, * >&& &@$&&  *& $>& , J% -;! -& "* %& ,&& -& O&& &- 126   > 1 1 &-% "! >& *$&&+ - $&@ *! "& & ;&  J  &-% %* * " "#  -;! -& "* ;&+ %& ;!& , M *&&   - , !-* %*  -& sequential decoding algorithms *&+ &  H!&  %   %*  ,  J@& -& >! * &,&-  %&+ - $&& , >+ *& ,- * *&+ - >& !&  &-&  & -& "* ;&+ %& M

*& #&+  !&% &H!& &-% %*   *& tree diagram   *,&%& &-& ,  On 19 -;! -& & ,  > &&% *& &-& -!& n &-& >  &;& O-, % 1&  ;&" *& &-% $-& --&$!+   "# O- >P9 *!%*  binary tree  &$-&  % 1&%  *&   ;&&@  ,& d $!  " >&   & ;&&@  *& &&  &$* d , *& &@ $!  NN-*  *& !$" &-L ,   NN1II  *& "" &-  &@ $& , *& $!   *& &-& " !$!   ;& *& >-*

*& &&  % 1! , -!& , + --&& &4 ,  -& *& && " >& J& , *& L* !- , *& -&  >&% !& *& && " & &  &$* L ‡ ML  >&+

61.

Convolutional codes

Figure 10.19 >+ && ,  On 19 -;! -&

&$* L 1 *&& " >&  >,!- , *& $* -& *&  M $!  *& &-& " "+ >& NN&  + ;& , * $*  *& &-&I ?> " >&  %!& "*-* $* *& &-& -!+ # -& *&& " >&   , 2 L $>& $* , L  %&  "   %&& >& $>&  - $& *& &-&;& &%&  &-* , *&& $* *& $$-* #&  * $>& >+ sequential decoding algorithms   &@$&  ;&+   !>& , *& $>& $* ,   %;& %&  -& $+ &@$& $* # $ %   &@$& ,!*&L ,  &    >&  *& $*  & /& * " &->& *& " >& #" &H!& &-% %*  *& stack algorithm  Fano's algorithm  !& *& & "& >&% "*  NN*!%* &@$& &II

10.4 Sequential decoding

617

Figure 10.20 *& >+ && -&%  *& L ˆ 6 !- ,  1

Example 10.9 %& * "& & !% *& &&  % 1 &H!&-& y ˆ O y1  y2  . . .  yn 9  $& , *& ,- * *& -&" & , ,,&& &%*

61)

Convolutional codes

& *&& ! $ , y  &-&;& *& &-% & $>>+ " >&  4& , *& &-& &&- *& -&" x i , "*-* *& - $>>+ Pfx i &jy &-&;&g   %&  $>& O-, > 2169 ! -& y  J@&  Pfx i jyg ˆ Pfx i  yg=Pfyg *  &H!;&  @ 4% Pfx i  yg ˆ Pfx i gPfyjx i g "*-*  %;& >+ *& ,"% , !D Pfx i & y &-&;&g ˆ pi

ni 

pO yj jxij 9

jˆ1

n 

pO yj 9

O1+ *& $;& - njˆ1 pO yj 9 #& %*   @ 4& *& &!% &@$& "*-*  -&  metricD   ni
pO yj jxij 9 1 1 ìOx i 9 ˆ : O1 ! *;& >&& 11L *&  $ $>>+ , * &;&  12 ! % * NN "&& + 1. !%  * "+ "& ;&  *& ,"% >&D '&&@ A B C D

&" >>+ 11   &H!+ #&+  >& <  1 O&& > 1! %;&% *&    pO&"&& *& ,! $* A B C D "& *! - $!& *& -&$% ,! &- !%
H O1+ 2 d   *& & n i 1 % p i 1 ,
H O1&  &@& , &@-+ & , *&  *&  #&+ $*  *& & , "*-* *&  &- ì Ox i 9  %& F& 
H O1-* %& O , x< &@&  &$* L  >&+ + & &@& " >& $>&L && % 1&L *& $*  &-* -# & &$&&& >+ *& &  ;&-&  *& -&$% &- & %;&  $&*&&

 &@ $& -# )  >& ,

-# 7 >+ &&% C  &$-% 

Figure 10.21  L ˆ 2 M ˆ 2 && , &H!& &-%

10.4 Sequential decoding

613

"* F  G F %&  *& $ , *& -# >&-!&  &- O 19  %& * * , G I  H *& &-&I J &-  *& ;&&@ KL *& -&$% ,  > & 1&-!& , *& ;&+ " &- ,  $&&-& G    $& -*-&4 , *& $* -*& >+ *& -# %* && > 1+ - $% *& &- ì , * ;&&@ O- $!& >+
H O1-* &%  *& ;&&@

*& & , I %* & %;&  *& K"-* , % 1+ !$& , Ä  $&&&-& -    & T " >& < *& &, *& -!& ;&&@ 6  & /  O-&% &$*9  *& &&L  &  5 /   5    - $!& *& &- ì ,  ;&&@ & >-*   *& &&  '
   ;& *& &-&   ;&&@ & >-*  5   '
 *;& -&$% &% , *& %* 5  , *&   ;&&@ "& ! & *& &- ì  >& 1 *&& *   $&;&  >-#" ;& ,    . '  , '

A   %;& ' *& 
 '  * ;&&@ & >-* &&$&  *& && "* *& %& &-L *& / '  *& & "* *&  & &- 7 *& *&* T  tight   %;& ;&&@ , -&% T >+ Ä "!

62
-* , *& &@ ì > P H!&+  * &!  &!-% T  2L *! "& *;& ;&  &$ :  *& >& 12!16D  &$ 12 *& &-&   ;&&@ B "* T ˆ 2 *;% ?! ;& >-#" , E *& "&  *&  / 'P H!&+  
 -& E  *& / ' &;&  B >-#" #  AO ì ˆ & *&$,!  #"  &*% , *& *&+ , !& ;& $-$ & %L -! &% % F3G *$& A' &- 29 & F >&  J&  & FFxG && *& % , $+   *& && & x ;& F & FFxG n && *& & ,  n!$& f ˆ O f 1 Ox9 . . .  f n Ox99 , $+  , FFxG convolutional code C  &J&  >&  !>& , FFxG n  "*-*  O9 -& !& - $&"&   !>-  O>9 -& !& !$- >+ && & , FFxGL *  , f ˆ O f 1  . . .  f n 9 2 C *&   af ˆ Oaf 1  . . .  af n 9 ,  a 2 FFxG basis , C   & fg1   . . .  g k g  FFxG n !-* * &;&+ f 2 C - >& &@$&& !H!&+  k f ˆ iˆ1 ai g i ,  & a1  . . .  ak 2 FFxG *& ! >& k  -& *&  & O #9 , *& -& O9 *" * &;&+ -;! -& C * !-*  >  * &;&+ > , C * *&  & ! >& , && &L  J  > , FFxG n &, O>9 *" * *& &-% &->& >+
H O1 fg1  . . .  g k g , C  > ff 1  . . .  f n g , FFxG n   $+  a1  . . .  ak "* ai ja i‡1  i ˆ 1 2 . . .  k 1 !-* * g i ˆ ai f i  i ˆ 1 2 . . .  k *& $+  ai & !H!&  & -& *& invariant factors , *& -&  !-* >& , I 1  2 O$ 2329  , *& O2 19 -& "* $+  %&& @ G ˆ Ox ‡ 1 x 2 ‡ 19 n O9 &J& *& dot product f
g ˆ iˆ1 f i g i 2 FFxG >&"&& " && & , n FFxG   *" * f
g ˆ < ,  g 2 FFxG n ,, f ˆ 0 O&9 , ff 1  . . .  f n g   > , FFxG n  dual basis   > ff 1  . . .  f n g !-* * f i
f j ˆ < , i 6ˆ j ˆ 1 , i ˆ j *" *  ! > &@   !H!& O,9 , C   -;! -&  dual code C ?  &J&  C ˆ fh 2 FFxG n D h
g ˆ < ,  g 2 Cg  >&%   $ O-9 O9 O&9 *" * ff k‡1  . . .  f n g ,   > , C ?  *&-& * C *  & n k   > , *& ! -& , &-* , *& *&& -& -&  $ O-9 O%9 & C >&  -;! -& "* k 3 n $+  %&& @ G C    >& ;&>& , *&& &@  k 3 n $+  @ H !-* * GH T ˆ I k  *& k 3 k &+ @ *" * C  ;&>& ,, *& ; ,- a1  . . .  ak O&& $ O-99 &  &H!  1  &@$- ;&& , I 1  2 , * -*$& O*9 & %&&+  @ H  -&  øOx9 inverse , G , GH T ˆ øI k  *" * G *  øOx9 ;&& ,, ak jø   øOx9 ;&& , G ˆ Fx ‡ 1 x 2 ‡ 1G "* øOx9 ˆ x ‡ 1 /& &&  $ 236 *  On k9 >-# -& -! >& ;&"&   On k9 -;! -& "* M ˆ -# -&P   $>& *  %;& $ , &  *& & % ,  -;! -& " >& ?& >+ & * & &%& /*& " * --!P &  J@& On 19 -;! -& !-&  &%* L *" * *&  ! , "# ;;&  &-% >+ *& >!&,-& NN- $& *& &-&;& &H!&-&  &-* , *& 2 L $>& -&"II &*  >! 2 L K  &  !-*  * ;;&  !% *& '&> %*  O&& K ˆ *& -&I - &%*L &&
H O1& '&>I &-% %*  *& &@ + , -& "* k ˆ 1 &&4&  %&& k O/-* ! , !$& &%& >&"&& &Q&& > 1-# -& -& *&+ & M ˆ < -;! -&9 & C >& *& O6 29M ˆ 1 -;! -& &->& >+ *& %&& @  Gˆ

1‡x x

1 1‡x

 1‡x :
! + *;& & && &!  a9 ;&  %&& , ! , Bi 10.9 & *& ,"% >&& &-& %$*D

 $!& *&   >&"&& ;&-& a  b  &  , *& &%& >& A B C D O! $$&@  , &J9 10.10  $!& *& $* &! & AOx9  *& - $&& $* &! & AOx y z9 ,  2 10.11 & C >& *& O6 19M ˆ 2 -;! -& &->& >+ *& %&& @ Ox 2 ‡ 1 x 2 ‡ x ‡ 1 x 2 ‡ x ‡ 19  $!& AOx9  AOx y z9 , C " !$$& *& -&   >& !&  *& ,"% >+ &!& -*&D

;& !$$& >!  *& J & $>>+ PE1 *& & &;& $>>+ PE  *& > & $>>+ Pe 10.12 & !%  >+ On k9 -;! -&   J@& "$!   "*-*   && *  "*& -$-+  & * % 2 & PO&L9 && *& $>>+ *  & & & &;& " --!  *& &-% ,  -&" , *& L* !- , *& -& *" *  L!1 POL9 ˆ 1 O&& $ 6!  Pe %;& 
H O1! , &- 1 :2>+ J& PE ˆ áã d f ‡ OOã d f ‡1 9 ,   ã  &,+ *& - a &;& %! &! , PE1  Pe O-,
@ $& 1! O> :2.9  *& L* !- ,  On k9 -;! -&  *" *  ,&& -& O&& $ 6119 d f J& 2 df
& & "*  -& %*  *& & % ,  -;! -& *& %* !$!  &%;& &%&L +! & !$$&  J%!& ! "* *& &%& &$&&   ! & * &-* &%& , *& & %  >&& "* *& ! >& , 1I !$! >+ *& &-& "*& *& &  -&$%  * &%&  &  &@ $& ,  1 *& & % "! # #& * O-, % 1+ V L O29 *& >& ,  &%& -&-% & v i  & v j  lOv i  v j 9L O69 *& & &-*>&  & &$ ,  & v & -& *& successors , vQ, &@ $& *& !--& , & d  *& >;& & % & fd bgL O.9 *& 4& &  && >+ v& D O9 /* & %* X P O>9 $$+ %* X  *& -;! -& "* $+  %&& -& Fx 6 ‡ x 2 ‡ 1 x 6 ‡ x 2 ‡ x ‡ 1G Fx 6 ‡ x ‡ 1 x 6 ‡ x 2 ‡ 1 x 6 ‡ x 2 ‡ x ‡ 1G   1‡x 1 1‡x : x 1‡x
&  & "*  &! $>& * - & , *& "% # , -;! -&  -*& catastrophic error propagation  > 1 1* *& $!  !$! &H!&-& & J& 10.18 & *& O2 19M ˆ 2 -;! -& "* $+  %&& @ G ˆ Fx ‡ 1 x 2 ‡ 1G O9 " *& & %  O>9 & $  --!& *& $* &! & AOx9 /*+ & *& & $ ,P O-9 ! & *& -&  >&% !&    "* " > & $>>+ < , p , 12  !$$& * *& $! &  &;& * " -&-!;& > & & -&  --! &;&!+G 10.19  > 1& , -*& & -! -!&  J& ! >& , &-& & -& , "*-* * - *$$&  -&  catastrophic -&  * $>& "& * %;& &;& &H!;& - , &% "*&*&    %;& -&  -$*-

*! & C >&  On k9 -;! -& "* $+  %&& @ G  *" * *& ,"% - & &H!;& O >&;& * *& -& , > 1& , 1I !-* * *& -&$% !$! & * +  J& ! >& , 1I O>9 *&& &@ k  ,!- I j Ox9 ˆ pj O@9=qi Ox9 j ˆ 1 2 . . .  k O& , *& qj Ox9I >&% ;>& >+ Ox9 !-* * *& ;&-

Problems

10.20

10.21 10.22 10.23

62:

C ˆ OC1 Ox9 . . .  Cn Ox99 ˆ OI 1 Ox9 . . .  I k Ox99
G * $+  - $ & O-9 *&&   -& $*  *& & % O*& * *& & $  *& 4& &9  "*-* &;&+ &%& * >& & , &-& & F HintD & *& -*& & $& >& *& ;&- C , $ O>9G O9 *& k* ; ,- ak , *& -&  not  $"& , x O-, > 1&&  , G    $"& , x  $-! , k ˆ 1   > 1& -*& >+ *& -# %* O *&  -*-&4 , *& $* * "  >& -*&9 ! & "& *;&  && !-*  *& & &$-&  % 1& -*& >+ *& -# %*  O&& > 1+ I %* 9 10.24 % &" &-  *& && , % 19 *& $* -*& >+ *& -# %*   O-9 *& $* -*& >+ I %* "  >& ,,&&

62(

Convolutional codes

 *& &@ : $>&  "& " &;&$ &;& , *&   $ $$&& , I %* O% 1 1&*;  &->& >+  &H!&-& , $ Ov1  T1 9 Ov2  T2 9 . . .  "*&& v i   ;&&@  Ti  *& ;!& , *& *&*  v i  >&% ;& *& &- ,  ;&&@ v  && >+ ìOv9L "& ! & * *& &- *;& >&& !& ,,  * *&+ &  !$& , *& H!+ Ä , " ;&-& v  v9 &  *&  & $* *!%* *& &&  *& &$* , v9  %&& * * , v "& + v9   successor , v v   predecessor , v9  v9   immediate !--& ,  &$*  ?! & & * *& &$* , v 10.25 !$$& *&  %*   &"*&&   $& $!-& *& &H!&-& Ov T 9 Ov1  T1 9 . . .  Ov n  Tn 9 "*&& v1   !--& , v v n ˆ v >! v i 6ˆ v , i ˆ 1 2 . . .  n 1 *" * T n ˆ T  10.26 & *&  & ! $   > 1& &H!  *& &- , *& ;&&@ 10.28 *" * ,&  !!--&,! ," # ,  ;&&@ v "* *&* T O& "*& *&  >-*  #& , *& !$$& ì > T P H!&+9 &;&+ $* ," , v ! -  & ;&&@ "*& &-  , T  10.29 , *&  %*   Ov T 9 *" * &;&+ $* ," , &;&+ $&&-& , v ! -  ;&&@ "* &- < T   -+ *" * &;&+  && !--& , v "* &- . T ! &+ *;& >&& ;& 10.30 &,&  *& #&-* , > 1 1 1+ &!& -*& "* &!& $>>+ ED

&-& *& ,"% &-&;& &H!&-&D PPPPPP11   ! F2)G 7 O$ 23(9 % , % 1!   $ - $& ,  $-- &-&   !$&@% "-* * "! & && > , C<  C1 " *& -*& ) O$ 6!  -+ &-&+ , *&  $& & ,  '&> &-& *& & % & - &+ &;&   + - & , *& %* I >&*; : O$ 6>+ "& * - $!& &  ,- &$&& , *& ! &  & $* O&& *&  " &&-&  *& $, , *&& :7 $ 1779 1< O$ 6169 *& &  * &-  $&&&  *& - $&& &;&& , *& *- & , -;&+ "*-*   ,"D  %* O13)69 -# %* O13)39 +- ?!J- , *&  &- O13:29 *& $$-* #& *&&  !&  &+ F21G 11 O$ 61:9 -!+ *& NN-#II  >&& ;&"&   NN$+ H!&!&II   !-!& &J&  *$& . , * $-,    F1G

11 '>&&%* !-& -%1

11.1 Introduction &  -&& & +& !-& "* !-& - p ˆ O p<  p1  . . .  p r 1 9 *   &H!&-& U 1  U 2  . . . , &$&& &-+  >!&  ;>& "* -  >! PfU ˆ ig ˆ pi  i ˆ &  &$&& * !-& ,*,!+ !%  ;&%& , r 1 2 H 2 O p9 ˆ iˆ< pi %2 pi > $& !-& + >  * -*$& "& !+  -;& -!-;& $-&!& , % * -& variable-length source coding

 %& *& %&& & O *& &&  "& * *& ,"%  &@ $&   &"* &&-!9 -& *& $-! !-& p ˆ 12 1. 1( 1(  "*& &$+  H 2 O12 1. 1( 1(9 ˆ 1::7 > *& !-& $*>&  AU ˆ f& &-&  + &-% &;&+*%  & $$&+ &?+& >+ *& -& , >& 111  *   uniquely decodable *  *& !-& &H!&-& - >& &-!-& $&,&-+ , *& &-& &   &@ $& *& > & 1& &$&&& ,*,!+ ;& >+  ?! >+ $*>&  &- 112 "& !+ *& $!&+ - > $>& , !H!& &->+  &- 116 "& -* -&  !-&  &- 11. "& $&& !,, I , ! %* , -!-% $  ; >&&%* -& ,  %;& !-&

11.2 Uniquely decodable variable-length codes & S >&  J& & -% s && &L !& *&"& $&-J& S ˆ f+ Æ  *& % -%  + > , ó ˆ s1 s2 . . . sk  ô ˆ t1 t2 . . . tl & % *& concatenation ó  ô  *& % s1 s2 . . . sk t1 t2 . . . tl  , ó ˆ ó 1  ó 2  ó 6  ó 1  -&  pre®x ó 2  substring  ó 6  suf®x , ó  *! *& & $+ % Æ   $&J@ !,J@  !>% , &;&+ % *& length ,  % ó  && >+ jó j  ," * jó  ôj ˆ jó j ‡ jôj , + $ , % ó  ô

O;>&&%*9 code 6 ;& S   J& & , % ;& S *& % - $%  -& C & -& *& codewords , C & C1  C2 >& " -& *& product && C1  C2 - ,  % , *& , ó 1  ó 2  ó 1 2 C1  ó 2 2 C2  *& $!- ,  -& "* &, k  & " >& && >+ C k *  C k ˆ C  C  . . .  C Ok ,-9

*& -& C  -& uniquely decodable O9 , &-* %  &-* C k &  + & "+   --& , -&" * & * , + ô1  ô2 


 ô k ˆ ó 1  ó 2 


ó k  &-* , *& ôI  ó I   -&

662

Variable-length source coding

" *& ô1 ˆ ó 1  ô2 ˆ ó 2  . . .  ô k ˆ ó k  *! &;&+ %  C k - >& !H!&+ &-&   --& , -&" O  &H!;& &J ,  && > 1119 Example 11.1 & s ˆ 2  -& *& -& C1 ˆ f&-!& , *&& " &! * $&J@  -& & &+ -&&L $&J@ -& & H!& &+   $& & O&& > 1169  ,   -& &@ "*  -& & , &%*  $&J@ -& "  &@ Theorem 11.1 OMcMillan9. If C ˆ fó <  ó 1  . . .  ó r 1 g is a UD code over  r 1 ni S ˆ f&  $;& &%& *& -& C   !k !k k
n @ r 1



ni jó j s ˆ s ˆ s jó j ˆ K l k s 1  iˆ
 O1119 h Example 11.2 #& s ˆ 6 *&   -& "* &%* O1 2 2 2  2 2 6 6 6 69 -! &@ -& 6 ni ˆ 2(=2: * ,- "!  >& &+  $;& &-+ h Theorem 11.2 OKraft9. If code with lengths ni 

r

1 ni iˆ< s

< 1, there exists a pre®x Oa fortiori UD9

Proof & ! && *& ni I  * n< < n1 <


< n r 1  &J& &%&  j 1 nj ni w  j ˆ + w< ˆ 1 -& j r 1 ni nj s < 1  ," * w < s 1 ,  j " &J& *& % ó j j iˆ<  >& *& s+ &$&& , *& &%& wj "* &!%*  & %&   >& 112 -& * , "& +  $$+ * -!-  *&  
@ $& 112 "& %& w3 ˆ 2: ˆ 1 11)  11:9 h 11.3 Matching codes to sources ;% " &&  &*% >! *& $!&+ - > $$&& ,  -& "& & $&$&  -* -&  !-& , &,J-& - ! - &- O$ 6239 *  %;&   "* !-& $*>& A ˆ f+ *& $>>+ ;&- p ˆ O p<  p1  . . .  p r 1 9  s-ary code , *& !-& p  &J&  >&  $$% , A   -& fó <  ó 1  . . .  ó r 1 g ;& S ˆ f i  *& -&" ó i    ó i  , , &@ $& "& !& * &%+    k !-& + > + i1  i2  . . .  ik *& &-&;& " > *& % ó ˆ ó i1  ó i2 


 ó i k  , "& " *& &-&;&  >& >&  &-;& i1  . . .  ik , ó  "& ! ! & * *& -&  

*& average length , *& -& fó <  ó 1  . . .  ó r 1 g "* &$&-  *& r 1 !-& p ˆ O p<  . . .  p r 1 9  &J&  >& n ˆ iˆ< pi jó i j  $-- %J--&  * *& ! >& , s+ + > O& + > , S9 &H!&  &-& *& J k !-& + >  #&+  >& >! k
n , %& k *! n   &!& , *& ;&%& %  *& -*&  $& >+ *& !& , *& -& *&& 116 O"*-* -!+ ," , &!   &9 %;&  &&% "& >!  n /& ! & * ni && *& &%* , *& i* -&"  C *  ni ˆ jó i j Theorem 11.3 If the code C is UD, its average length must exceed the s-ary entropy of the source, that is, n > H s Op9 ˆ

r 1
iˆ
! *& -! ;!& , ns Op9 *& &@- --! ! >& &,  &- 11. >! *& ,"% & &  H!& !&,! Theorem 11.4 H s Op9 < ns Op9 , H s Op9 ‡ 1: FNote. The lower bound of Theorem 11.3 is repeated here for aesthetic reasons.G Proof &J& ni ˆ d% s p i 1 e i ˆ + *&& 112 *&& &@   -& "* &%* iˆ< s r 1 n<  n1  . . .  n r 1  *& ;&%& &%* , !-*  -&  iˆ< pi ni  >! -&  r 1 ni , % s p i 1 ‡ 1 * !  , iˆ< pi O% s p i 1 ‡ 19 ˆ H s Op9 ‡ 1 h Example 11.4 & p ˆ O:1 :. :79 s ˆ 2 *& H s Op9 ˆ 1:6)1 *& ni I !%%&&  *& $, , *&& 11. & O. 2 19 &!%   ;&%&  &%* , 1: "&;& *& &%* O2 2 19  ,+ 2 nk < 1    ,- ns Op9 < 1:7 *& $  * *& !$$& >! , *&& 11.  !!+  *$ O>! && > 1139L  & &   %& %&&+ 

66)

Variable-length source coding

&& $&;!+  "+  -!- $   -& ,  %;& !-& " >& %;&  &- 11. h

 * $ "& *;&  $>&   & * *& &! , *$& 6 & ! *  *! >& $>&  &-& *& !-& p ,*,!+ !%  *& ;&%& + H s Op9 s+ + > $& !-& + >  *& *& * "& *;& ?! && *    $>&   * "*   s+ -& , p -& ns Op9  !!+ -+ %& * H s Op9

*& !  * $>&   -& -& , *& extended !-& p m  m ˆ 1 2 . . .  *& !-& p m  &J&  >& *& !-& "* !-& $*>& Am O&- * A  *& !-& $*>& , p9  "*-* *& !-& + > Ou1  u2  . . .  um 9  %& *& $>>+ PfU1 ˆ u1  . . .  U m ˆ um g ˆ pu1 pu2 . . . pu m   &,,&- "*& "& -& *& !-& p m  "& & >-#% *& !-& &H!&-& U 1  U 2  . . .  -&-!;& >-# , m + > , A  &%% &-* !-* >-#   %& + > , *& $*>& Am  "   -& , p m "* ;&%& &%* nm " &H!&  ;&%& , nm s+ + >  &$&& & + > , Am  & + > , Am &$&& m + > , *& % !-&   * -& ,*,!+ &$&& *& % !-& p !% + nm =m s+ + > $& !-& + > *& ,"% *&& *" * >+ #% m %& &!%*  !% * &-*H!& "& - #& *& ;&%& ! >& , + > &&&  &$&& p ,*,!+  -&  H s Op9  && Theorem 11.5 

m!1

1 ns Op m 9 ˆ H s Op9: m

Proof   &+  && * H s Op m 9 ˆ mH s Op9 O> 11129 &-& >+

*&& 11. mH s Op9 < ns Op m 9 , mH s Op9 ‡ 1 *&& 117 ," >+ ;% >+ m  #%   h

*&& 117  ,+% *&&-+ -&  & ! * *& !-& p - && >& &$&&& ,*,!+ !% O$&*$ %*+ & *9 H s Op9 s+ + > $& !-& + >  &;&  &*%  >& && ,  -!- ;& ;&"$ *"&;& -&  &&  *& "&# -!- , *&& 11.  *& &@ &- "& * & &+ * ! >+ $&&%  &-*H!& "*-* " &>& !  -!- *& >& $>& -& , *& !-& p m 

11.4 The construction of optimal UD codes (Huffman's algorithm) 66: 11.4 The construction of optimal UD codes (Huffman's algorithm)

--%  *&& 11. ns Op9 &  &"*&& >&"&& H s Op9  H s Op9 ‡ 1  * & &  &H!& ,  & $!$& O&% *& $, ,

*&& 1179 !   !  "& >! *& &@- ;!& , ns Op9 ,  J@& s  p  * &- "& $&&  %* !&  ; !,,  * *"  + *"  - $!& ns Op9 >!  *"  -!-   O&&  $&J@9 -& "* ;&%& &%* ns Op9 &,& %;%  ,  &-$ , !,, I %*  "& * "#  &@ $& *!%*! * &- "& &,&    s+ -& , p "*& ;&%& &%*  ns Op9   optimal -& , p Example 11.5 & s ˆ . p ˆ O:2. :21 :1: :16 :1% *& *&&  & $>>&  p *! p9 ˆ O:2. :21 :1: :16 :1-#"  -!- $  -& C9  C , p9  p >+ NN&@$%II *& -& C 0    $& "+  "& -!-  $  -& , p9 -& * *&& , *& $>>& O;4 :2. :21 :1:9 "&&  -*%&  *& &!- , p9  p0 *& !&  * -&  *  *& &@$ , C 0  C9 *&

Figure 11.1 *& !--&;& &!- , p

66(

Variable-length source coding

-&$% -&"   -*%& &*& "&;& $>>+ 6(  p 0 &@$  ,! $>>& O:16 :1& !,, I %* L ! & % #Q"*-*  !$%+ -#+Q  *" *  "# & *  $!-&  s+ $&J@ -& , p "*& ;&%& &%*      $>& *& ,"% *&&  -!-L  %!&& * *&&  "+  $  $&J@ -& , p  "*-* *&  s9 " & , *&  & &%*  %&& &@-&$ , *&  - $&

 J@ ! & & ! " ! & * *& $>>& p<  p1  . . .  p r 1 & %&  &-&% &D p< > p1 >


> p r 1  /& -& $&J@ -& , p ;& *& $*>& f+ ó <  ó 1  . . .  ó r 1  "* &%* ni ˆ jó i j i ˆ 2, there exists an optimal s-ary pre®x code for p with the following two propertiesD O9 n< < n1 <


< n r 1  O>9 The last s9 Osee Eqs. O1129 and O11699 codewords are identical except for their last component, that is, there exists a string ó of length n r 1 1 such thatD ór ór

s9

ˆ ó  + *&& 112 *&& "! &@  $&J@ -& "* *&& &%* * &" -& !  >& $  O , -!& p r 1 ˆ !  * "! -- *& ! $ * ni    /& --!& * < < Ä < s 2  "*  *&  & *% s

Ä 2 f2 6 . . .  sg:

O11:79

&@ & r9 && *& ! >& , " , &%* n r 1  *& -& , r9 ˆ 1 "& -! *& *& !H!& %& -&" >+ &&%   - $& "*! ;% *& $&J@ - %  -- &-& r9 > 2:

O11:)9

, "& &@ &
H O11.9 ! s "& %& Ä  r9 O  s9 *  r9  s Ä O  s9 ! ,
H O1179  O11)9  ," * r9 ˆ ks ‡ Os

Ä9

,  & k > s9 *  *&& &  & s9 %& -&" &@ >&;& * , ó 9 ˆ ó  a  & , *&& r9 -&" , &%* n r 1  "*&& ó   % , &%* n r 1 1  a 2 f+ &$-% -& " , &%* n r 1  , *& , ó  a "* & * & "& ;&   $  -& "*-* * ó  p1 >


> p r 1  r > s *& s-ary Huffman reduction , p  &J& >+ p9 ˆ O p<  p1  . . .  p r

s9 1 

pr

s9

‡


‡ p r 1 9:

 " p9  >& , p >+ - >% *&  & s9 $>>&  p "*&& s9  &J& >+
H O1129  O1169 *& ,"% *&& *" *"  $  -& , p + >& -!-& , & , p9 -& p9 * ,&"& + > * p  +&  &-!;& $-&!& , -!-% $  -& , p

11.4 The construction of optimal UD codes (Huffman's algorithm) 6.1 Theorem 11.7 If C9 ˆ fô<  ô1  . . .  ô r then C ˆ fô<  . . .  ô r

s9 1 

s9 1 

ôg is an optimal code for p9

ô  &% >+ $;% O1139 & On<  n1  . . .  n r s9 1  n9 && *& &%* , *& -&"  C9 + &J p< n< ‡


‡ p r s9 1 n r s9 1 ‡ pn ˆ ns Op99 "*&& p ˆ p r s9 ‡


‡ p r 1  , C  !&   -& , p  ;&%& &%*  ns Op99 ‡ pL *&-& ns Op9 < ns Op99 ‡ p:

O11:1&  $  -& , p * &?+ $$&& O9  O>9 , *&& 11) *& -& fó <  . . .  ó r s9 1  ó g !& , p9 * ;&%& &%* ns Op9 p *  ns Op99 < ns Op9

p:

O11:119

 >%
H O111& , !-& &!- &H!&  dOr s9=Os 19eL ,& * + &!- *& !-& " *;& &@-+ s + >  f 11219 F&

*& >+ !-& &$+  H 2 Op9 ˆ 1:32 >G Example 11.7 & s ˆ 2 r ˆ 2 p ˆ O:3 :19 *&& H 2 Op9 ˆ :.)3 "*& !,, I %* +& *& ,"% ;!&D n2 Op9 ˆ 1:+ *& &&L && > 11139 *& $ *&&  * *& -;&%&-& , m 1 ns Op m 9  H s Op9 O&& *&& 1179  !!+ H!& $ O>! && > 1129 

*&& &@  &-$*&>& % "* &$&-  C r 1 jó i j O-9 ˆ 1 iˆ< s 11.3 ;& *  $&J@ -& C  &-&+   &%  &,J-& %* , &-% C *  , ,-%  J& --& , -&" , C   - $& -&" 11.4 *" * *& $!- , $&J@ -&  %  $&J@ -&  *& $!- ,  >+ $ ,  -& &-&+ P O&& &J $ 66& +! &  +  -!-  -& ;& S ˆ f& "&  ó ˆ r1  i  r2  ô ˆ r1  j  r6  "* i j 2 S  i , j *" * &@-%$*- &%    &% *  , + $ Oó  ô9 "* ó 6ˆ ô &*& ó ,
ô  ô ,
ó O!9 & *& ,"% && , ! , *& -!-   *&& 112D *& ó < ˆ & &@-%$*-+ & % , &%* n k‡1 * *  $&J@&  *& & fó <  ó 1  . . .  ó k g *" * * -!- "+ +& *&  & -&  *& & %;&  *&& 112 *" * ns Op9 ˆ H s Op9 ,, *&& &@ $;& &%& On<  n1  . . .  n r 1 9 !-* * pi ˆ s ni  i ˆ   $$&   J@& ! >& n , -*& $! + > * $>&  &%&  *" *"  ;>&&%* -& - >& J&  J * & *!%*! *& !-& p ˆ O p<  p1  . . .  p r 1 9 " >& J@&  *& -*& " >& 

6..

Variable-length source coding &&   "* $!  !$! $*>& >* &H!  S ˆ f>& %;& >+  pO yjx9 ˆ

11.11

11.12 11.13 11.14 11.15 11.16

1
& J@& &%& " -& *& ,"% &%+ , $$% k !-& + > U1  U 2  . . .  Uk  n + > , S  $ &-* !-& + >  *& -&$% -&" , C  , *& --& ó , *&& -&" *& &%* , ó   % ;& S    ;>& >! "& ,+ ó  *  * &%* n  ," , *& &%* , ó &@-&& n "& &&& *&  jó j n + >L ,   & * n "& ? n jó j   ;& *  *&+ "&& erased >+ *& +&  && >+ PE *& $>>+ *  & & , *& k + >  *& !-& >-#  && & å . < >& >+ O9 *" * , + í . ns Op9 *&& &@  -*& & , *& >;&  "* n=k < í  PE , å O>9 % *& &! , $ O9  *&& 117 *" * , + r , b H s Op9c 1  å . >+ >! , U " !$$& OU 1  U 2  . . .9   J& #; -* O&& > 12 2  & ;&P *" * , r > s ,& dOr s9=Os 19e s+ &!- , p ˆ O p<  p1  . . .  p r 1 9 *& &!% !-& * &@-+ s + > & $$! "+ , ;% *& $>& *  *& J !,,  &!- ,  !-& p *& ! >& , + > - >& + >& & * s   ? s s9 &@ NN! +II + >  *& !-& $*>& &-* --!% "* $>>+ >& O &@ $& 
@ $& 117 s s9 ˆ 1  *& !-& "! >&- & p ˆ O:2. :21 :1: :16 :1>+ ;&- O p1  p2  p6  p. 9 "* p1 > p2 > p6 > p. , "*-* *&&  & " -& & $   *& -;&@ *! , *&& J@& $>>+ ;&-L  J *&& *&& ;&- 11.22 &  % & , NN"&+ H!&II  "*-* +! & &H!&  && & ,& #%  -& ! >& , H!& * - >& "&& NN+&II  NNII *& !- & , &  ,  $ , -& O *& !- &  *&&,& & , *& &%& 2 6 . . .  129 /*  *&  ! ! >& , H!& +! &&  #  *& ;&%&P F HintD , +! #& NN   2PII NN   6PII &- +! "! ;&%&  & !& @ H!&   $>&   >&& *"&;&G 11.23 *& >?&- , * $>&  , +!  & *"  &%  $  binary $&J@ -& , *& + &- r+ !-& p r ˆ O1=r 1=r . . .  1=r9 O9 !- !-*  -& , p2  p6  p.  p7 O>9  %&& "* & *& $>& &%* , *& "   $  >+ $&J@ -& , p r P O-9  $!& n2 Op r 9  %&&

Notes 1 O $ 66?&- & , * -*$&  >& $&&&  &   &@  ,  *&+ -&    $-!+ &&$  &+ &  &$&  + , *& $&-&% & "&;& "& *;& -*&  $-&  *&& >&-!&  &$&&  -!-;& $$-*  *& !-& -% $>&  ?!  *& &  *$& :B1< &$&&  -!-;& $$-*  *& -*& -% $>&  2 O $ 66& & >* &H!  f>+ , å ,   å9  *& &

6.)

Variable-length source coding

-&  * -& *& & ,!- ROä9 &;!&  *& &&  Oä ˆ &&%* -& - >& , &   -& &&   >-# -& O&& > 111&"&& s+ $&J@ -&  s-ary trees "*-*  & $*&   &@  "& *! &   & >&K+  !& *& & -& *& $&J@ -& f-*  && >+  NN+ -&  &@ O&% >  F12G %& F1:G9 $;& + *& s ˆ 2 -&  &;& *& O;M9 %&&4   $>&  O&&  > 111)9

12 !;&+ , ;-& $- ,  "

12.1 Introduction

* -*$& &;& *&  & ,!- ,  "  *$& ) &;& ,  & *   ! 4&  & , *&   $ &!  -% *&+ "*-* *;&  >&& &&  *$& :B11  &- 122 126  12. "& & -*& -% O>-# -& -;! -&   - $ , *& "9 +  &- 127 "& -! !-& -%

12.2 Block codes

*& *&+ , >-# -&  &  -*& * *& *&+ , -;! -&   * &-  !-* %& * &- 126 O *  >-& &  $$+  $-- $$- *"&;&L && &- 12.9  &  %;& * &-  & & >-& , %4 "& * -,+ *& &!  >& -& --%  &&# $I F17G  , *& *&& ? $>&  , -% *&+D 1 " % & *& >& -&P 2 " - "& &% % -&P 6 " - "& &-& !-* -&P · How good are the best codes? & , *& && $>&  "*-* &  -% *&+ " * , J% perfect codes , "& ;&"  -& , &%* n ;& *& J& J& Fq   !>& fx1  x2  . . .  x M g , *& ;&- $-& Vn OFq 9 *& -&    >& $&,&- O -& $-#&9 , ,  & &%& e *&  % $*&& , ! e ! *& M -&" - $&&+ J Vn OFq 9 "*! ;&$ + 137< &;& # , $&,&- -& "&& #"D 6.:

6.(

Survey of advanced topics for Part two n

2e ‡ 1 Oq m 19=Oq 26 11

q

e

& #

2 O>+9 &$& -&L && > :1( 19 O>+ 1  % -&L && &- :. $ & > :13 $"&9 2 6 + + -&L && &- 3( 6 2

&+ + -&L && > 36& &-&-% $"&  &&-*& !+ +&&  &@& *& >;& >& "&;& ,&  % && , &@& &+ ,J-! ;&% &%  *& &+ 13:& &  -/   & F13G *$& )

*&&  &@ $&,&- -& "* *&  & $ &&  *&  % -& "*-* &  &H!;&  *& L *& *& *&& # *"&;& & !H!&

*& H!& , *& &@&-& , $&,&- -& ;& $*>& "* q + > "*&& q is not a prime power   * "%  - $&&+ &&  &&  !#&+ *"&;& * + &@ $$& >+ *& &@&-& , *& $&,&- -& &&-*& "&& *& &-*  -& * &    %D nearly perfect  uniformly packed -& ;& J& J& !-* -& & "  -J& , e > .L , e < 6 + $ &! & #"L && ; >% F.)G

*& &@&-& , *& O26 129 $&,&- + -& & ! * *& @ ! ! >& , ;&- , V26 OF2 9 "*-* - >& -*&  * &-* $ , ;&-  &$& >+   % -& > :  &@-+ .+ n  :  &$-& >+ d   "& &J&

AOn d9 ˆ *& %& &%& M !-* * *&& &@ M -&" fx1  . . .  x M g , Vn OF2 9 !-* * d H Ox i  x j 9 > d , i 6ˆ j:

&;&+ O12:19 AOn d9 ˆ *& %& ! >& , -&" $>&   -& , &%* n   ! -& d:

12.2 Block codes

6.3

*& !+ , *& ! >& AOn d9  &%& >+ +  *& -& $>& , -% *&+ *!%* AOn d9  &+ #" &@-+ !& n  d & &;&+    2d > n  %& & , J& &&-* * %&  *& $>& , J% % !$$&  "& >! , AOn d9  *& &@ ,&" $%& "& " ! 4&  & , * &&-* >! , *& "*& + *& &&  &,&&  -/   & F13G *$& 1:  &;&+   ;!& , n  d &&+ *& + #" "+ , >% "& >! , *& , AOn d9 > M   &@$-+ &@*> M ;&- , Vn OF2 9 ,+% *& - -&  O1219 /& -! !-* -& -!- >&" O&& $ 6.3 ,,9  *& *& *  &  >  !$$& >! , *& , AOn d9 , M   &-&+  *" * &;&+ !>& fx1  . . .  x M g , Vn OF2 9 -  & & $ Oi j9 !-* * d  Ox i  x j 9 , d *&& &  %& + $>& "+ , % * >! -!&+ *&  $"&,! &-*H!& $$&  >& *& linear programming $$-* "*-* "& " &->& , C ˆ fx1  . . .  x M g   >+ -& , &%* n *   !>& , Vn OF2 9 , &-* i ˆ & *& ! >& , -&"  C  -& i , x *& distance distribution , C  &J&  >& *& On ‡ 19!$& OA<  A1  . . .  An 9 , &%;& & ! >& "*&& 1
Ai ˆ Ai Ox9: M x2C & && &+ $$&& , *& Ai & 

 , *& -& *

&&D

A< ‡ A1 ‡


‡ An ˆ M:

O12:29

A< ˆ 1:

O12:69

 !

-& d   ," *

A1 ˆ A2 ˆ


ˆ A d

1

ˆ !  AOn d9 ,   &;&+ $ On d9 - & , *& & $% % >!   J- ,   &@ $& *& >!  AO16 )9 >& >+ - >% *&  &H!& O1279 "* *&  *- &H!+ A1< ‡ .A12 < . O"*-*  &+  > - >+9  AO16 )9 < 62  &;& +& >&,& *& -;&+ , *&  $$-* AO16 )9 " *&  $& !#" ;!& , AOn d9  &&-*& * !%%& "*  !% *&-+ - $-& - > %! & >! 62 < AO16 )9 < 67 " *& >& *&+ -!  , -!& *&  "& >!Q"*-* & >&-!& *&&   #" -& *& & -& "* n ˆ 16 d ˆ ) M ˆ 62Q- >& "* *& &" !$$& >! && *& &D AO16 )9 ˆ 62 &*$ *& >& #" ;& , *& AOn d9 $>&  *& asymptotic problemD , Od n 91 nˆ1   &H!&-& , &%& "* d n =n ! ä < < ä < 1 *" & AOn d n 9 >&*;&  n ! 1P & , + -& *& & ,  -& , &%* n "* M -&"  1=n %2 M "& &J& 1 %2 AOn d n 9 n!1 n 1 %2 AOn d n 9 ROä9 ˆ ,  n!1 n

ROä9 ˆ !$ 

O12:)9 O12::9

"*&& *& NN!$II  O12)9  *& NN, II  O12:9 & >* #& ;&  &H!&-& Od n 91 nˆ1 ,+% d n =n ! ä   +% * >*  !$$&   "& ;!& , ROä9 ! >& &J&L &;&+>+ >&&;& O>! >+ * $;&9 * ROä9 ˆ ROä9 ,  ä   $-+  "* ," *"&;& "& * &,& +  ROä9  >&% !& *  !$$& >!  ROä9   !$$& >!  ROä9  "& >!   "& >!  ROä9   #"  &;&+ &+  $;& * RO! ROä9  !#" , < , ä , 12 *& >& #" "& >! 2 , ROä9  * %&  ROä9 > 1

H 2 Oä9

O12:(9

 &! $;& >+ >&  1372 O * >!   + $- , , *& >! >&  > :219  & * !--&&&  % >&I "& >! >! *&& * >&&  "*& && , &+ &-&% !$$& >!  ROä9 *& -!& &-*& >& >+ -
&-& & -* ! &+  /&-* !% *& & $% % $$-*  ROä9
!  >&-!& *&+ $&   $&  &,J-& &-% %*    , *& -& "& * &->& *"&;&  % &-% %*  +& #" /& * &->&  & , *& &&-* * * >&& &;&  *& &% , &-% %*   *& &@ !>&- O&& $ 676 ,,9

Figure 12.1 *& >& #" !$$&  "& >!  ROä9

672

Survey of advanced topics for Part two

>>+ *&   $&;& &!  -!-;& -% *&+ & "* *& - , cyclic codes  On k9 & -& C ;& Fq    >& -+-- , "*&&;& c ˆ Oc<  c1  . . .  c n 1 9   C *&   *& -+-- *, c9 ˆ Oc n 1  c<  c1  . . .  c n 2 9  + ;!& , q  n > 6 *&& & "+  & ,! -+-- -& , &%* nD · · · ·

*& 4& -&" 0 ˆ O&  4& , x n 1 , n  q & &;&+ $ & O -+-- -& , "*-* *   *& -& & !!+ !&&%9 *& 4& , x n 1 & f1 â . . .  â n 1 g "*&& â   $ ;& n*  , !+ ;& Fq L *! *& 4& , gOx9 & , *& , fâ a D a 2 Ag "*&& A  f 2t ‡ 1 /& "   & , *&   $ &! #" >! -+-- -& O & , q ˆ 2 &;& *!%* %&&4  %& q & !!+ #"9 · , n ˆ 2 m 1  B ˆ f1 6 . . .  2t 1g *& d  > 2 m 1 Ot 192 m=2 O&& ;  F1(G *&& )6)9 · , n   $ & -%!&  1 O  .9  A ˆ fH!- &!& O  n9S *& k ˆ On ‡ 19=2 

d



& p ' n p > 1 ‡ .n 6 ( 2

, n  1 O  (9 , n 

1 O  (9:

*&& & *& quadratic residue -&L && -/   & F13G *$& 1) · , *&&   j!$& b1  b2  . . .  bj 2 B !-* * b1 ‡


‡ bj  < O  n9 *& &;&+ -&" * "&%* ;>& >+ 2 j O %!& ;& , *& $&- -& j ˆ 2 , * *&& $$&  &  7 $ 2:(  $,  &@&;& %&&4 -! &&  -
&-& F6.G9 · , B ˆ f1g *& -& & -& irreducible -+-- -& O>&-!& *& -*&-# $+  hOx9  &!->&9  minimal -+-- -& O>&-!& *&+ -  ; -+-- !>-&9 *&&   &>& *&+ , *& L && -/   & F13G *$& (  -
&-& F63G · , n ˆ 2 m 1 B ˆ f< < b , 2 m 1 D b - >& "&   ! , < d $"& , 2S *& &!% -&  &&+ &H!;&  *& &&B !& -&  Om d9 , > :6!     &"* $$% >&-!& *& >& -& & &+ -+-- + * "& & !%*+ * ,   + -*& $ On d9   H!& !#&+ * *&& " &@  -+-- -& "* AOn d9 -&" O&& &J O12199  ,- *& >& -& & ,&  &;& &  *&&  >&%%  >& H!&  &$&->& -&- , % & -&  & &@ $& O%  -& *;& q ˆ 29 ,"D · *& >& -& "* n < 2d & $--+  &  & &&  Hadamard matrices && -/   & F13G *$& 2

67.

Survey of advanced topics for Part two

· *& Kerdock & -& *;& $ && n ˆ 22 m  M ˆ 2.m -& " d  ˆ 22m 1 2O2 m 29=2  m > 2 *&  $& &@ $& Om ˆ 29   -& , &%* 1) "* 27) -&"  d  ˆ ) *& >& & -& "* n ˆ 1) d ˆ ) * + 12( -&" && -/   & F13G *$& 17 2m · *& Preparata -& *;& $ && n ˆ 22m  M ˆ 22 .m  d  ˆ ) m > 2 *&+ $$&  >& !  *& 5&-# -&L && -/    & F13G *$& 17 + "& & * -/   & F13G $$&@  %;&  >& , *& >& #" -& , d < 23  n < 712 + , *& -& & *&& & -&;&  *- -!- * "& - &->& *&& · How can we decode such codes? *& $>& , J% *& %& $>& -& "*  %;& n  d   $  -*&%% !  -!+ $! !-*  -&  && + -&  !&   + -*&   &-&+  *;&  $-- &-% %*  , *& ! >& , -&"    &!%*  *   ,&>&  - $& &-* , *&  *& &-&;& " *&&   $>&  , *& -&   On k9 & -&  q n k    &!%*  * *& + & >& $$-* , &- :2  :7  $>& %  $>&  "&;&   $-- !   !#&+ *  -&   &!%*  " & , *&& $ ;& &-% %*  " >& $"&,! &!%*   *& ?>  * & -&>& &,, * >&& &;&  J% % &-% %*  , %&  $"&,! -&  * !>&- "& * &->&  & , *&& %*      $  *& B &-% %* * "& &->&  &  *$& 3 -& *  !-*   $ &! "& $!& *&&  $+ >!&   -;&&  1373  13)< - & *& % $$& >+ -H!&%*& OF17G $$ :2B :.9  &  +*!*! OF17G $$ :7B:(9    $  & * *&& &+ $$& --&& *& &;& + "* *& -!- , *& -&  *& &-%  13)< && OF17G $$ 1+  $& &-% %* "*-* ;;& *& ! ,  ! &! & &H! ;& -& J& J& & +& & &&  =&& OF17G $$ (:B(39 %&&4&  *& $&;! "#  >+ -&  ,! * *& %&&4 -!& *& -& -;&& >+ &&    OF17G $$ :&,& *& &  + *!*! $$& * >&& $!>*&M & >&#*!%*  *& &-% ,  -& - &  13)( "*& &&# $ OF17G $$ 1.7B1.(9 O&&  *$& :  F1.G9 $!>*& * >&!,! &;& &-% %* 

+&  &%&& "* "*&   4& >!&+ *& - $&@+ , 

12.2 Block codes

677

 &-&   "& ;&  !& &&# $I $-&!& ! -&  $ --&$! ;-& "&& +&  - &  13:< $$ OF17G $$ 1&!,! ! %&&4 ,  -& & *"& * *&& & % $$ -& * & + $-+ %  *& && * *&+ & >;& *& >& "& >! , % 121 O * - "* *& ,- *  -&  $& , *& !&,!& & + $-+ NN>II  *& && *  + &H!&-& ,  -& "* & > R . < *&  d  =n ! $$-* &!,! ,- "*-* "& -!&  &%*  *$& 3 * & - !&
!-I %*  &-&   &&B  -& -!+ "*& $$&   -&
!-I % *  ,& >+    - ,-  &&# $I   !-* &&  !& *"&;&  *& -&>& %*  &&# $I % *   ,- "* *%*   " $>&  ;&" &&# $I %*    $;& ;& ,
!-IM +  13:7  --& * >+ #&" " >&% , & " - $&& "*& && F66G  *& *"& * $$I -& - >& >& >+ ,+% *& -& &&B  -&M

*!%* *& %&>- &-% , B -&  *&   $ &!  *& & &;& *& H!& !&,! &-% &* &  ;>& /& * " &->&  & , *& 

*& && ; &-% %* -;&& " threshold decoding "*-*   # , %&&4 , *& NN ?+ ;&II &-% &&  *& !- -;&& >+ &&  137. , &-% &&B !& -& O-, > :6+ -&% *& O: 69 >+ & -& "* $+-*&-# @

1 1 H ˆ < 1

1 < 1 1

< 1 1 1

1 < <
&% "* *$& 16 , -/   & F13G  "* &+ F2+ !$! H!4 + * "& & *& ,"% + >+ ;&- x ˆ Ox1  . . .  xn 9 2 Vn OF2 9 - >&  & 2 ;&  ! -*& "* ;&%&  $"& - EOX 9 + $$% xI 4& -&  ‡ â  xI & -&  â , + x  ,   -&  & "*&  !& *& &-% %* , *$& 3  *& &-&;& ;&- y & " *;&  >&% >+ , % *& -& , yQ"*-* & & ! >&Q + $$%

12.3 Convolutional codes

67:

$;& -&  <  &%;& &  1 *  "* "& & >+ >+ !$! H!4 ,!&+ * J &$ &-&& *& -* &I -$-+ O>+ >! 2L && > .179  , * & -& "*  *&&+ >+ &-% %* & $--+ !&&  + ! -*& "&;&  "& *;& &&  &-% , J& &&B !& -&   *&&+ >+   *&& -& & -;& , !&  ! -*& *& &&$$-& - !- -*&  "& &&   ! -*&L     !$% * , 13)3  13:)  ,   I Mariner - &&$$-& $>& "&& &H!$$& "*  O62 )9 &&B !& -& O*& &&$$-& $>& *;& >&& &H!$$& "* -;!  -&L && &- 12.9

12.3 Convolutional codes

*& *&+ , -;! -&  !-* & "& &;&$& * *& *&+ , >-# -& $+ >&-!& *& !>?&-  !-* +!%& >! + >&-!&   -+ & ,J-!  &@ $&  13:< &+ OF17G $$ 216B2629 "&  &  $$& * $;& &! , -;! -& %!  *&  $& &!  &- :1 , >-# -&M *    + * -;! -% *&+ *  $!-&  $ &&-*  & &@$ *"&;& "*+ * &-  !-* *& * *& $&-&% & O  -;! -& & ,& !$&  >-# -&  $-- !L && &- 12.9

*& free distance ,  -;! -& O&& $ 6119 &;& *&  & &  & *&  ! -& ,  >-# -&D  % -& ,  J@& & *& & "* *& >%%& ,&& -&  #&+  >& >& !-* &,, * >&& $&  *& $>& , J% -;! -& "* *& %& $>& ,&& -& *!%*  , + , -;! -& - $ >&  + *&  >-# -& * +& >&& ,! *&& &@ &@&;& -%!& , $&-J- % -&  &@ $& & F6:G %;&  >& , % O2 19 O6 19  O. 19 -;! -& "* & + M < 16L 0*& F6)G  % O2 19 -& !$  M ˆ 67   -*-&- , *& !>?&- * *&& >& *! >& &-&  On k9 ˆ O2 19 O6 19 O. 19 &$&-+ O2 19 *& &@ &%&& , ,&& $&&  -;! -&  &+ *& & + M #& *& !+ , O2 19 -;! -&  &@& &+ -* !>?&- ! &, "*&& O2 19 >-# -& & , -!& ;  *!%* *& $>+ , !+% -;! -& "* J@& M  ;+% On k9 !%%& &, >+ * +& $!!&  ;&+ ,

67(

Survey of advanced topics for Part two

*& -*& -% *&& 2. " $;& !% >-# -&   !  "& "*&*&  - >& $;& !% -;! -& *& "&  ,   R!&  - >& $;& !% tree codes O-& >& >+ %% >&   && !-!& #& *& &  % 1+ %% >&   & !-!& #& *& &  % 1-& , *&  !-* -& - >& >& , *, &%& O&& % 1& , $* * *& &H!& &-& ! ;&%& >&,&  &-& *& J L ,  >   #" * PfC . xg  Lx

á



"*&& á   $;& - * &$&  *& -*&  *& -& & >!   x  ! ! * , *& -&I &   -&  -*& -$-+ *& ;!& , á & >&"&& 1  2 *  !  $& * *!%* *&  ;>& C * J& &  ;-&  J&  * * !$& $--  $-  & & >& * "*& *& &-&  +%  J  "+ *!%* *& && & + > & >&% &-&;& *&& + > "%  >& &-& ! >& &   >!,,& , J& &%* , *& &-& &H!&  + - $! * >!,,& " ;&K"  ,  " >&  , *& ;-& , C  J& * >!,,& " >& ;&K"%  *&  &  &H!& &-%  !&&

*& &  "*-* * !$& $*& & --!  -& *& computational cutoff rateL   ?! *& & R< "& *;& &+ &-!&&  > 221B22)  .6B.3L R<    & & && >+ *& + > R- $ 
;&  & >&" R& ,& && & *& -&I $&, -&  &@ $& >+ !%  O2 19 -;!  -& "* ;&+ %& & +  ,&& -&   "&> !

12.4 A comparison of block and convolutional codes

673

-*& & - &+ #& *& > & $>>+ &@& &+   >!   ;&+ ,J-!  &-&& *& erasure $>>+ O& *& $>>+ , *&&  >&%  &  &-&  %;& >9 >&" + 1< 6 *& ,! + , &H!& &-% - $! >!   >+ '&>   ! F2)G *$& )    '&> &-%  &H!& &-% *&&   *  $ +$& , &-% , -;! -& -& threshold decoding   %!  *& *&* &-% , >-# -& "*-* "& &->&  &- 122  " &;&$&  13)6 >+ &+ F2+ %& F1:G &- )(L   ;&+ -&;& $$- , *&* &-%   --&& -;! -% +& * &-&+ >&& %;& >+ /! F.(G + "& & *  13:) -*#"?#  '-# F..G ;&&  &-% %* *   %!& , -;! -& , + & &-% O&& &- :29 , >-# -&   * !-*  -  "* '&>I &-% %* 

12.4 A comparison of block and convolutional codes  * &- "& #&  >&, & $  - $& *& &;& & , >-#  -;! -&  $-- $$-   ,+ -& * *& &&$&    $&;& *&&- &!  -% *&+ O>-#  -;!9  *& %&>- &-% , B -& "*-* "& &->&  *$& 3    $  >&   *"&;& * *&& -& "&& &%& ,  ;&+ $&- - , -*& *& q+ + &- -*& O $-! *& >+ + &- -*&  *& -& ,  -&9  * *& &-% %*  &  &+ O,  9 $>&  *& -*&  *& *& * *& "   $ &-% %*  , -;! -& '&>  &H!& &-% *!%* $&*$   *& -+ $,! & &@& &+ >!  - *& &+ >& $&   ;&+ "& - , -*&  -&  &;&+   ,- , *& - !- -*& * &  $--& & "& && >+  H    !$% *  &;&+ %& ,- , $-- $$- , -% *&+ ;;& -;!   >-# -& /& " !& * $ "* " $&-J- &@ $& & J  >+ + &- -*& "* " > & $>>+ p < , p , 12  %!& 122 "& %;&  %$* , *& $&, -& , " $-- -% +&  , * -*&  O12: ).9 1-#  -;! -&   

"* *& &-% %* &->&  % 31   O2 19M ˆ ) -;!  -& "* %&& @ G ˆ Ox ) ‡ x . ‡ x 6 ‡ x ‡ 1 ) 7 . 6 x ‡ x ‡ x ‡ x ‡ 19  ;!& , p %&& * >! p ˆ :&& >! ,   & p *&  -&  >&&L , &@ $&  p ˆ :& H!& ,J-!   $& &  & 1=2 -;! -& "* $&, -& #&+ !$&  * *"  % 122 -& *& + "+  %& >&& $&, -&   !&  -& "* & & +L >! , *& '&> %* -&% M >&+ :  (  &@& &+ ,J-!  , &H!& &-%  !&   -& "* %& M *& &!& $>>+ O& *& $>>+ * *& %* " ,  #&  &-

12.4 A comparison of block and convolutional codes

6)1

>&-!& , *& - $! $>&  &->&  &- 1269 "  & *& -&I $&, -&  *& *& *  "!  >& ,J-!  %    -& , & $$@ &+ 1=2  &%* 277  711  %& $&, -& #&+ !$&  *  % 122   & -  -&& - $&@+ /& --!& *& * ,  >+ + &- -*& !& *& -*&  &@& &+ +  >-# -&  $-!   -& " $>>+ >& $&,&>&   -;! -&

  &- &@ $& -& *& "&> ! -*& &->&  *$& . O$ 379  % 126 "& %;& $&, -& -!;& , *&  & " -&  >&,&L *& H!+ Eb  *&  P=R "*&& P  *&  & $"& O "9  R  *& !-& & O > $& &-9 *! Eb *  & ?!& $& >L   !!+ -& *& energy per bit  Eb =N <  -& *& bit signal-to-noise ratio *&  NNII & * "* 

Figure 12.3  $ , >-#  -;! -&  "&> ! -*&

6)2

Survey of advanced topics for Part two

$&   Eb =N < &-+ >! 1< %1< Eb =N <  *  4  *&  & , * -*&L && , &@ $& &&  0-> F67G  % 126 *&  -&  + *   !$&+L   " #&+ ,&  *& -;! -& *   >&-!&  &  $$+ *& %&>- &-% %* , *$& 3  *&  -&   &-&+  !& >+ !$! H!4 O-, $ 6679 "*&& *& '&> %* - >& $&  --&$ !H!4& & ! >&  $! "* &;& && O-, &- 1+ >+ !$! H!4   $+  %&  >& - $&& , >+ *& ,- * *&  -&  *&&+ & $"&,! * *& -;! -&

*&  $ *%  #" >! *&& " &@ $&  * *& "&> ! -*& &   "& ;&+ , $-- - !- !  "*&&   *+ &;& --!   $!& ,    *!  >& !$% * -;! -& *;& >&& !&   ?+ , $-- $$- , &-&-% -& O  &&% &@ $& , $-- $$-D *& - !- # >&"&&  $-& ;&*-&  *& &*  !!+ H!& "& &&   "&> ! -*&  &;& $*-& -% -*& & *;& >&& !&  &&$$-& $$- -& *& & 13)!  13:: * +& " -$$&  ,; , *& O2 19M ˆ ) -;! -& &->&  * &- *& $-&  O&%   I Pioneer   /& & +I Helios9 *;& !& &H!& &-%  % OM > 2.9 O2 19 -;! -&9 & !  & *  13:) &;& , !    I 0& $! >+ &;&  &-% %* ,  $"&,! >-# -& *& O.( 2.9 H!- &!& -& * &  &H!& >+ !$! H!4

*& $&, -& , * -& * >&& $&  *& *& -!;&  % 126 *!%*  $&, -&   !$&  * , *& -;! -&  !%%& *& ,"% $&-!D -& *& O12: ).9  -&   >&& -& * *& O.( 2.9 -& Od  ˆ 21 ;: d  ˆ 129 ,  % >+ &-% %* -! >& ,! , *&  -&  %* "& ! !  >& %J-+ !$&  *& -;! -& + "& & * *& $&, -& , *& -;! -&  % 126  *& & , *&  + , ;!& , Pe , >! 1< 6  &  &@& &+   ;!& , Pe  *& >& #" $-- +&   --&  -*& & O&& &- )2  3)9  "*-* *& & -&   -;! -&  *& !& &   &&B  O>-#9 -& O&,&&-&D &"& F.1G9

12.5 Source codes

6)6

12.5 Source codes *I !-& -% *&& 6. !,,& , *&  & ? K"  * -*& -% *&& D   -!-;& O&& !-+ & #  *$& :9   *&&,& !  !$$& * &&-*& + *;& &;&$&  -!-;& *&+ , !-& -% %!  *& -!- ;& *&+ , -*& -% $&&&  *$& :B19 *& --&$>&   $--& &    * &;&+ & - >& %&L O-9 +   !-& * &  $--& & &@& &+ ,J-!  & *& -+  & *& -&$% & !& ,  *& -- &@ $& , *&  $*& &  *& -& , ;!  , &@ $& $*%$* ,  $& /& " &->&  & , *& &  $ &! * *;& >&& >&  -!-;& !-& -% /& >&% "* *& & +& ! !-& "* &H!& & J&+ -& O-, &- .29    $  *;& &@$- !-& -% &-*H!& , * !-& >* >&-!&  & ,&H!&+  $--&  >&-!& $&,&-     $>& -& RO& L1 ,


, Ln O-& H!4 levels9  T1 ,


, T n 1 O-& H!4 thresholds9 !-* * f Ou9 ˆ Lk ,, T k 1 , u < Tk  "*&& T< ˆ 1  Tn ˆ ‡1 * $$% ,  *& -!! & +& !-& U O & &9   -&& & +& !-& f OU 9 * ! & *& ;!& Lk "* $>>+ pk ˆ PfT k 1 , U < Tk g *& ;&%&  * &! , &$&&% U >+ f OU 9  , -!& ä ˆ EFU f OU 9G2  !*& &  "& &&  *$& 11 *& ;&%& ! >& , > &H!&  &$&&  + > , *& !-& f OU 9  HF f OU 9G *! O&,&%  % .29 *& $ R ˆ HF f OU 9G ä ˆ EFU f OU 9G2  -*&;>& >+ * H!4 -*& & !$%+ @  *& *;& *" >+ ! &- &-*H!& * , *& &;&  *&* & $$&+ -*&   $>&  * "+  -*&;& $&, -& *  H!& -& O"* & >9  *& ROä9 -!;& O @I $$&   $$ 2):B2:)  F27G &%& F16G &- 71

6).

Survey of advanced topics for Part two

$;&  & && ! + , H!4 &-*H!& -!%  -! , *& $$->+  ! !-& "* & + &&  $$& 13B27  F1)G9  &  &->&  & , *& & $*-& !-& -% &-* H!& * & #" "& $!&  -! *& %  + >&"&& !-& -%  -*& &-%  -+ "& * &- !&;&  *& >+ + &- !-& &;&  *&  %  O&& $$ (2B(69 &- *  !-& -& , &%* k  & Rs   !>& Cs ˆ fv1  v2  . . .  vM s g  Vk OF2 9 ˆ Vk "* k 1 %2 M s ˆ Rs      %;& >+ äOCs 9 ˆ

1
d  Fu f s Ou9G 2 k
k u2Vk

"*&& , &-* u 2 Vk *& &-% ,!- f s Ou9  &J&  >& *& !-& -&" , Cs -&  u "* &$&-  *&  % -& d  O-,
H O61(99 " &- *  -*& -& , &%* n "* & Rc   !>& Cc ˆ fx1  x2  . . .  x M c g  Vn "* Rc ˆ n 1 %2 M c  + *& -&" x i &  & >! >&-!& , -*& & x i - >& &-&;&  + y 2 Vn  *& &-% ,!- f c ! *& $ Vn  *& -& Cc L &+ f c $ *& &-&;& y  *& -&" -&  y "* &$&-  *&  % -& d    >* -& *& ,!- f $  ;&- , *& "*& $-&  *& -& -&" * !%%& * %;&  % -*& -&  &-% %*  *&  & -& - >& !&   !-& -& "*&& " *& -*& &-% %*  !&   !-& &-% %*  O -& + *& -*& &-% %* *! >& !&  *& !-& &-% %*   +! && "*+P9 * &   ,-  % & >!   & "* $,  "& * " &@$ /& *;& &+ && &@ $& , !%  -*& -& >-#"  %&  !-& -&  *& !-D *&& "&  *& &$& -& >-#"  &  $ "&  *& O: .9  % -& >-#"  %& R ˆ & !& +   -*& , "*-* & * *&& &   &@& & +  $>>& &;& ! , !-& -%  "*-* &-* , *& ;&-  V61  &H!+ #&+  >& & &    &L , (7V , *&  & *& &-&E&-& " > > !M O *& !! ,!& & " >& * *& &- $+  ó Ox9 " *;& &%&& 6 >! " *;& ,&"& * 6 4&&  *& J& F62 9

*& >;& -& *;& & &&-*&  +  &% $-- &-% %*  ,  -& * " $!-& *& && -&"  & "* *& &-& $!   %&&-&-%  -& *&&   $>& D *& &-% %*  &+ - $&& &&# $ F1.G *$& 1) * %;&  - $&& %* , *& "&-&-% -&  J+ '&*  &%& F.:G *;& !& *& *&&& -&-% -&

*!%* "& *;& !& *& --&$ , & $+%  -*& -&   !-& -& !% >-# -& & -! ?!  "& !& -;! -&  && &;& &&-*& *;& &   &@ $& &%& F16G *$& ) -!& -& J- , &H!& &-% * %;& % &! & &-& $$& , & F2(G %;&  $ , *& -# %*  !-& -% *& *%+ F1)G && >+ ;  + - &;& & $$&  *&  & & -!%  &&% $$-  $&&-* - $&

Appendix A >>+ *&+

/& & $*4&  -& * ! >?&- *&&    &-* *& && $>>+ *&+  * $!$& "& &- & &*& *& * ># >+  $& F(G  *& ";! & !+ >+ && F.G ! >?&-  &+ ?!  &&    -& -;&      & *& "&# " , %& ! >& "*-*  *& >-  !&  *& $, , *& -% *&&  , *$& 2  6

*& >- --&$  * ,  probability space 8Ù B P9 "*&& Ù   & $+ & -& *& sample space B   & J& , !>& , Ù  P   &%;& ,!- &J& ,  A 2 B "* *& $$&+ * PfÙg ˆ 1  91  1 8
P An ˆ PfA n g nˆ1

nˆ1

$;& &-* A n 2 B  *& A n I & ? P  -& *& probability measure  &@ $& , Ù ˆ fù1  ù2  . . .g  J&  -!>& B  *& -&- ,  !>& , Ù 8 p1  p2  . . .9   &H!&-& , &%;&  ! >& "*& !  1 *& *& &J PfAg ˆ f p n D ù n 2 Ag #& 8Ù B P9  $>>+ $-&L   -&  discrete $>>+ $-&

random variable X   ,!- $$% Ù   & & R -& *& range of X 8/& * &&  ;>& >+ !$$& -& && , & *& & , *& $*>&9  + "& ! & R   !>& , *& & ! >& >!  & & R " >& *& # , &  &@ $& , R   !>& , n &
!-& $-& X " >& -&  random vector  >& $&  >,-&D X *& - $& , X " >& && >+ X 1  X 2  . . .  X n L *!   ;&- X ˆ 8X 1  . . .  X n 9 - >& ;&"&    , 8& &9  ;>&

"  ;>& X  Y &J&  *&  & $>>+ $-& & 6))

Appendix A

6):

  >& &H! almost everywhere , *& & fùD X 8ù9 6ˆ Y 8ù9g * P &!&  X ˆ Y :&: ,, PfX 6ˆ Y g ˆ & "* %& R &H!  *& & &  expectation  ;&%&  &J& >+  E8X 9 ˆ X 8ù9 dPL Ù

 " E8X 9  *& &>&%!& &% , *& ,!- X "* &$&-  *& &!& P

*&&  *& &H!;& &J , &@$&- *  ,& & -;&& , - $! *& &!& P &J&  Ù !-&  $> >+ &!& PX  R ; *& &J PX 8S9 ˆ PfùD X 8ù9 2 Sg , + & & , & * &!&  -& *& distribution of X  *& ,!- S F X 8x9 ˆ PfùD X 8ù9 < xg  -& *& distribution function of X *& &@$&- , X  &H!+ %;& >+  1 E8X 9 ˆ x dPX ˆ x dF X 8x9 8 :19 R

1

 *& && , &>&%!&B&?& /& $!&  - &  " $&- -& , 8 19  , Ù  -&&  , , &-* x 2 R "& &J& p8x9 ˆ PX 8fxg9 ˆ PfùD X 8ù9 ˆ xg &J 8 19 >&- &
E8X 9 ˆ p8x9
x x

"*&& *& %& , !   $!$&+ &, !$&-J&L  - >& ;& + -&& !>& S , & !-* * PfX 2 Sg ˆ 1 & %&&+ , f  + &;!& ,!- &J&  S f 8X 9   &"  ;>&   &@$&-  %;& >+
EF f 8X 9G ˆ p8x9 f 8x9: 8 :29 x

!  , * # $$& *!%*!  & , * >#  ,& f 8x9 " ! & *&  $$& ;!& 1  " >& !&J& , -& ;!& , x "&;&  " "+ ! ! * f 8x9 " >&*;& +   & ,

6)(

Appendix A

&!& &  $>>+ $-&  & A1  A2  . . .  A n >& & , B *& NN&;&II A1  A2  . . .  A n &   >& independent , , &;&+ !>& , *&  A i1  . . .  A i m  "& *;& PfA i1 \ A i2 \


\ A i m g ˆ PfA i1 g . . . PfA i m g: &  $  -&- X 1  . . .  X n ,  ;>&  &J&  *&  &  $& $-&    >& &$&& , *& &;& A i ˆ fùD X i 8ù9 2 S i g & &$&& , + -*-& , S1  S2  . . .  S n 2 B *    &+ -  ;&,+ &-+  "& %;&  && , !  *& -& , + "  ;>& , X  Y & + $ , &  ;>& &J&  *&  &  $& $-& &$&&   *& $$% ù ! 8X 8ù9 Y 8ù99 !-&  &!& PXY  *& J& , " & & & *  -& *& joint distribution , X  Y  *& ,!- F XY 8x y9 ˆ PfùD X 8ù9 < x Y 8ù9 < yg  -& *& joint distribution function  + * X  Y & &$&&   + * *& &!& PXY   $!- &!& *  * , + $ S  T , & & & & PXY 8S 3 T 9 ˆ PX 8S9PY 8T 9 "*&& PX  PY & *& & & &!& !-& >+ X  Y 
H!;&+ X  Y & &$&& ,, F XY 8x y9 ˆ Fx 8x9FY 8 y9 "*&& F X  FY & *& >! ,!- , X  Y   *& -&& -& , "& &J& , &-* x y  8%& , X  %& , Y 9 _ p8x y9 ˆ PfùD X 8ù9 ˆ x Y 8ù9 ˆ yg X  Y & &$&& ,, p8x y9 ˆ p8x9 p8 y9 "*&& p8x9 ˆ PfX 8ù9 ˆ xg p8 y9 ˆ PfY 8ù9 ˆ yg 8& *& &>&& !& ,  >%!! D *&  & + > p8:9  !& , " - $&&+ ,,&& -&& >! ,!-L *&+ - >& %!*&

Appendix A

6)3

+ >+ *& ,- * && &  *& %& , X & && >+ xI && &  *& %& , Y >+ yI9  *& -!! -& , X  Y >* *;& && p8x9  q8 y9 X  Y & &$&& ,, x  y F XY 8x y9 ˆ p8s9q8t9 ds dt 1

1

*  X  Y *;&  joint density p8x y9 ˆ p8x9q8 y9 /& - " & *& "&# " , %& ! >& "*-*  ;  *& $, , *& -% *&&  Weak Law of Large Numbers  &-* n & X 1  X 2  . . .  X n >& &$& &  ;>& &-* "* J& &@$&- ìQ&& &-* "* *&  & >! ,!- *& , &-* å . å ˆ & K  E n O
!-& n$-&9  -& convex , *& & &% & ?% + " $ , K  -&  KD

*& & &% & ?% x1  x2  , + &J&  fx D x ˆ tx1 ‡ O1 t9x2  t 2 F& %;&  &  , convex combinations , + J& ! >& , $  ," $ x    >&  -;&@ - > , x1   x2  . . .  x m , *&& &@ &%;& - á1  á2  . . .  á m "* á i ˆ 1   á i xi ˆ x *& & ,  -;&@ - > , x1  x2  . . .  x m  -& *& convex hull O% 19 , x1  x2  . . .  x m    &+  *" *  & K  -;&@ ,, &;&+ -;&@ - > , $  K    K

Figure B.1 ;&@ *!  "  &

6:
& -!!  >!+ $ , K D & K ˆ F    Information Theory and Coding &" #D -" #  13)6

* && &+  *%*+ &>& !-  ,  *&+ -! % "*  $, , *& -*& -% *&& , -&& & +& -*& 16 &%& >+ Rate Distortion Theory
%&" ,, 0D &-& 13:1

 ;-& ># &% "*+ "* *& !-& -% *&&   %&&4   $-- $$- 1. &&# $
"+ Algebraic Coding Theory &" #D -" #  13)(

6(.

References

6(7

&% "+ *& $+ --&&&  *& ,!*& &-*& , *& *&+ , >-# -&

tour de force >+  !*&- $* %&! 17 &&# $
"+ & Key Papers in the Development of Coding Theory &" #D


& 13:.

 *%+ , ..  $ $$&  >& , *& &@&;& & - &&# $

& >+

1) ; &&  + >& & Data Compression !>&% D "& !-*    13:)

 *%+ , .)  $ $$& >! & *&+  !-& -% &-*H!& 1: %& >& Information Theory and Reliable Communication &" #D 0* /&+   13)(   & +& *&  ;-& &@>#  *& !>?&- -!& &@&;& & & , >-# -% & &@$&  ";&, -*& 1( '  0->! Coding Theory &-!& &  *& -  2&,  !- !-  *& *&+  $$- , -;! -& 22 && /&&+ Error-Correcting Codes  >%& D    & 13)1

*& -- &@  -% *&+ & & " >!  >& ,  & & , *, &%&  $& &   >& , &!->& $+  8&& &@ &+9 26 && /&&+  /&
" Error-Correcting Codes 2 &  >%& D    & 13:2

- $&&+ &;& ;& , *& $&-&% &+ -%  *& *&+ , $+  -&

!-* &" ,  >+

2. #&   Information and Information Stability of Random Variables and Processes

& >+  &&  --D &+ 13).

6()

References


%*  , *& -- ! &@ -% *& + ;>& - $&& & & , !! ,  , >+  ;>& 27 &$ ; & Key Papers in the Development of Information Theory &" #D 


 & 13:.

 *%+ , .3  $ $$&  ,  *&+ 12 >+ * -!% * -- NN *& - *&+ , - !-II  NN% *&&  ,  -&& !-& "*  J&+ -&II 2) '&> &"   ! 0  Digital Communication and Coding &" #D -" #  13:(
-+-$&- -;&%& , ,  *&+ &$&-+ &-& ;-&  & &@$&  & *&+
$*   -;! -& *!%*! 2: /,"4 0-> The Coding Theorems of Information Theory 6
 &D $%&'&% 13:(   "& ;&+ , %&&4 , *& -*& -% *&&

  -;&&

3. Original papers cited in the text FNote  , *&& $$& $$&&  *& IEEE Transactions on Information Theory "*-* "& *&&,& >>&;&  ITG 2( & 0* NN -# %* , !-& &-% "*  J&+ -&II IT 20 813:.9 211B22) 23   !%!! NN  %* , - $!% *& -$-+ , >+ -&& & +& -*&II IT 18 813:29 1.B2+ &1E2 -;! -&II IEEE Trans. Commun. COM-24 813:)9 3::B3(7 .7 !%+  ! 5*  " *%&-*   &#" **# NN &* , ;% #&+ &H! , &-% $$ -&II Information and Control 27 813:79 (:B33 .) ' >% &-! NN, + $-#& -&II * & &-*-*& %&-*
*;& *& &*& 13:) .: '&* 0&  &%& >+ NN $&& &-% , $&&-&-% >+  -&II IT 22 813:)9 16(B1.: .( /! /  NN&" -;! -&Q II IEEE Trans. Commun. COM-24 813:)9 3.)B377 .3 /+&  NN&-& &!  *& * *&+II IT 20 813:.9 2B1! &-$ ! >& , >! $& , &%* b  % >! , >! & '! & >! , >! -&- &&;% *&& , >! -&- &&;% -+-- -& % >! & -&-

*& & -!- &%%I & 

1:1 1:2 1:. 1(( 131 13) 2&   &H!& &-% 67( !& -&- 133 ,, ! , & >&*; , &H!& &-% %*   $&&-& , 62( 8> 1 :629 &&B  -& , -&-% 2)6B2). $-+ - $! , 12) -! , %J--& 7(B73 , >+ + &- -*& 11 , -&& & +& -*& 7. :6 8& .9 , + &-   7: 8 * 269 $-+- ,!- -!+ ,  ⠈ â   )3 8> 279 -;&@+ , 72 ,   71 ,, , ! -*& 37 %&& &J 116 *& *& - $$&& 7. %J--& -!& 72

63
 1 219 ): 8> 22 22)9 62( 8> 1+ + &- 1 16 8& 29 21 7< >- 12) -&& & +& 21 7< ! > & -$-+ , 3: &J& 37 %&&4& 127 $&, -& , & -&  1)2 "&> 3: 12< 8> 769 6)1B6)2 &  --&& +&  2)7 #; :< 8> 21 :119 q+ + &- 1.. r+ + &- 7( %+ + &- )( 8> 269 + &- 7: "&#&+ + &- )( 8> 229 *& -& See & -*& *& -% *&&  See -% *&&  *-&- , J& 6:) &

>  2& )<  8*$& 3  &;&   -& &-% %* , 26: ,,L *+ , 67.B677 &J& 262B266 %&&4& 2(6 8> 3119 %&& $+  , 267 $+-*&-# @ , 262 $>& !&  ! -*& 6)1B6)2 >-# 8 -&  *$& :B3 & >-# -&9 1)7 8& 29 - $&  -;! -& 673B6)< --&& "* -;! -& 6)2 ! >-#"  !-& -& 6).B6)7 -*& %&& &J ,   )< -K$$% . 12 8> - $$&& , 622 8> 1 1 :2.9   216 &&;& 1). 8> :629 2 36.B3639 5&-# 67. & 8*$& :  &;&  & -&9 -*& -% *&& - >& $;& , 123 -;! -&  %&&4 , 236 &J& 1.< %&&  $+-*&-# @ 1.6 8 * :19 !-& -% *&& - >& $;& , 167 & 67< &+ $&,&- 6.( & 676 ,, $&,&- 36 8> 6169 1.3 6.: ,, $&J@ 662

632

Index

$&$ 67. H!- &!& 676 6)2 &&B !& 1). 8> :6  :1(9 %&& -&-% 1.( %&& -&-% !>&& -&-% 172 8
@ :.9 !-& %&& &J , (. !-& && 162 66< ,, +& - 1.2 && 167 67( See also & -;! !, + $-#& 6.( ;>& &%* 8*$& 11  &;&  ;>&&%* -&9 ;&%& &%* , 66. &J& 661 !,,  %* , -!-% 66) ,, -*&     66. $ + &J& 66: % *&&  -*& ,  (B11 -;&&  117 16< , %&&   )2 8 * 2.9 :. 8& 1 221B22)9 %&& 116 126 , ! -*& 33 8 * .29 R<  , ! -*& 1& &!& >+ !! ,  2( &+ ,  ;>& 6)( &;;& ,  2. 31(9 Deus ex machina 3<  & ,  -& %* , - $!% 2(2 8> 3.9  & , & -& 1.< -&& & +& -*& See *& -& >! , -& 6.3  ;&%& ,  %&& - !- +&  117  &!& 8,  ,!-9 ,   :7 %&&4& 117  % (2 6.7 8& 29 H!&& , ! !-& 33 >! ,!- 6): ; --! *, &%& 1(: 8% (69   See *& -&& & +& d   1.:   See !-& -&& & +& ! > , -;! -& 622 8> 1 1+ 1 +  &H!& -% 673
!& + > 22 ) :629 133
 &;! $+  2.6
 &;& 61<
 - 27<
 - $+  2.6
 $& 7 1.6 171
 $>>+ > , 8: .9  % -& ( >-# ,  % -& : , -K$$% -& . 12 8>   :1.9 217
 ;!& 2)<
!-I %* , $+  2.. ,, 677
!&I ö,!- 6:)
@$&- ,  ;>& 6):
@$!% 1)1 8> :1:9 1)) 8& 169
@&& !-& 66)

E> = See  %& 
$+ % 661
-% %* 

 *- $;% !-&-*& -% *&&  11: 8% 729 , -;! -& 233 8% 1& 1: 6:. , -&&  ;&- 66 , &%- $-& 12. f&$+ .7 8> 1:9 , !  ;>& .1B.2 8
@ 1169 ? 2( , #; -* .)B.: 8> 12>+ ;&- 1(

-& -&  &-& - &  122 8& .9 n 1 ;& H 1(2 8 >& (19 6:( -% x  %*  62< 8% 1 1 111 1129  !& ,  67) &&>-# >+ &!& -*& &H!$$& "* :1 8> 22! :. 8& 179 127 & J& 6:7 ,,  & $>>+ 611 $K$ 1(. &+  123 623 8& .9 67: " & -*& , - $!% R8ä9 , ! !-& 1 .1(9 && -& , -;! -& %* , && % 627 8> 1 3.9 , %&& -+-- -& 1:6 , &&B  -& 27. >&B'* ; >! 1)2 8> :219 67< + -& See & + $$ -& 677 &&  67. $* &J , 6(<   @ 676 %&>%& / 16 8& 79  % >! 1)6 8> :229 2 2169 :1 8> 21:9 1 .)9 1.) 173 8> :.9  % "&%* 1.7  &- 22 1 .179 See also R!4 && 0 6)2 Helios $-&-, 6)2 "  6)2 -H!&%*&   266 67. $-, 0 123 !,, I %*  , -!-% $   -& 66) ,, &-$*&>& % 6.2 8> 1129 &$&&-& ,  ;>& &J 6)( ,  *& - &J 1( ,  !! See !! ,  ,  & 1)< 8> :169 ,  !-& See !-& $! $*>& See $*>& &&;% , -& , >!& $&- 1). 8> :629 2& (19 676 6:7 0->  6)2 0&&I &H!+ 6:2B6:. 0*&  67: 0* & 3) 0!&& -& See & 0!&& 5*  677 5 16 8& 29 5&+ &H! , &-% E -& 263 8 * 3.9 5,B -  &H!+ 662 5,I *&&   -!-% $&J@ -& 666 8 * 1129 & 5 67: " , %& ! >& 6 )) 3 6169 &%* , & -& 1.< &@-%$*- &% 6.6 8> 11) 11:9 & -& See & & & $% % >! 6.3B67< ;  0 676 -
&-& 0 12( 67+ 1)6 8> :2:9 & 1)6 8> :2(9 6.3 Mariner $-&-, 67: 6)2 #; -* &J& 23 &$+ , -&& .) 8> 12&&%* -& , 6.. 8> 11119 &+ 0 123 2(: 8> 36:9 623 8& .1 2169 &%%I &  21: 8 * (179 & + , -;! -& 236 233 & +& ! $ ,   6) , %&&4& -*& .6B.. &&-! &@ $& 66< &-  I %*  61:  '&>I %*  6+ &-% :< 8> 216 21.9  ! "&%* , & -& 1.: W>! ,!- 6:7  g8x9 --! 1(: 8% (69 !& 
 1). 8> :6! 1)6 8> :2.9 627 8> 1 7:9 &J@ 661 See also & $&J@  ;& $+  13:B13( 6::  ;&  6:) >>+ $-& 6)) >>+ ;&- 1:B1( !- ,  I )3 8> 239 R!4 &,,&-  -$-+ , ! -*& 1 221B22)9 12( 8% ).9 , ! -*& 1& &J& 6))  ;&- &J& 6)) & , -;! -& 23. 23: 233 , %&& -*& -& 73 , %&& - !- +&  117 , & -& 1.< , !-& -& (. & ,!- , >+   &;&   %  3. 8& )9 -!+ ,  ä ˆ ä   31 8> 6.9 &J ,   :)B:: &J %&& 11. &,J-& - $! , 16. , ! !-& 1 & $>>+ 2 +*!*! 5 266  67. &!-& -*& , - $!% C   77 &!- , !-& 6.< &! > See +-*&-# &&  67.B67) &&B !& -& See & &&B !&

63)

Index

&&B  -& See & &&B  &>+ &@$& 12: &$& -& See & &$& & -*
 67< 
8" &!-&
-*&9 @ 1.1 1)7 8& 79 ! &+  0 1 .169 67<  $& $-& 6)) -*#"?# 0  673 -*"4I &H!+ 1!  R8ä99 32 8> 61 :619 *& $* $>&  &&;-&  &-% -;! -& 6 12< 8> 769 6)1 &$B/, *&&  162B166 & 0  6.3 676 67) !-& >+ + &- 1 -&& & +& :7 &@&& 67) ! 33 ,, 16. 6)6B6). && + 11. + &- (2B(6 See also & ,!-    1)) 8& 129 67. !-& -& See & !-&-*& -% *&&  -! , 113B12< & & , 117 8 * 719 !-& -% *&&  See % *&&  $&&-* - $& 6)7 H!&&  -& See  &!& H!&";& 22 -# %*  61:B61(  + , & -& 1.7 -*- @ .) 8> 12% 661 !,J@ 661

!%+   677 ! ,  I )3 8> 2(9 !$&-*& 123 !;;  '&>I %*  6-#" , - $!% R8ä9 ,  ! !-& 1+ !$! H!4  ! -*& 1 .1(9   0 123 -&+ See
$+ H!& &->+ 66 1119 ' & &&>&% 1)6 8> :2)9 '&* 0 6)7 '& & &&  266 2(2 8> 369 '>& &%* -& 617 ,, See also *$& 11 '* ; See >&B'* ; >! '& %  , & & >&% >- ,- >! &$+  !! ,  2( 8% 179 '-# 0 673 '&>  12( 16. 167 623 8& .9 67( '&> &-% See &-% %* 

Index /&# -;&&  -% *&& ,  I 16< /&%* &! & 176 ,, /&%* , & $& ) See also  % "&%*L  ! "&%* , & -& /&-*  67< /&> ! -*& See *& /,"4 0 127

63:

/-&  11: /! / 673 /+&  12) /+&B=; *&& , 166B16. =& & -$-+ :1 8> 2139 :. 8& 1.9 =&& 0 67.