Constructing a prefix-free machine by hand requires keeping its domain prefix-free, which is often tedious. The KC theorem, an effective form of the Kraft inequality, removes this difficulty: to obtain descriptions of prescribed lengths, it suffices to enumerate requests whose total weight is at most . It yields a characterization of as the least function, up to a constant, that is approximable from above and satisfies the Kraft inequality, and the coding theorem, which identifies with the probability that the optimal machine outputs on a random input. The lesson ends with Chaitin’s halting probability .
4.1 Bounded request sets
A request is a pair , asking for a description of of length . A set of requests is bounded if its weight is at most . A prefix-free machine satisfies if for every request there is a string of length with , these strings being distinct for distinct requests.
For every c.e. bounded request set , there is a prefix-free machine that satisfies . In particular for every . Moreover, an index of is computable from an index of .
Enumerate as , one request per stage (if is finite, the construction stops). We assign to the request a string of length , and set . Along the construction we maintain a finite set of free strings, with the following invariants:
- the strings and the elements of form a prefix-free set, the being pairwise distinct;
- the elements of have pairwise distinct lengths;
- .
Initially and no string is assigned; the invariants hold.
At stage , the weight of gives . Some free string has length at most : otherwise, by invariant 2, the free strings have distinct lengths larger than , and their weight is less than . Let be the longest free string with , and . Let , and
The new free strings have the lengths ; no element of had one of these lengths, by the choice of , so invariant 2 is preserved. The strings and , , are pairwise incomparable extensions of , and was incomparable with the other strings of invariant 1; so invariant 1 is preserved. Finally , so invariant 3 is preserved.
The construction is effective, so is a machine; its domain is prefix-free by invariant 1, and it satisfies . The construction is uniform in an index of . The bound on follows from optimality: .
The free strings record the unused part of Cantor space as a union of disjoint cylinders of distinct sizes, like the binary expansion of the remaining weight. A request is served by the smallest free cylinder that is large enough, which is split as needed.
4.2 The characterization of K
A function is approximable from above, or right-c.e., if there is a computable function , non-increasing in , with . As noted after Theorem 3.3, is right-c.e., and by Proposition 3.4 (4) it satisfies .
If is right-c.e. and , then . So is, up to an additive constant, the least right-c.e. function satisfying the Kraft inequality.
Let be an approximation from above. Enumerate the request for each , and the request whenever . The requests concerning have pairwise distinct lengths , with ranging over the values taken by , which are at least . Their weight is at most . The total weight is at most . By Theorem 4.2, , since the request is enumerated when the approximation reaches its limit.
4.3 The coding theorem
For a prefix-free machine , the output probability of is
It is the probability that outputs when its input is produced by tossing a fair coin, the machine reading bits as needed. By Proposition 3.2, . Clearly , the contribution of a shortest description.
For every prefix-free machine , for every with . In particular .
Let be the sum over the descriptions of such that and ; it is computable, non-decreasing in , with limit . For each and , enumerate the request when for the first time. Fix with , and let be the least with ; then , since . The requests about are those with , of total weight . So the request set is bounded by . By Theorem 4.2, . By minimality of , , so , and .
The theorem says that a string that is produced with large probability has a short description. Equivalently, up to a multiplicative constant, the shortest description accounts for the whole output probability of a string: .
There is a constant such that for all and , fewer than strings of length satisfy .
Let be the prefix-free machine , with the domain of . Then . By the coding theorem, for a constant . Let be the number of strings of length with . Each contributes more than to , so , and .
With , fewer than half of the strings of length have . This proves the second part of Corollary 3.5: most strings of length have complexity , the maximal value.
4.4 Chaitin’s halting probability
The halting probability of is
By Proposition 3.2, , and since has a non-empty domain. The theorem below shows that is irrational, so and has a unique binary expansion , with ; we identify with the sequence . The rationals form a computable non-decreasing sequence with limit : is left-c.e.
- .
- is irrational, and for every , the prefix determines, uniformly in , the set of strings of length at most in the domain of .
- There is a constant with for every .
Settled stages. If and , then every string of length at most in the domain of satisfies : otherwise is not counted in , and . So the strings of length at most in the domain of are those that converge by stage .
The domain of computes . Let be the prefix-free machine with , where is the least stage with , and undefined elsewhere. Let be such that . If converges at stage , then . With the domain of as an oracle, compute the outputs of the strings of length at most in the domain, and let be the largest number with among them ( if there is none). Then if and only if .
is irrational. If were rational, then, given , the search for a stage with would be effective, since and are rationals, and it terminates since tends to . By the first paragraph, the domain of would be computable, and so would be , which is not the case.
Prefixes of . Let . Since is irrational, . Search for with ; it exists, and then , so is settled for . This procedure only uses , which proves 2. In particular computes the domain of , hence .
. For a dyadic rational , holds if and only if for some , a c.e. condition, so decides it; and , since is irrational. The prefix is the binary expansion of the largest with , so computes . This proves 1.
Incompressibility. Let be the machine that, on input , computes and , runs the procedure of 2 as if were , obtaining a stage if the search terminates, lists the outputs of the strings of length at most that converge by stage , and outputs the first string, in the length-lexicographic order, that is not in this list. The domain of is contained in that of , so is prefix-free. If , the list contains every string with , by 2, so has complexity larger than . Let be such that , and let be a shortest description of . Then .
Statement 3 says that is random in the sense announced in the previous lesson. The first bits of solve the halting problem for all programs of length at most , and yet they cannot be compressed: up to a constant, no prefix-free description of them is shorter than .
4.5 Exercises
Show that there is a constant such that every string has at most descriptions with , for every .
Solution
By the coding theorem, . Each description with contributes at least to . So there are at most of them.
Let be computable with . Show that . Deduce that for , and show that for infinitely many .
Solution
Let be such that . The computable function is right-c.e. and satisfies the Kraft inequality, so by Theorem 4.3. The series converges, which gives the second bound with for . If for all , then , which diverges, contradicting Proposition 3.4 (4).
4.6 Notes and references
The KC theorem was proved by Levin (1971) and Schnorr (1973), and by Chaitin (1975), who gave it its usual form; the name stands for Kraft and Chaitin. The coding theorem is due to Levin (1974) and Chaitin (1975). Chaitin (1975) introduced ; Solovay, Calude, Hertling, Khoussainov, Wang, Kučera and Slaman showed that the left-c.e. reals that are random in the sense of the next course are exactly the halting probabilities of optimal prefix-free machines.
- L. A. Levin, Laws of information conservation (nongrowth) and aspects of the foundation of probability theory, Problems of Information Transmission 10 (1974), 206–210.
- G. J. Chaitin, A theory of program size formally identical to information theory, Journal of the ACM 22 (1975), 329–340.
- C. P. Schnorr, Process complexity and effective random tests, Journal of Computer and System Sciences 7 (1973), 376–388.
- A. Kučera and T. A. Slaman, Randomness and recursive enumerability, SIAM Journal on Computing 31 (2001), 199–211.