Cite

This course has no released version yet: the text at this address may change. Cite a released version when one exists.

/courses/kolmogorov-complexity/coding-theorem
@misc{kolmogorov-complexity-coding-theorem,
  title = {{The KC theorem and the coding theorem}},
  howpublished = {Lesson 4 of \emph{Kolmogorov complexity}, Computability Theory course notes},
  url = {/courses/kolmogorov-complexity/coding-theorem},
  note = {Unreleased working version}
}

To cite a single statement, use the # link that appears next to its heading.

Lesson 4 of 4

The KC theorem and the coding theorem

Building prefix-free machines from bounded request sets; K is the least right-c.e. function satisfying the Kraft inequality; the coding theorem identifies K with the logarithm of the halting probability of a string; counting strings of small complexity; Chaitin's number Ω.

45 min4 questions
Before this lesson:Prefix-free complexity
Kolmogorov complexity · lesson 4 of 4
  1. Plain Kolmogorov complexity
  2. Complexity of initial segments
  3. Prefix-free complexity
  4. The KC theorem and the coding theorem

Constructing a prefix-free machine by hand requires keeping its domain prefix-free, which is often tedious. The KC theorem, an effective form of the Kraft inequality, removes this difficulty: to obtain descriptions of prescribed lengths, it suffices to enumerate requests whose total weight is at most 11. It yields a characterization of KK as the least function, up to a constant, that is approximable from above and satisfies the Kraft inequality, and the coding theorem, which identifies 2−K(σ)2^{-K(\sigma)} with the probability that the optimal machine outputs σ\sigma on a random input. The lesson ends with Chaitin’s halting probability Ω\Omega.

4.1 Bounded request sets

Definition 4.1.#

A request is a pair (σ,ℓ)∈2<N×N(\sigma, \ell) \in \bstr \times \nat, asking for a description of σ\sigma of length ℓ\ell. A set of requests RR is bounded if its weight ∑(σ,ℓ)∈R2−ℓ\sum_{(\sigma, \ell) \in R} 2^{-\ell} is at most 11. A prefix-free machine MM satisfies RR if for every request (σ,ℓ)∈R(\sigma, \ell) \in R there is a string τ\tau of length ℓ\ell with M(τ)=σM(\tau) = \sigma, these strings τ\tau being distinct for distinct requests.

Theorem 4.2 (KC theorem; Levin, Schnorr, Chaitin).#

For every c.e. bounded request set RR, there is a prefix-free machine MM that satisfies RR. In particular K(σ)≤+ℓK(\sigma) \leq^+ \ell for every (σ,ℓ)∈R(\sigma, \ell) \in R. Moreover, an index of MM is computable from an index of RR.

Proof of Theorem 4.2.

Enumerate RR as (σ0,ℓ0),(σ1,ℓ1),…(\sigma_0, \ell_0), (\sigma_1, \ell_1), \dots, one request per stage (if RR is finite, the construction stops). We assign to the request (σs,ℓs)(\sigma_s, \ell_s) a string τs\tau_s of length ℓs\ell_s, and set M(τs)=σsM(\tau_s) = \sigma_s. Along the construction we maintain a finite set FsF_s of free strings, with the following invariants:

  1. the strings τ0,…,τs−1\tau_0, \dots, \tau_{s-1} and the elements of FsF_s form a prefix-free set, the τi\tau_i being pairwise distinct;
  2. the elements of FsF_s have pairwise distinct lengths;
  3. ∑ρ∈Fs2−∣ρ∣=1−∑i<s2−ℓi\sum_{\rho \in F_s} 2^{-|\rho|} = 1 - \sum_{i < s} 2^{-\ell_i}.

Initially F0={ε}F_0 = \{ \varepsilon \} and no string is assigned; the invariants hold.

At stage ss, the weight of RR gives 2−ℓs≤1−∑i<s2−ℓi=∑ρ∈Fs2−∣ρ∣2^{-\ell_s} \leq 1 - \sum_{i < s} 2^{-\ell_i} = \sum_{\rho \in F_s} 2^{-|\rho|}. Some free string has length at most ℓs\ell_s: otherwise, by invariant 2, the free strings have distinct lengths larger than ℓs\ell_s, and their weight is less than ∑j>ℓs2−j=2−ℓs\sum_{j > \ell_s} 2^{-j} = 2^{-\ell_s}. Let ρ\rho be the longest free string with ∣ρ∣≤ℓs|\rho| \leq \ell_s, and k=ℓs−∣ρ∣k = \ell_s - |\rho|. Let τs=ρ 0k\tau_s = \rho\, 0^{k}, and

Fs+1=(Fs∖{ρ})∪{ρ 0j1:0≤j<k}.F_{s+1} = (F_s \setminus \{ \rho \}) \cup \{ \rho\, 0^j 1 : 0 \leq j < k \}.

The new free strings have the lengths ∣ρ∣+1,…,ℓs|\rho| + 1, \dots, \ell_s; no element of FsF_s had one of these lengths, by the choice of ρ\rho, so invariant 2 is preserved. The strings τs\tau_s and ρ 0j1\rho\, 0^j 1, j<kj < k, are pairwise incomparable extensions of ρ\rho, and ρ\rho was incomparable with the other strings of invariant 1; so invariant 1 is preserved. Finally 2−∣ρ∣=2−ℓs+∑j<k2−(∣ρ∣+j+1)2^{-|\rho|} = 2^{-\ell_s} + \sum_{j < k} 2^{-(|\rho| + j + 1)}, so invariant 3 is preserved.

The construction is effective, so MM is a machine; its domain {τs}\{ \tau_s \} is prefix-free by invariant 1, and it satisfies RR. The construction is uniform in an index of RR. The bound on KK follows from optimality: K≤KM+cMK \leq K_M + c_M.

The free strings record the unused part of Cantor space as a union of disjoint cylinders of distinct sizes, like the binary expansion of the remaining weight. A request is served by the smallest free cylinder that is large enough, which is split as needed.

ρother stringsρ1: freeρ01: freeρ00: assignedfree stringassigned at this stageunaffected

A stage of the KC construction. The longest free string of length at most 44 is ρ\rho, of length 22. A request of length 44 receives ρ00\rho 00, and ρ1\rho 1, ρ01\rho 01 replace ρ\rho among the free strings.

4.2 The characterization of K

A function f:2<N→Nf : \bstr \to \nat is approximable from above, or right-c.e., if there is a computable function (s,σ)↦fs(σ)(s, \sigma) \mapsto f_s(\sigma), non-increasing in ss, with lim⁡sfs(σ)=f(σ)\lim_s f_s(\sigma) = f(\sigma). As noted after Theorem 3.3, KK is right-c.e., and by Proposition 3.4 (4) it satisfies ∑σ2−K(σ)≤1\sum_\sigma 2^{-K(\sigma)} \leq 1.

Theorem 4.3.#

If f:2<N→Nf : \bstr \to \nat is right-c.e. and ∑σ2−f(σ)≤1\sum_\sigma 2^{-f(\sigma)} \leq 1, then K≤+fK \leq^+ f. So KK is, up to an additive constant, the least right-c.e. function satisfying the Kraft inequality.

Proof of Theorem 4.3.

Let (fs)(f_s) be an approximation from above. Enumerate the request (σ,f0(σ)+1)(\sigma, f_0(\sigma) + 1) for each σ\sigma, and the request (σ,fs+1(σ)+1)(\sigma, f_{s+1}(\sigma) + 1) whenever fs+1(σ)<fs(σ)f_{s+1}(\sigma) < f_s(\sigma). The requests concerning σ\sigma have pairwise distinct lengths v+1v + 1, with vv ranging over the values taken by fs(σ)f_s(\sigma), which are at least f(σ)f(\sigma). Their weight is at most ∑v≥f(σ)2−v−1=2−f(σ)\sum_{v \geq f(\sigma)} 2^{-v-1} = 2^{-f(\sigma)}. The total weight is at most ∑σ2−f(σ)≤1\sum_\sigma 2^{-f(\sigma)} \leq 1. By Theorem 4.2, K(σ)≤+f(σ)+1K(\sigma) \leq^+ f(\sigma) + 1, since the request (σ,f(σ)+1)(\sigma, f(\sigma) + 1) is enumerated when the approximation reaches its limit.

4.3 The coding theorem

For a prefix-free machine MM, the output probability of σ\sigma is

PM(σ)=∑τ:M(τ)=σ2−∣τ∣.P_M(\sigma) = \sum_{\tau : M(\tau) = \sigma} 2^{-|\tau|}.

It is the probability that MM outputs σ\sigma when its input is produced by tossing a fair coin, the machine reading bits as needed. By Proposition 3.2, ∑σPM(σ)≤1\sum_\sigma P_M(\sigma) \leq 1. Clearly PU(σ)≥2−K(σ)P_U(\sigma) \geq 2^{-K(\sigma)}, the contribution of a shortest description.

Theorem 4.4 (Coding theorem; Levin, Chaitin).#

For every prefix-free machine MM, K(σ)≤+−log⁡PM(σ)K(\sigma) \leq^+ -\log P_M(\sigma) for every σ\sigma with PM(σ)>0P_M(\sigma) > 0. In particular K(σ)=+−log⁡PU(σ)K(\sigma) =^+ -\log P_U(\sigma).

Proof of Theorem 4.4.

Let PM,s(σ)P_{M,s}(\sigma) be the sum over the descriptions τ\tau of σ\sigma such that ∣τ∣≤s|\tau| \leq s and M(τ)[s] ⁣ ⁣↓M(\tau)[s]\converge; it is computable, non-decreasing in ss, with limit PM(σ)P_M(\sigma). For each k≥1k \geq 1 and σ\sigma, enumerate the request (σ,k+1)(\sigma, k + 1) when PM,s(σ)>2−kP_{M,s}(\sigma) > 2^{-k} for the first time. Fix σ\sigma with PM(σ)>0P_M(\sigma) > 0, and let k0k_0 be the least kk with PM(σ)>2−kP_M(\sigma) > 2^{-k}; then k0≥1k_0 \geq 1, since PM(σ)≤1P_M(\sigma) \leq 1. The requests about σ\sigma are those with k≥k0k \geq k_0, of total weight ∑k≥k02−k−1=2−k0<PM(σ)\sum_{k \geq k_0} 2^{-k-1} = 2^{-k_0} < P_M(\sigma). So the request set is bounded by ∑σPM(σ)≤1\sum_\sigma P_M(\sigma) \leq 1. By Theorem 4.2, K(σ)≤+k0+1K(\sigma) \leq^+ k_0 + 1. By minimality of k0k_0, PM(σ)≤2−(k0−1)P_M(\sigma) \leq 2^{-(k_0 - 1)}, so k0≤1−log⁡PM(σ)k_0 \leq 1 - \log P_M(\sigma), and K(σ)≤+−log⁡PM(σ)K(\sigma) \leq^+ -\log P_M(\sigma).

The theorem says that a string that is produced with large probability has a short description. Equivalently, up to a multiplicative constant, the shortest description accounts for the whole output probability of a string: PU(σ)≤2c−K(σ)P_U(\sigma) \leq 2^{c - K(\sigma)}.

Theorem 4.5 (Counting theorem; Chaitin).#

There is a constant cc such that for all nn and kk, fewer than 2n−k+c2^{n - k + c} strings σ\sigma of length nn satisfy K(σ)<n+K(n)−kK(\sigma) < n + K(n) - k.

Proof of Theorem 4.5.

Let MM be the prefix-free machine τ↦∣U(τ)∣\tau \mapsto |U(\tau)|, with the domain of UU. Then PM(n)=∑∣σ∣=nPU(σ)≥∑∣σ∣=n2−K(σ)P_M(n) = \sum_{|\sigma| = n} P_U(\sigma) \geq \sum_{|\sigma| = n} 2^{-K(\sigma)}. By the coding theorem, 2−K(n)≥2−cPM(n)2^{-K(n)} \geq 2^{-c} P_M(n) for a constant cc. Let NN be the number of strings σ\sigma of length nn with K(σ)<n+K(n)−kK(\sigma) < n + K(n) - k. Each contributes more than 2−n−K(n)+k2^{-n - K(n) + k} to PM(n)P_M(n), so N 2−n−K(n)+k<PM(n)≤2c−K(n)N\, 2^{-n - K(n) + k} < P_M(n) \leq 2^{c - K(n)}, and N<2n−k+cN < 2^{n - k + c}.

With k=c+1k = c + 1, fewer than half of the strings of length nn have K(σ)<n+K(n)−c−1K(\sigma) < n + K(n) - c - 1. This proves the second part of Corollary 3.5: most strings of length nn have complexity n+K(n)−O(1)n + K(n) - O(1), the maximal value.

4.4 Chaitin’s halting probability

Definition 4.6.#

The halting probability of UU is

Ω=∑τ:U(τ) ⁣ ⁣↓2−∣τ∣=∑σPU(σ).\Omega = \sum_{\tau : U(\tau)\converge} 2^{-|\tau|} = \sum_\sigma P_U(\sigma).

By Proposition 3.2, Ω≤1\Omega \leq 1, and Ω>0\Omega > 0 since UU has a non-empty domain. The theorem below shows that Ω\Omega is irrational, so 0<Ω<10 < \Omega < 1 and Ω\Omega has a unique binary expansion ω=ω0ω1⋯\omega = \omega_0 \omega_1 \cdots, with Ω=∑iωi2−i−1\Omega = \sum_i \omega_i 2^{-i-1}; we identify Ω\Omega with the sequence ω\omega. The rationals Ωs=∑{2−∣τ∣:∣τ∣≤s, U(τ)[s] ⁣ ⁣↓}\Omega_s = \sum \{ 2^{-|\tau|} : |\tau| \leq s,\ U(\tau)[s]\converge \} form a computable non-decreasing sequence with limit Ω\Omega: Ω\Omega is left-c.e.

Theorem 4.7 (Chaitin).#
  1. Ω≡T∅′\Omega \equiv_T \emptyset'.
  2. Ω\Omega is irrational, and for every nn, the prefix ω ⁣↾n\omega \uh n determines, uniformly in nn, the set of strings of length at most nn in the domain of UU.
  3. There is a constant cc with K(ω ⁣↾n)≥n−cK(\omega \uh n) \geq n - c for every nn.
Proof of Theorem 4.7.

Settled stages. If Ω−Ωs<2−n\Omega - \Omega_s < 2^{-n} and s≥ns \geq n, then every string τ\tau of length at most nn in the domain of UU satisfies U(τ)[s] ⁣ ⁣↓U(\tau)[s]\converge: otherwise τ\tau is not counted in Ωs\Omega_s, and Ω≥Ωs+2−∣τ∣≥Ωs+2−n\Omega \geq \Omega_s + 2^{-|\tau|} \geq \Omega_s + 2^{-n}. So the strings of length at most nn in the domain of UU are those that converge by stage ss.

The domain of UU computes ∅′\emptyset'. Let MM be the prefix-free machine with M(1e0)=0sM(1^e 0) = 0^s, where ss is the least stage with Φe(e)[s] ⁣ ⁣↓\Phi_e(e)[s]\converge, and MM undefined elsewhere. Let dd be such that K≤KM+dK \leq K_M + d. If Φe(e)\Phi_e(e) converges at stage ss, then K(0s)≤e+1+dK(0^s) \leq e + 1 + d. With the domain of UU as an oracle, compute the outputs of the strings of length at most e+1+de + 1 + d in the domain, and let tt be the largest number with 0t0^t among them (t=0t = 0 if there is none). Then Φe(e) ⁣ ⁣↓\Phi_e(e)\converge if and only if Φe(e)[t] ⁣ ⁣↓\Phi_e(e)[t]\converge.

Ω\Omega is irrational. If Ω\Omega were rational, then, given nn, the search for a stage s≥ns \geq n with Ω−Ωs<2−n\Omega - \Omega_s < 2^{-n} would be effective, since Ω\Omega and Ωs\Omega_s are rationals, and it terminates since Ωs\Omega_s tends to Ω\Omega. By the first paragraph, the domain of UU would be computable, and so would be ∅′\emptyset', which is not the case.

Prefixes of ω\omega. Let q=0.ω0⋯ωn−1q = 0.\omega_0 \cdots \omega_{n-1}. Since Ω\Omega is irrational, q<Ω<q+2−nq < \Omega < q + 2^{-n}. Search for s≥ns \geq n with Ωs>q\Omega_s > q; it exists, and then Ω−Ωs<2−n\Omega - \Omega_s < 2^{-n}, so ss is settled for nn. This procedure only uses ω ⁣↾n\omega \uh n, which proves 2. In particular ω\omega computes the domain of UU, hence ∅′\emptyset'.

ω≤T∅′\omega \leq_T \emptyset'. For a dyadic rational qq, q<Ωq < \Omega holds if and only if Ωs>q\Omega_s > q for some ss, a c.e. condition, so ∅′\emptyset' decides it; and q≠Ωq \neq \Omega, since Ω\Omega is irrational. The prefix ω ⁣↾k\omega \uh k is the binary expansion of the largest q=j2−kq = j 2^{-k} with q<Ωq < \Omega, so ∅′\emptyset' computes ω\omega. This proves 1.

Incompressibility. Let NN be the machine that, on input τ\tau, computes ρ=U(τ)\rho = U(\tau) and n=∣ρ∣n = |\rho|, runs the procedure of 2 as if ρ\rho were ω ⁣↾n\omega \uh n, obtaining a stage ss if the search terminates, lists the outputs of the strings of length at most nn that converge by stage ss, and outputs the first string, in the length-lexicographic order, that is not in this list. The domain of NN is contained in that of UU, so NN is prefix-free. If U(τ)=ω ⁣↾nU(\tau) = \omega \uh n, the list contains every string σ\sigma with K(σ)≤nK(\sigma) \leq n, by 2, so N(τ)N(\tau) has complexity larger than nn. Let cc be such that K≤KN+cK \leq K_N + c, and let τ\tau be a shortest description of ω ⁣↾n\omega \uh n. Then n<K(N(τ))≤∣τ∣+c=K(ω ⁣↾n)+cn < K(N(\tau)) \leq |\tau| + c = K(\omega \uh n) + c.

Statement 3 says that ω\omega is random in the sense announced in the previous lesson. The first nn bits of Ω\Omega solve the halting problem for all programs of length at most nn, and yet they cannot be compressed: up to a constant, no prefix-free description of them is shorter than nn.

4.5 Exercises

Exercise 4.8 ★.

Show that there is a constant cc such that every string σ\sigma has at most 2c+d2^{c+d} descriptions τ\tau with ∣τ∣≤K(σ)+d|\tau| \leq K(\sigma) + d, for every dd.

Solution

By the coding theorem, PU(σ)≤2c−K(σ)P_U(\sigma) \leq 2^{c - K(\sigma)}. Each description τ\tau with ∣τ∣≤K(σ)+d|\tau| \leq K(\sigma) + d contributes at least 2−K(σ)−d2^{-K(\sigma) - d} to PU(σ)P_U(\sigma). So there are at most 2c−K(σ)/2−K(σ)−d=2c+d2^{c - K(\sigma)} / 2^{-K(\sigma) - d} = 2^{c + d} of them.

Exercise 4.9 ★.

Let f:N→Nf : \nat \to \nat be computable with ∑n2−f(n)<∞\sum_n 2^{-f(n)} < \infty. Show that K(n)≤+f(n)K(n) \leq^+ f(n). Deduce that K(n)≤+log⁡n+2log⁡log⁡nK(n) \leq^+ \log n + 2 \log \log n for n≥4n \geq 4, and show that K(n)>log⁡n+log⁡log⁡nK(n) > \log n + \log \log n for infinitely many nn.

Solution

Let cc be such that ∑n2−f(n)−c≤1\sum_n 2^{-f(n) - c} \leq 1. The computable function f+cf + c is right-c.e. and satisfies the Kraft inequality, so K≤+f+cK \leq^+ f + c by Theorem 4.3. The series ∑n≥41/(n(log⁡n)2)\sum_{n \geq 4} 1/(n (\log n)^2) converges, which gives the second bound with f(n)=⌈log⁡n+2log⁡log⁡n⌉f(n) = \lceil \log n + 2 \log \log n \rceil for n≥4n \geq 4. If K(n)≤log⁡n+log⁡log⁡nK(n) \leq \log n + \log \log n for all n≥n0n \geq n_0, then ∑n2−K(n)≥∑n≥n01/(nlog⁡n)\sum_n 2^{-K(n)} \geq \sum_{n \geq n_0} 1/(n \log n), which diverges, contradicting Proposition 3.4 (4).

4.6 Notes and references

The KC theorem was proved by Levin (1971) and Schnorr (1973), and by Chaitin (1975), who gave it its usual form; the name stands for Kraft and Chaitin. The coding theorem is due to Levin (1974) and Chaitin (1975). Chaitin (1975) introduced Ω\Omega; Solovay, Calude, Hertling, Khoussainov, Wang, Kučera and Slaman showed that the left-c.e. reals that are random in the sense of the next course are exactly the halting probabilities of optimal prefix-free machines.

  • L. A. Levin, Laws of information conservation (nongrowth) and aspects of the foundation of probability theory, Problems of Information Transmission 10 (1974), 206–210.
  • G. J. Chaitin, A theory of program size formally identical to information theory, Journal of the ACM 22 (1975), 329–340.
  • C. P. Schnorr, Process complexity and effective random tests, Journal of Computer and System Sciences 7 (1973), 376–388.
  • A. Kučera and T. A. Slaman, Randomness and recursive enumerability, SIAM Journal on Computing 31 (2001), 199–211.