"There is a belief that learning to compress well will lead to intelligence. Recently, language modeling has been shown to be equivalent to compression, which offers a compelling rationale for the success of large language models (LLMs): the development of more advanced language models is essentially enhancing compression which facilitates intelligence. Despite such appealing discussions, little empirical evidence is present for the interplay between compression and intelligence. In this work, we examine their relationship in the context of LLMs, treating LLMs as data compressors."

Source

This paper tries to correlate two metrics:

1. Performance on "average benchmark scores" as "Intelligence;"
2. Ability to compress language.

The first metric is self explanatory. The second is much more difficult to understand. How do we measure a LLM's ability to compress language?

• The authors propose using the bit's per character and propose the formula:

"Intuitively, we seek to utilize pmodel(x) to encode rare sequences with more bits and the frequent ones with fewer bits."

Due to the variety of tokenizers employed by different LLMs, the average bits per token are not directly comparable. Therefore, we utilize the average bits per character (BPC) as the metric.

But the question remains: how do we know the "bits" per character for a LLM? It's not like this a readily available by examining the generated text. What are the authors talking about. 

We need a bit more context here. Let's look at an earlier paper that developed this idea more:

• We start with a piece of text like 

• My first attempt at writing a book

• We use the first 5 words (or tokens) to try to predict the next:

• The correct term is at 0, we store that value = 0
• Then we move forward one word:

• The correct term is at 0, we store that value = 0

• A perfect LLM will always predict the next word with perfect accuracy
• The indexes will always be 0

• [0, 0, 0, 0, 0, 0, .... ]

• This array of indexes can be compressed maximally giving a the smallest compression.

• A terrible LLM will do a terrible job predicting

• These values will be incredibly hard to compress without loss
• The bits will not be much less than the original string itself. Bits = Words

• There is an old idea in AI that compression is intelligence, or, at least, highly correlates with intelligence.
• Claude Shannon introduced Information Theory and published much about compressing the English language efficiently.
• Kolmogorov Complexity. In the 1960s, Andrei Kolmogorov and Gregory Chaitin developed concepts around the complexity of a string defined as the length of the shortest possible description of that string within a fixed computational model. This idea suggests that the more you can compress a dataset (i.e., the shorter the description), the more you understand the underlying patterns and structure of the data.

• As a thought experiment, let's assume that we have a giant database, of every sentence every written.
• Compression of any sentence now becomes a single digit number