Large Language Models (llms)

Source A Survey of Large Language Models

Content

What is a language model?
Applications of language models
Statistical Language Modeling
Neural Language Models (NLM)
Conditional language model
Evaluation: How good is our model?
Transformer-based Language models
Practical LLMs: GPT, BERT, Falcon, Llama, CodeT5
How to generate text using different decoding methods
Prompt Engineering
Fine-tuning LLMs
Retrieval Augmented Generation (RAG)
Ask almost everything (txt, pdf, video, etc.)
Evaluating LLM-based systems
AI Agents
LLMs for Computer vision (TBD)
Further readings

Introduction: What is a language model?

Simple definition: Language Modeling is the task of predicting what word comes next.

"The dog is playing in the ..."

park
woods
snow
office
university
Neural network
?

The main purpose of Language Models is to assign a probability to a sentence, to distinguish between the more likely and the less likely sentences.

Applications of language models:

Machine Translation: P(high winds tonight) > P(large winds tonight)
Spelling correction: P(about fifteen minutes from) > P(about fifteen minuets from)
Speech Recognition: P(I saw a van) > P(eyes awe of an)
Authorship identification: who wrote some sample text
Summarization, question answering, dialogue bots, etc.

For Speech Recognition, we use not only the acoustics model (the speech signal), but also a language model. Similarly, for Optical Character Recognition (OCR), we use both a vision model and a language model. Language models are very important for such recognition systems.

Sometimes, you hear or read a sentence that is not clear, but using your language model, you still can recognize it at a high accuracy despite the noisy vision/speech input.

The language model computes either of:

The probability of an upcoming word: $P(w_5 | w_1, w_2, w_3, w_4)$
The probability of a sentence or sequence of words (according to the Language Model): $P(w_1, w_2, w_3, ..., w_n)$

Language Modeling is a subcomponent of many NLP tasks, especially those involving generating text or estimating the probability of text.

The Chain Rule: $P(x_1, x_2, x_3, …, x_n) = P(x_1)P(x_2|x_1)P(x_3|x_1,x_2)…P(x_n|x_1,…,x_{n-1})$

$P(The, water, is, so, clear) = P(The) × P(water|The) × P(is|The, water) × P(so|The, water, is) × P(clear | The, water, is, so)$

What just happened? The Chain Rule is applied to compute the joint probability of words in a sentence.

Statistical Language Modeling:

n-gram Language Models

Using a large amount of text (corpus such as Wikipedia), we collect statistics about how frequently different words are, and use these to predict the next word. For example, the probability that a word w comes after these three words students opened their can be estimated as follows:

P(w | students opened their) = count(students opened their w) / count(students opened their)

The above example is a 4-gram model. And we may get:

P(books | students opened their) = 0.4
P(cars | students, opened, their) = 0.05
P(... | students, opened, their) = ...

We can conclude that the word “books” is more probable than “cars” in this context.

We ignored the previous context before "students opened their"

Accordingly, arbitrary text can be generated from a language model given starting word(s), by sampling from the output probability distribution of the next word, and so on.

We can train an LM on any kind of text, then generate text in that style (Harry Potter, etc.).

We can extend to trigrams, 4-grams, 5-grams, and N-grams.

In general, this is an insufficient model of language because the language has long-distance dependencies. However, in practice, these 3,4 grams work well for most of the applications.

Building Statistical Language Models:

Toolkits

SRILM is a toolkit for building and applying statistical language models, primarily for use in speech recognition, statistical tagging and segmentation, and machine translation. It has been under development in the SRI Speech Technology and Research Laboratory since 1995.
KenLM is a fast and scalable toolkit that builds and queries language models.

N-gram Models

Google's N-gram Models Belong to You: Google Research has been using word n-gram models for a variety of R&D projects. Google N-Gram processed 1,024,908,267,229 words of running text and published the counts for all 1,176,470,663 five-word sequences that appear at least 40 times.

The counts of text from the Linguistics Data Consortium LDC are as follows:

File sizes: approx. 24 GB compressed (gzip'ed) text files

Number of tokens:    1,024,908,267,229
Number of sentences:    95,119,665,584
Number of unigrams:         13,588,391
Number of bigrams:         314,843,401
Number of trigrams:        977,069,902
Number of fourgrams:     1,313,818,354
Number of fivegrams:     1,176,470,663

The following is an example of the 4-gram data in this corpus:

serve as the incoming 92
serve as the incubator 99
serve as the independent 794
serve as the index 223
serve as the indication 72
serve as the indicator 120
serve as the indicators 45
serve as the indispensable 111
serve as the indispensible 40

For example, the sequence of the four words "serve as the indication" has been seen in the corpus 72 times.

Limitations of Statistical Language models

Sometimes we do not have enough data to estimate. Increasing n makes sparsity problems worse. Typically we can’t have n bigger than 5.

Sparsity problem 1: count(students opened their w) = 0? Smoothing Solution: Add small 𝛿 to the count for every w in the vocabulary.
Sparsity problem 2: count(students opened their) = 0? Backoff Solution: condition on (opened their) instead.
Storage issue: Need to store the count for all n-grams you saw in the corpus. Increasing n or increasing corpus increases storage size.

Neural Language Models (NLM)

NLM usually (but not always) uses an RNN to learn sequences of words (sentences, paragraphs, … etc) and hence can predict the next word.

Advantages:

Can process variable-length input as the computations for step t use information from many steps back (eg: RNN)
No sparsity problem (can feed any n-gram not seen in the training data)
Model size doesn’t increase for longer input ($W_h, W_e, $), the same weights are applied on every timestep and need to store only the vocabulary word vectors.

As depicted, At each step, we have a probability distribution of the next word over the vocabulary.

Training an NLM:

Use a big corpus of text (a sequence of words such as Wikipedia)
Feed into the NLM (a batch of sentences); compute output distribution for every step. (predict probability dist of every word, given words so far)
Loss function on each step t cross-entropy between predicted probability distribution, and the true next word (one-hot)

Example of long sequence learning:

The writer of the books (is or are)?
Correct answer: The writer of the books is planning a sequel
Syntactic recency: The writer of the books is (correct)
Sequential recency: The writer of the books are (incorrect)

Disadvantages:

Recurrent computation is slow (sequential, one step at a time)
In practice, for long sequences, difficult_ to access information_ from many steps back

Conditional language model

LM can be used to generate text conditions on input (speech, image (OCR), text, etc.) across different applications such as: speech recognition, machine translation, summarization, etc.

Evaluation: How good is our model?

Does our language model prefer good (likely) sentences to bad ones?

Extrinsic evaluation:

For comparing models A and B, put each model in a task (spelling, corrector, speech recognizer, machine translation)
Run the task and compare the accuracy for A and for B
Best evaluation but not practical and time consuming!

Intrinsic evaluation:

Intuition: The best language model is one that best predicts an unseen test set (assigns high probability to sentences).
Perplexity is the standard evaluation metric for Language Models.
Perplexity is defined as the inverse probability of a text, according to the Language Model.
A good language model should give a lower Perplexity for a test text. Specifically, a lower perplexity for a given text means that text has a high probability in the eyes of that Language Model.

The standard evaluation metric for Language Models is perplexity Perplexity is the inverse probability of the test set, normalized by the number of words

Lower perplexity = Better model

Perplexity is related to branch factor: On average, how many things could occur next.

Transformer-based Language models

Instead of RNN, let's use attention Let's use large pre-trained models

What is the problem? One of the biggest challenges in natural language processing (NLP) is the shortage of training data for many distinct tasks. However, modern deep learning-based NLP models improve when trained on millions, or billions, of annotated training examples.
Pre-training is the solution: To help close this gap, a variety of techniques have been developed for training general-purpose language representation models using the enormous amount of unannotated text. The pre-trained model can then be fine-tuned on small data for different tasks like question answering and sentiment analysis, resulting in substantial accuracy improvements compared to training on these datasets from scratch.

The Transformer architecture was proposed in the paper Attention is All You Need, used for the Neural Machine Translation task (NMT), consisting of:

Encoder: Network that encodes the input sequence.
Decoder: Network that generates the output sequences conditioned on the input.

As mentioned in the paper:

"We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely"

The main idea of attention can be summarized as mentioned in the OpenAi's article:

"... every output element is connected to every input element, and the weightings between them are dynamically calculated based upon the circumstances, a process called attention."

Based on this architecture (the vanilla Transformers!), encoder or decoder components can be used alone to enable massive pre-trained generic models that can be fine-tuned for downstream tasks such as text classification, translation, summarization, question answering, etc. For Example:

"Pre-training of Deep Bidirectional Transformers for Language Understanding" BERT is mainly based on the encoder architecture trained on massive text datasets to predict randomly masked words and "is-next sentence" classification tasks.
GPT, on the other hand, is an auto-regressive generative model that is mainly based on the decoder architecture, trained on massive text datasets to predict the next word (unlike BERT, GPT can generate sequences).

These models, BERT and GPT for instance, can be considered as the NLP's ImageNET.

As shown, BERT is deeply bidirectional, OpenAI GPT is unidirectional, and ELMo is shallowly bidirectional.

Pre-trained representations can be:

Context-free: such as word2vec or GloVe that generates a single/fixed word embedding (vector) representation for each word in the vocabulary (independent of the context of that word at test time)
Contextual: generates a representation of each word based on the other words in the sentence.

Contextual Language models can be:

Causal language model (CML): Predict the next token passed on previous ones. (GPT)
Masked language model (MLM): Predict the masked token based on the surrounding contextual tokens (BERT)

💥 Practical LLMs

In this part, we are going to use different large language models

🚀 Hello GPT2

GPT2 (a successor to GPT) is a pre-trained model on English language using a causal language modeling (CLM) objective, trained simply to predict the next