Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Large Language Models: In this repository Language models are introduced covering both theoretical and practical aspects.
| Date | Stars |
|---|---|
| 2026-07-31 | 392 |
| 2026-08-01 | 392 |
| 2026-08-06 | 392 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Large Language Models (LLMs)
_Source_ [A Survey of Large Language Models](https://arxiv.org/pdf/2303.18223.pdf)
## See also
- [LMMs: Large Multimodal Models](https://github.com/IbrahimSobh/llms/tree/main/LMMs)
- [AI Multi-Agent Systems](https://github.com/IbrahimSobh/llms/tree/main/MultiAgents)
- [Agentic Workflows](https://github.com/IbrahimSobh/llms/tree/main/Agentic%20Workflows)
## Content
- What is a language model?
- Applications of language models
- Statistical Language Modeling
- Neural Language Models (NLM)
- Conditional language model
- Evaluation: How good is our model?
- Transformer-based Language models
- Practical LLMs: GPT, BERT, Falcon, Llama, CodeT5
- How to generate text using different decoding methods
- Prompt Engineering
- Fine-tuning LLMs
- Retrieval Augmented Generation (RAG)
- Ask almost everything (txt, pdf, video, etc.)
- Evaluating LLM-based systems
- AI Agents
- LLMs for Computer vision (TBD)
- Further readings
---
## Introduction: What is a language model?
Simple definition: Language Modeling is the task of predicting what word comes next.
"The dog is playing in the ..."
- park
- woods
- snow
- office
- university
- Neural network
- ?
The main purpose of **Language Models** is to assign a probability to a sentence, to distinguish between the more likely and the less likely sentences.
### Applications of language models:
1. Machine Translation: P(high winds tonight) > P(large winds tonight)
2. Spelling correction: P(about fifteen minutes from) > P(about fifteen minuets from)
3. Speech Recognition: P(I saw a van) > P(eyes awe of an)
4. Authorship identification: who wrote some sample text
5. Summarization, question answering, dialogue bots, etc.
For Speech Recognition, we use not only the acoustics model (the speech signal), but also a language model. Similarly, for Optical Character Recognition (OCR), we use both a vision model and a language model. Language models are very important for such recognition systems.
> Sometimes, you hear or read a sentence that is not clear, but using your language model, you still can recognize it at a high accuracy despite the noisy vision/speech input.
The language model computes either of:
- The probability of an upcoming word: $P(w_5 | w_1, w_2, w_3, w_4)$
- The probability of a sentence or sequence of words (according to the Language Model): $P(w_1, w_2, w_3, ..., w_n)$
> Language Modeling is a subcomponent of many NLP tasks, especially those involving generating text or estimating the probability of text.
The Chain Rule: $P(x_1, x_2, x_3, …, x_n) = P(x_1)P(x_2|x_1)P(x_3|x_1,x_2)…P(x_n|x_1,…,x_{n-1})$
> $P(The, water, is, so, clear) = P(The) × P(water|The) × P(is|The, water) × P(so|The, water, is) × P(clear | The, water, is, so)$
What just happened? The Chain Rule is applied to compute the joint probability of words in a sentence.
---
## Statistical Language Modeling:
### n-gram Language Models
Using a large amount of text (corpus such as Wikipedia), we collect statistics about how frequently different words are, and use these to predict the next word. For example, the probability that a word _w_ comes after these three words *students opened their* can be estimated as follows:
- P(w | students opened their) = count(students opened their w) / count(students opened their)
The above example is a 4-gram model. And we may get:
- P(books | students opened their) = 0.4
- P(cars | students, opened, their) = 0.05
- P(... | students, opened, their) = ...
> We can conclude that the word “books” is more probable than “cars” in this context.
We ignored the previous context before "students opened their"
> Accordingly, arbitrary text can be generated from a language model given starting word(s), by sampling from the output probability distribution of the next word, and so on.
We can train an LM on any kind of text, then generate text in that style (Harry Potter, etc.).
<!--
### How to estimate these probabilitiesExcerpt of 58,844 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:f597db40a1778936, topic:nlp