Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Code repository for supporting the paper "Atlas Few-shot Learning with Retrieval Augmented Language Models",(https//arxiv.org/abs/2208.03299)
| Date | Stars |
|---|---|
| 2026-07-31 | 560 |
| 2026-08-06 | 560 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Atlas: Few-shot Learning with Retrieval Augmented Language Models
REPO NO LONGER MAINTAINED, RESEARCH CODE PROVIDED AS IT IS
This repository contains pre-trained models, corpora, indices, and code for pre-training, finetuning, retrieving and evaluating for the paper [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://arxiv.org/pdf/2208.03299.pdf)
Read our [Atlas blog post](https://research.facebook.com/blog/2023/1/atlas-few-shot-learning-with-retrieval-augmented-language-models/) for a quick overview of the project and how to run the code with torchrun (slurm free option).
We jointly pretrain a retrieval-augmented seq2seq language model, comprised of a passage-based dense retriever and a encoder-decoder language model.
We perform evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and
study the impact of the content of the document index, showing that it can easily be updated.
Notably, Atlas reaches over 45% accuracy on Natural Questions using only 64 examples when supplied with wikipedia index from 2018,
outperforming a 540B parameters model by 6% despite having 50x fewer parameters.
Atlas also works very well when finetuned on larger datasets - when finetuned on the full Natural Questions data, Atlas sets a new state-of-the-art of 64%, 8 points higher than the current state of the art.
This repository supports pretraining and finetuning, for *both* large and small datasets. This repository can be supports the following features:
* Training large fusion-in-decoder seq2seq models, tested up to 11B parameters
* Distilling relevance signals from fusion-in-decoder models into dense retrieval models using a variety of different distillation approaches.
* Performing end-to-end retrieval-augmented training over a user-supplied corpus of passages (tested with up to 400M passages, ~40B words) with retrieval-in-the-training-loop
* Support for training on Masked-Language modelling, prefix-language modelling, wikipedia section generation, Open-Domain Question Answering, Multiple Choice Question Answering, Fact checking, and KILT (arbitrary seq2seq tasks can also be supported)
* A fast, parallel distributed GPU-based exact and approximate maximum inner product search for dense vector retrieval
* Support for fast in-place index refreshes
* Various memory optimizations and methods for maintaining fast and accurate retrieval while training retrievers in-the-loop.
* plus more, see the command line arguments or the readme for additional features
## Table of Contents
* [Installation](#installation)
* [Getting Started and Codebase at a Glance](#getting-started-and-codebase-at-a-glance)
* [Available Data and Models for download](#available-data-and-Models-for-download)
* [Corpora](#corpora)
* [Models](#models)
* [Pre-built Indices](#prebuilt-indices)
* [Tasks](#tasks)
* [Basic](#base-task)
* [Masked Language Modelling](#mlm-task)
* [Wikipedia Section Generation](#section-task)
* [Open-Domain Question Answering (e.g. NaturalQuestions, TriviaQA, TempLama)](#qa-task)
* [Multiple Choice Question Answering (e.g. MMLU)](#mcqa-task)
* [Fact Checking](#fever-task)
* [KILT](#kilt-task)
* [Retrieval and Index Details](#retrieval-and-index-details)
* [Flat vs Faiss](#flat-vs-faiss)
* [Index Saving and Loading](#index-saving-and-loading)
* [Strategies for dealing with stale indices](#strategies-for-dealing-with-stale-indices)
* [Index Refresh](#strategies-for-dealing-with-stale-indices)
* [Over-Retrieve with Reranking](#strategies-for-dealing-with-stale-indices)
* [Query-Side Finetuning](#strategies-for-dealing-with-stale-indices)
* [Retrieve-only mode](#retrieve-only-mode)
* [Using pre-retrieved or cached passages](#using-pre-retrieved-or-cached-passages)
* [Other features](#other-features)
* [Closed book mode](#closed-book-mode)
* [Specifying formats](#specifying-formats)
* [Implementing your own task](#implementing-your-own-task)
* [Full list of Excerpt of 73,355 characters
Read on GitHub
10
1
1
1
1
1
Pasquale Minervini · @EdinburghNLP · United Kingdom
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:4bfd973d93f37ad5, desc:retrieval augmented