Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
An Extensible Continual Learning Framework Focused on Language Models (LMs)
| Date | Stars |
|---|---|
| 2026-07-31 | 293 |
| 2026-08-02 | 293 |
| 2026-08-06 | 293 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# ContinualLM
<p align="center">
<br>
<a href="https://github.com/UIC-Liu-Lab//ContinualLM">
<img src="https://github.com/UIC-Liu-Lab/ContinualLM/blob/main/docs/icon.png" width="200"/>
</a>
<figcaption>Imagine an LM that not only effortlessly acquires new knowledge but also retains its mastery of skills, all while successfully transferring knowledge. Is it even possible?</figcaption>
<br>
<p>
## News
🔥 We have added [checkpoints](#checkpoints-in-huggingface) in Hugging Face for easier reproduction!
🔥 We have added [continual_pretrain.ipynb](https://github.com/UIC-Liu-Lab/ContinualLM/blob/main/continual_pretrain.ipynb) as a **self-contained example** of the soft-masking scenario. It runs well without GPUs!
🔥 Soft-masking can also work in **conventional continual fine-tuning**. Check out our latest [EMNLP23](https://arxiv.org/abs/2310.09436) paper!
🔥 Wondering whether you can adapt a **black-box LLM** without worrying about the update of its parameters? Check out our latest paper on retrieval-augmented generation (RAG) [here](https://arxiv.org/abs/2401.06954)!
## Quick Links
- [Introduction](#introduction)
- [Simple Example](#simple-example)
- [Dataset](#dataset)
- [Architecture](#architecture)
- [Installation](#installation)
- [Domain-adaptive Pre-training](#domain-adaptive-pre-training)
- [End-task Fine-tuning](#end-task-fine-tuning)
- [Checkpoints in Huggingface](#checkpoints-in-huggingface)
- [Reference](#reference)
- [Contact](#contact)
## Introduction
In 2021, we introduced [Pycontinual](https://github.com/ZixuanKe/PyContinual), a straightforward and flexible framework for continual learning. Our research has benefited significantly from this framework. Today, we are excited to share the **ContinualLM**, an extensible continual learning framework focused on language models (LMs), designed to sustain the benefits of continual learning (CL) in this field.
Continual learning for LMs is distinct from traditional CL because
- Each task is treated as a **domain-specific corpus** (at present, our primary focus is on domain-adaptive pre-training, which is also known as pre-finetuning or post-training).
- Moreover, the evaluation process involves **fine-tuning** the corresponding end-task.
Our repository includes a PyTorch implementation of a collection of state-of-the-art (SoTA) methods, using the same training and evaluation pipeline. This repository is committed to advancing the field of continual learning for LMs. The methods included are:
* From our group:
* **DAS**: [Continual Learning of Language Models](https://arxiv.org/abs/2210.05549), ICLR 2023
* **CPT**: [Continual Training of Language Models for Few-Shot Learning](https://arxiv.org/abs/2210.05549), EMNLP 2022
* **DGA**: [Adapting a Language Model While Preserving its General Knowledge](https://arxiv.org/abs/2301.08986), EMNLP 2022
* **CTR**: [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://proceedings.neurips.cc/paper/2021/hash/bcd0049c35799cdf57d06eaf2eb3cff6-Abstract.html), NeurIPS 2021
* **CLASSIC**: [CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks](https://aclanthology.org/2021.emnlp-main.550/), EMNLP 2021
* **B-CL**: [Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks](https://www.aclweb.org/anthology/2021.naacl-main.378.pdf), NAACL 2021
* From other groups **(more to come)**:
* **DEMIX**: [Demix layers: Disentangling domains for modular language modeling](https://aclanthology.org/2022.naacl-main.407);, Gururangan et al., NAACL 2022)
* **EWC**: [Overcoming catastrophic forgetting in neural networks](https://arxiv.org/abs/1612.00796), Kirkpatrick et al., PNAS 2017
* **DER++**: [Dark experience for general continual learning: a strong,Excerpt of 12,281 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:4d00c6951c9bbaa5, topic:language-model
matched fp:4d00c6951c9bbaa5, topic:natural-language-processing