Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[ACL 2021] LM-BFF: Better Few-shot Fine-tuning of Language Models https://arxiv.org/abs/2012.15723
| Date | Stars |
|---|---|
| 2026-07-31 | 727 |
| 2026-08-02 | 727 |
| 2026-08-06 | 727 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# LM-BFF (**B**etter **F**ew-shot **F**ine-tuning of **L**anguage **M**odels)
This is the implementation of the paper [Making Pre-trained Language Models Better Few-shot Learners](https://arxiv.org/pdf/2012.15723.pdf). LM-BFF is short for **b**etter **f**ew-shot **f**ine-tuning of **l**anguage **m**odels.
## Quick links
* [Overview](#overview)
* [Requirements](#requirements)
* [Prepare the data](#prepare-the-data)
* [Run the model](#run-lm-bff)
* [Quick start](#quick-start)
* [Experiments with multiple runs](#experiments-with-multiple-runs)
* [Using demonstrations with filtering](#using-demonstrations-with-filtering)
* [Automatically searched prompt](#automatically-searched-prompt)
* [Ensemble](#ensemble-model)
* [Zero-shot experiments](#zero-shot-experiments)
* [How to design your own templates](#how-to-design-your-own-templates)
* [Citation](#citation)
## Overview
In this work we present LM-BFF, a suite of simple and complementary techniques for fine-tuning pre-trained language models on a small number of training examples. Our approach includes:
1. Prompt-based fine-tuning together with a novel pipeline for automating prompt generation.
2. A refined strategy for incorporating demonstrations into context.
You can find more details of this work in our [paper](https://arxiv.org/pdf/2012.15723.pdf).
## Requirements
To run our code, please install all the dependency packages by using the following command:
```
pip install -r requirements.txt
```
**NOTE**: Different versions of packages (like `pytorch`, `transformers`, etc.) may lead to different results from the paper. However, the trend should still hold no matter what versions of packages you use.
## Prepare the data
We pack the original datasets (SST-2, SST-5, MR, CR, MPQA, Subj, TREC, CoLA, MNLI, SNLI, QNLI, RTE, MRPC, QQP, STS-B) [here](https://nlp.cs.princeton.edu/projects/lm-bff/datasets.tar). Please download it and extract the files to `./data/original`, or run the following commands:
```bash
cd data
bash download_dataset.sh
```
Then use the following command (in the root directory) to generate the few-shot data we need:
```bash
python tools/generate_k_shot_data.py
```
See `tools/generate_k_shot_data.py` for more options. For results in the paper, we use the default options: we take `K=16` and take 5 different seeds of 13, 21, 42, 87, 100. The few-shot data will be generated to `data/k-shot`. In the directory of each dataset, there will be folders named as `$K-$SEED` indicating different dataset samples. You can use the following command to check whether the generated data are exactly the same as ours:
```bash
cd data/k-shot
md5sum -c checksum
```
**NOTE**: During training, the model will generate/load cache files in the data folder. If your data have changed, make sure to clean all the cache files (starting with "cache").
## Run LM-BFF
### Quick start
Our code is built on [transformers](https://github.com/huggingface/transformers) and we use its `3.4.0` version. Other versions of `transformers` might cause unexpected errors.
Before running any experiments, create the result folder by `mkdir result` to save checkpoints. Then you can run our code with the following example:
```bash
python run.py \
--task_name SST-2 \
--data_dir data/k-shot/SST-2/16-42 \
--overwrite_output_dir \
--do_train \
--do_eval \
--do_predict \
--evaluate_during_training \
--model_name_or_path roberta-large \
--few_shot_type prompt-demo \
--num_k 16 \
--max_steps 1000 \
--eval_steps 100 \
--per_device_train_batch_size 2 \
--learning_rate 1e-5 \
--num_train_epochs 0 \
--output_dir result/tmp \
--seed 42 \
--template "*cls**sent_0*_It_was*mask*.*sep+*" \
--mapping "{'0':'terrible','1':'great'}" \
--num_sample 16 \
```
Most arguments are inherited from `transformers` and are easy to understand. We further explain some of the LM-BFF's arguments:
* `few_shot_type`Excerpt of 17,931 characters
Read on GitHub
Danqi Chen · Princeton University
5
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e4a9068e20a54f9d, desc:fine-tuning, desc:fine tuning