Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Python library & examples for Masked Language Model Scoring (ACL 2020)
| Date | Stars |
|---|---|
| 2026-07-24 | 350 |
| 2026-07-25 | 350 |
| 2026-07-28 | 350 |
| 2026-07-30 | 350 |
| 2026-07-31 | 350 |
| 2026-08-06 | 350 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Masked Language Model Scoring
[](LICENSE)
This package uses masked LMs like [BERT](https://arxiv.org/abs/1810.04805), [RoBERTa](https://arxiv.org/abs/1907.11692), and [XLM](https://papers.nips.cc/paper/8928-cross-lingual-language-model-pretraining.pdf) to [score sentences](#scoring) and [rescore n-best lists](#rescoring) via *pseudo-log-likelihood scores*, which are computed by masking individual words. We also support autoregressive LMs like [GPT-2](https://openai.com/blog/better-language-models/). Example uses include:
- [Speech Recognition](examples/asr-librispeech-espnet): Rescoring an ESPnet LAS model (LibriSpeech)
- [Machine Translation](examples/nmt-tedtalks-ace): Rescoring a Transformer NMT model (IWSLT'15 en-vi)
- [Linguistic Acceptability](examples/lingacc-blimp): Unsupervised ranking within linguistic minimal pairs (BLiMP)
**Paper:** Julian Salazar, Davis Liang, Toan Q. Nguyen, Katrin Kirchhoff. "[Masked Language Model Scoring](https://arxiv.org/abs/1910.14659)", ACL 2020.
<p align="center"><img src="mlm-scoring.png" width="750px"></p>
## Installation
Python 3.6+ is required. Clone this repository and install:
```bash
pip install -e .
pip install torch mxnet-cu102mkl # Replace w/ your CUDA version; mxnet-mkl if CPU only.
```
Some models are via [GluonNLP](https://github.com/dmlc/gluon-nlp) and others are via [🤗 Transformers](https://github.com/huggingface/transformers), so for now we require both [MXNet](https://mxnet.apache.org/) and [PyTorch](https://pytorch.org/). You can now import the library directly:
```python
from mlm.scorers import MLMScorer, MLMScorerPT, LMScorer
from mlm.models import get_pretrained
import mxnet as mx
ctxs = [mx.cpu()] # or, e.g., [mx.gpu(0), mx.gpu(1)]
# MXNet MLMs (use names from mlm.models.SUPPORTED_MLMS)
model, vocab, tokenizer = get_pretrained(ctxs, 'bert-base-en-cased')
scorer = MLMScorer(model, vocab, tokenizer, ctxs)
print(scorer.score_sentences(["Hello world!"]))
# >> [-12.410664200782776]
print(scorer.score_sentences(["Hello world!"], per_token=True))
# >> [[None, -6.126736640930176, -5.501412391662598, -0.7825151681900024, None]]
# EXPERIMENTAL: PyTorch MLMs (use names from https://huggingface.co/transformers/pretrained_models.html)
model, vocab, tokenizer = get_pretrained(ctxs, 'bert-base-cased')
scorer = MLMScorerPT(model, vocab, tokenizer, ctxs)
print(scorer.score_sentences(["Hello world!"]))
# >> [-12.411025047302246]
print(scorer.score_sentences(["Hello world!"], per_token=True))
# >> [[None, -6.126738548278809, -5.501765727996826, -0.782496988773346, None]]
# MXNet LMs (use names from mlm.models.SUPPORTED_LMS)
model, vocab, tokenizer = get_pretrained(ctxs, 'gpt2-117m-en-cased')
scorer = LMScorer(model, vocab, tokenizer, ctxs)
print(scorer.score_sentences(["Hello world!"]))
# >> [-15.995375633239746]
print(scorer.score_sentences(["Hello world!"], per_token=True))
# >> [[-8.293947219848633, -6.387561798095703, -1.3138668537139893]]
```
(MXNet and PyTorch interfaces will be unified soon!)
## Scoring
Run `mlm score --help` to see supported models, etc. See `examples/demo/format.json` for the file format. For inputs, "score" is optional. Outputs will add "score" fields containing PLL scores.
There are three score types, depending on the model:
- Pseudo-log-likelihood score (PLL): BERT, RoBERTa, multilingual BERT, XLM, ALBERT, DistilBERT
- Maskless PLL score: same (add `--no-mask`)
- Log-probability score: GPT-2
We score hypotheses for 3 utterances of LibriSpeech `dev-other` on GPU 0 using BERT base (uncased):
```bash
mlm score \
--mode hyp \
--model bert-base-en-uncased \
--max-utts 3 \
--gpus 0 \
examples/asr-librispeech-espnet/data/dev-other.am.json \
> examples/demo/dev-other-3.lm.json
```
## Rescoring
One can rescore n-best lists via log-linear interpolation. Run `mlm rescore --help` to see all options. Input one is a file with original scores; input two are scorExcerpt of 4,903 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:f7adcd9b9f0674be, topic:nlp, readme:tokenizer, readme:machine translation
matched fp:f7adcd9b9f0674be, topic:pytorch, readme:pretraining
matched fp:f7adcd9b9f0674be, topic:language-model
matched fp:f7adcd9b9f0674be, topic:speech-recognition, readme:speech recognition