Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Evaluation code for various unsupervised automated metrics for Natural Language Generation.
| Date | Stars |
|---|---|
| 2026-07-24 | 1391 |
| 2026-07-25 | 1391 |
| 2026-07-28 | 1391 |
| 2026-07-30 | 1391 |
| 2026-08-06 | 1391 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
[](https://travis-ci.org/Maluuba/nlg-eval)
# nlg-eval
Evaluation code for various unsupervised automated metrics for NLG (Natural Language Generation).
It takes as input a hypothesis file, and one or more references files and outputs values of metrics.
Rows across these files should correspond to the same example.
## Metrics ##
- BLEU
- METEOR
- ROUGE
- CIDEr
- SPICE
- SkipThought cosine similarity
- Embedding Average cosine similarity
- Vector Extrema cosine similarity
- Greedy Matching score
## Setup ##
Install Java 1.8.0 (or higher).
Install the Python dependencies, run:
```bash
pip install git+https://github.com/Maluuba/nlg-eval.git@master
```
If you are using macOS High Sierra or higher, then run this to allow multithreading:
```bash
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
```
Simple setup (download required data (e.g. models, embeddings) and external code files), run:
```bash
nlg-eval --setup
```
If you're setting this up from the source code or you're on Windows and not using a Bash terminal, then you might get errors about `nlg-eval` not being found.
You will need to find the `nlg-eval` script.
See [here](https://github.com/Maluuba/nlg-eval/issues/61) for details.
### Custom Setup ###
```bash
# If you don't like the default path (~/.cache/nlgeval) for the downloaded data,
# then specify a path where you want the files to be downloaded.
# The value for the data path is stored in ~/.config/nlgeval/rc.json and can be overwritten by
# setting the NLGEVAL_DATA environment variable.
nlg-eval --setup ${data_path}
```
### Validate the Setup (Optional) ###
(These examples were made with Git Bash on Windows)
All of the data files should have been downloaded, you should see sizes like:
```
$ ls -l ~/.cache/nlgeval/
total 6003048
-rw-r--r-- 1 ... 289340074 Sep 12 2018 bi_skip.npz
-rw-r--r-- 1 ... 689 Sep 12 2018 bi_skip.npz.pkl
-rw-r--r-- 1 ... 2342138474 Sep 12 2018 btable.npy
-rw-r--r-- 1 ... 7996547 Sep 12 2018 dictionary.txt
-rw-r--r-- 1 ... 21494787 Jan 22 2019 glove.6B.300d.model.bin
-rw-r--r-- 1 ... 480000128 Jan 22 2019 glove.6B.300d.model.bin.vectors.npy
-rw-r--r-- 1 ... 663989216 Sep 12 2018 uni_skip.npz
-rw-r--r-- 1 ... 693 Sep 12 2018 uni_skip.npz.pkl
-rw-r--r-- 1 ... 2342138474 Sep 12 2018 utable.npy
```
You can also verify some checksums:
```
$ cd ~/.cache/nlgeval/
$ md5sum *
9a15429d694a0e035f9ee1efcb1406f3 *bi_skip.npz
c9b86840e1dedb05837735d8bf94cee2 *bi_skip.npz.pkl
022b5b15f53a84c785e3153a2c383df6 *btable.npy
26d8a3e6458500013723b380a4b4b55e *dictionary.txt
f561ab0b379e23cbf827a054f0e7c28e *glove.6B.300d.model.bin
be5553e91156471fe35a46f7dcdfc44e *glove.6B.300d.model.bin.vectors.npy
8eb7c6948001740c3111d71a2fa446c1 *uni_skip.npz
e1a0ead377877ff3ea5388bb11cfe8d7 *uni_skip.npz.pkl
5871cc62fc01b79788c79c219b175617 *utable.npy
$ sha256sum *
8ab7965d2db5d146a907956d103badfa723b57e0acffb75e10198ba9f124edb0 *bi_skip.npz
d7e81430fcdcbc60b36b92b3f879200919c75d3015505ee76ae3b206634a0eb6 *bi_skip.npz.pkl
4a4ed9d7560bb87f91f241739a8f80d8f2ba787a871da96e1119e913ccd61c53 *btable.npy
4dc5622978a30cddea8c975c871ea8b6382423efb107d27248ed7b6cfa490c7c *dictionary.txt
10c731626e1874effc4b1a08d156482aa602f7f2ca971ae2a2f2cd5d70998397 *glove.6B.300d.model.bin
20dfb1f44719e2d934bfee5d39a6ffb4f248bae2a00a0d59f953ab7d0a39c879 *glove.6B.300d.model.bin.vectors.npy
7f40ff16ff5c54ce9b02bd1a3eb24db3e6adaf7712a7a714f160af3a158899c8 *uni_skip.npz
d58740d46cba28417cbc026af577f530c603d81ac9de43ffd098f207c7dc4411 *uni_skip.npz.pkl
790951d4b08e843e3bca0563570f4134ffd17b6bd4ab8d237d2e5ae15e4febb3 *utable.npy
```
If you're ensure that the setup was successful, you can run the tests:
```bash
pip install pytest
pytest
```
It might take a few minutes and you might see warnings but they should pass.
## Usage ##
Once setup has completed, the metrics can be evaluated with a Python API or in the command line.
Examples of the Python Excerpt of 8,419 characters
Read on GitHub
Hannes Schulz · @Microsoft · Germany
31
24
16
8
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:503167de45240f07, topic:nlp, topic:natural-language-processing, topic:machine-translation