Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Formerly known as code.google.com/p/1-billion-word-language-modeling-benchmark
| Date | Stars |
|---|---|
| 2026-07-31 | 450 |
| 2026-08-06 | 450 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
Copyright 2013 Google Inc. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--------------------------------------------------------------------------
The project makes available a standard corpus of reasonable size (0.8 billion words)
to train and evaluate language models.
A few sample results we obtained at Google on this data are detailed at:
papers/naaclhlt2013.pdf
Besides the scripts needed to rebuild the training/held-out data, it also makes
available log-probability values for each word in each of ten feld-out data sets,
for each of the following baseline models:
. unpruned Katz (1.1B n-grams),
. pruned Katz (~15M n-grams),
. unpruned Interpolated Kneser-Ney (1.1B n-grams),
. pruned Interpolated Kneser-Ney (~15M n-grams)
The corpus is derived from the training-monolingual.tokenized/news.20??.en.shuffled.tokenized data distributed at http://statmt.org/wmt11/translation-task.html, Monolingual language model training data (Download it all in one file, 11 GB, at http://statmt.org/wmt11/training-monolingual.tgz).
A copy of the already pre-processed data, paper describing the benchmark, and other stuff is hosted at: http://www.statmt.org/lm-benchmark/
Corpus preparation:
====================
. download the "Monolingual language model training data" (http://statmt.org/wmt11/training-monolingual.tgz, 11 GB).
$ tar --extract -v --file ../statmt.org/tar_archives/training-monolingual.tgz --wildcards training-monolingual/news.20??.en.shuffled
followed by:
$ ./scripts/get-data.sh
For a more detailed description of the steps involved, see README.corpus_generation.
The md5sums for the files generated from the statmt.org download are listed at: README.corpus_generation_checkpoints.
You can use:
$ md5sum -c README.corpus_generation_checkpoints
to check that the files you produced match the checksums of those used to produce the results in the paper.
Baseline Language Models:
==========================
. trained and evaluated Katz and Interpolated Kneser-Ney language models.
. the word-level probability assignment for each word in the first 10 shards of the test data
(including the 00000 shard above for which we report LM performance) is available at:
baseline-lms/log/output.tar (tar-ed gzip files; due to their relatively large size they
are hosted on gDrive at https://drive.google.com/file/d/0B3u4EqGe3BUeMWhPS1hkdDZvTjA/edit?usp=sharing).
The output is in the following format:
...
WORDS: <S> Hello , world ! </S>
WORD IDS: 0 1044976486 1699010037 1539844246 217790329 1
Hello 1044976486 - 1.051147e+01 1.223186e+01 - COST=1.234405e+01
, 1699010037 - 8.899814e-01 8.908027e-01 3.305907e+00 - COST=1.806272e+00
world 1539844246 - 6.488597e+00 5.339889e+00 9.798830e+00 7.934176e+00 - COST=6.488597e+00
! 217790329 - 1.622149e+00 6.229870e+00 6.770101e+00 8.099472e+00 - COST=1.622149e+00
</S> 1 - NOTFOUND 3.559926e-01 3.563211e-01 2.975316e+00 - COST=3.559926e-01
- TOTAL ------------------------------------------------------------------ COST=2.261707e+01
The output first repeats the input sentence (with <S> and </S> added). Then it shows the corresponding integer word ids. Subsequent lines list ProdLM entries obtained with ProdLMClient and the resulting smoothed cost calculated by ProdLMWrapper. For each word, it shows the entries for the unigram, bigram, trigram etc. ending in the word shown at the beginning of the line. The example above shows a fourgram model. ThusExcerpt of 8,857 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:4894d6c6cbc3f207, llm:Description: 'Formerly known as code.google.com/p/1-billion-word-language-modeling-benchmark' — this is the well-known One Billion Word Language Modeling Benchmark dataset for language modeling research.
matched fp:4894d6c6cbc3f207, llm:Description: 'Formerly known as code.google.com/p/1-billion-word-language-modeling-benchmark' — this is the well-known One Billion Word Language Modeling Benchmark dataset for language modeling research.