Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Tensorflow implementation of contextualized word representations from bi-directional language models
| Date | Stars |
|---|---|
| 2026-07-31 | 1612 |
| 2026-08-06 | 1612 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# bilm-tf
Tensorflow implementation of the pretrained biLM used to compute ELMo
representations from ["Deep contextualized word representations"](http://arxiv.org/abs/1802.05365).
This repository supports both training biLMs and using pre-trained models for prediction.
We also have a pytorch implementation available in [AllenNLP](http://allennlp.org/).
You may also find it easier to use the version provided in [Tensorflow Hub](https://www.tensorflow.org/hub/modules/google/elmo/2) if you just like to make predictions.
Citation:
```
@inproceedings{Peters:2018,
author={Peters, Matthew E. and Neumann, Mark and Iyyer, Mohit and Gardner, Matt and Clark, Christopher and Lee, Kenton and Zettlemoyer, Luke},
title={Deep contextualized word representations},
booktitle={Proc. of NAACL},
year={2018}
}
```
## Installing
Install python version 3.5 or later, tensorflow version 1.2 and h5py:
```
pip install tensorflow-gpu==1.2 h5py
python setup.py install
```
Ensure the tests pass in your environment by running:
```
python -m unittest discover tests/
```
## Installing with Docker
To run the image, you must use nvidia-docker, because this repository
requires GPUs.
```
sudo nvidia-docker run -t allennlp/bilm-tf:training-gpu
```
## Using pre-trained models
We have several different English language pre-trained biLMs available for use.
Each model is specified with two separate files, a JSON formatted "options"
file with hyperparameters and a hdf5 formatted file with the model
weights. Links to the pre-trained models are available [here](https://allennlp.org/elmo).
There are three ways to integrate ELMo representations into a downstream task, depending on your use case.
1. Compute representations on the fly from raw text using character input. This is the most general method and will handle any input text. It is also the most computationally expensive.
2. Precompute and cache the context independent token representations, then compute context dependent representations using the biLSTMs for input data. This method is less computationally expensive then #1, but is only applicable with a fixed, prescribed vocabulary.
3. Precompute the representations for your entire dataset and save to a file.
We have used all of these methods in the past for various use cases. #1 is necessary for evaluating at test time on unseen data (e.g. public SQuAD leaderboard). #2 is a good compromise for large datasets where the size of the file in #3 is unfeasible (SNLI, SQuAD). #3 is a good choice for smaller datasets or in cases where you'd like to use ELMo in other frameworks.
In all cases, the process roughly follows the same steps.
First, create a `Batcher` (or `TokenBatcher` for #2) to translate tokenized strings to numpy arrays of character (or token) ids.
Then, load the pretrained ELMo model (class `BidirectionalLanguageModel`).
Finally, for steps #1 and #2 use `weight_layers` to compute the final ELMo representations.
For #3, use `BidirectionalLanguageModel` to write all the intermediate layers to a file.
#### Shape conventions
Each tokenized sentence is a list of `str`, with a batch of sentences
a list of tokenized sentences (`List[List[str]]`).
The `Batcher` packs these into a shape
`(n_sentences, max_sentence_length + 2, 50)` numpy array of character
ids, padding on the right with 0 ids for sentences less then the maximum
length. The first and last tokens for each sentence are special
begin and end of sentence ids added by the `Batcher`.
The input character id placeholder can be dimensioned `(None, None, 50)`,
with both the batch dimension (axis=0) and time dimension (axis=1) determined
for each batch, up the the maximum batch size specified in the
`BidirectionalLanguageModel` constructor.
After running inference with the batch, the return biLM embeddings are
a numpy array with shape `(n_sentences, 3, max_sentence_length, 1024)`,
after removing the special begin/end tokens.
#### Vocabulary file
The `Batcher` takes a vocabulaExcerpt of 14,724 characters
Read on GitHub
70
Philip May · @telekom
5
5
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:638468b2510bb43b, llm:Description: 'Tensorflow implementation of contextualized word representations from bi-directional language models' (AllenAI bilm-tf).
matched fp:638468b2510bb43b, llm:Description: 'Tensorflow implementation of contextualized word representations from bi-directional language models' (AllenAI bilm-tf).
matched fp:638468b2510bb43b, llm:Description: 'Tensorflow implementation of contextualized word representations from bi-directional language models' (AllenAI bilm-tf).