Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
[NeurIPS 2022] DRAGON 🐲: Deep Bidirectional Language-Knowledge Graph Pretraining
| Date | Stars |
|---|---|
| 2026-07-31 | 333 |
| 2026-08-07 | 333 |
| 2026-08-15 | 332 |
| 2026-08-24 | 333 |
| 2026-09-01 | 334 |
| 2026-09-09 | 335 |
| 2026-09-23 | 336 |
| 2026-10-22 | 337 |
| 2026-10-30 | 336 |
| 2026-11-04 | 336 |
Today
— stars today
This week
-1 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# DRAGON: Deep Bidirectional Language-Knowledge Graph Pretraining
This repo provides the source code & data of our paper "[DRAGON: Deep Bidirectional Language-Knowledge Graph Pretraining](https://arxiv.org/abs/2210.09338)" (NeurIPS 2022).
### Overview
DRAGON is a new foundation model (improvement of BERT) that is **pre-trained jointly from text and knowledge graphs** for improved language, knowledge and reasoning capabilities. Specifically, it was trained with two simultaneous self-supervised objectives, language modeling and link prediction, that encourage deep bidirectional reasoning over text and knowledge graphs.
DRAGON can be used as a drop-in replacement for BERT. It achieves better performance in various NLP tasks, and is particularly effective for **knowledge and reasoning-intensive** tasks such as multi-step reasoning and low-resource QA.
<p align="center">
<img src="./figs/dragon.png" width="1000" title="DRAGON model overview" alt="">
</p>
## 0. Dependencies
Run the following commands to create a conda environment:
```bash
conda create -y -n dragon python=3.8
conda activate dragon
pip install torch==1.10.1+cu113 torchvision -f https://download.pytorch.org/whl/cu113/torch_stable.html
pip install transformers==4.9.1 wandb nltk spacy==2.1.6
python -m spacy download en
pip install scispacy==0.3.0
pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.3.0/en_core_sci_sm-0.3.0.tar.gz
pip install torch-scatter==2.0.9 torch-sparse==0.6.12 torch-geometric==2.0.0 -f https://pytorch-geometric.com/whl/torch-1.10.1+cu113.html
```
## 1. Download pretrained models
You can download pretrained DRAGON models below. Place the downloaded model files under `./models`
| Model | Domain | Size | Pretraining Text | Pretraining Knowledge Graph | Download Link |
| ------------- | ------------- | --------- | ---- | ---- | ---- |
| DRAGON | General | 360M parameters | BookCorpus | ConceptNet | [general_model](https://nlp.stanford.edu/projects/myasu/DRAGON/models/general_model.pt) |
| DRAGON | Biomedicine | 360M parameters | PubMed | UMLS | [biomed_model](https://nlp.stanford.edu/projects/myasu/DRAGON/models/biomed_model.pt) |
## 2. Download data
### Commonsense domain
You can download all the preprocessed data from [**[here]**](https://nlp.stanford.edu/projects/myasu/DRAGON/data_preprocessed.zip). This includes the ConceptNet knowledge graph as well as CommonsenseQA, OpenBookQA and RiddleSense datasets. Specifically, run:
```
wget https://nlp.stanford.edu/projects/myasu/DRAGON/data_preprocessed.zip
unzip data_preprocessed.zip
mv data_preprocessed data
```
**(Optional)** If you would like to preprocess the raw data from scratch, you can download the raw data – ConceptNet Knowledge graph, CommonsenseQA, OpenBookQA – by:
```
./download_raw_data.sh
```
To preprocess the raw data, run:
```
CUDA_VISIBLE_DEVICES=0 python preprocess.py -p <num_processes> --run common csqa obqa
```
You can specify the GPU you want to use in the beginning of the command `CUDA_VISIBLE_DEVICES=...`. The script will:
* Setup ConceptNet (e.g., extract English relations from ConceptNet, merge the original 42 relation types into 17 types)
* Convert the QA datasets into .jsonl files (e.g., stored in `data/csqa/statement/`)
* Identify all mentioned concepts in the questions and answers
* Extract subgraphs for each q-a pair
### Biomedical domain
You can download all the preprocessed data from [**[here]**](https://nlp.stanford.edu/projects/myasu/DRAGON/data_preprocessed.zip). This includes the UMLS biomedical knowledge graph and MedQA dataset.
**(Optional)** If you would like to preprocess MedQA from scratch, follow `utils_biomed/preprocess_medqa.ipynb` and then run
```
CUDA_VISIBLE_DEVICES=0 python preprocess.py -p <num_processes> --run medqa
```
<br>
The resulting file structure should look like this:
```plain
.
├── README.md
├── models/
├── general_model.pt
├── biomed_model.pt
└── data/
├── cpnet/ (pExcerpt of 7,671 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:ce5f847e6c2335b8, topic:knowledge-graph, topic:question-answering, desc:knowledge graph
matched fp:ce5f847e6c2335b8, topic:pretraining, desc:pretraining, readme:pretraining
matched fp:ce5f847e6c2335b8, topic:transformer, topic:language-model