Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Research and Materials on Hardware implementation of Transformer Model
| Date | Stars |
|---|---|
| 2026-07-24 | 309 |
| 2026-07-25 | 309 |
| 2026-07-28 | 309 |
| 2026-07-30 | 309 |
| 2026-08-06 | 309 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Transformer Models Silicon Research
> **Research and Materials on Hardware implementation of Transformer Models**
<!-- <p align="center">
<img src="https://img.shields.io/badge/-WIP-ff69b4?style=flat-square"/>
</p>
<p align="center">
<img src="https://img.shields.io/badge/Progress-%2599-ef6c00?labelColor=1565c0&style=flat-square"/>
</p> -->
## How to Contribute
**You can add new papers via pull requests, Please check `data/papers.yaml` and if your paper is not in list, add entity at the last item and create pull request.**
## Transformer and BERT Model
* BERT is a method of **pre-training language representations**, meaning that we **train a general-purpose *language understanding model*** on a large text corpus (like Wikipedia) and then use that model for downstream NLP tasks.
* BERT was created and **published in 2018 by Jacob Devlin and his colleagues from Google**. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks.
<p align="center">
<img src="./data/img/BERT-ARCH.png" width='480' />
</p>
* **BERT is a Transformer-based model.**
* The architecture of BERT is similar to the original Transformer model, except that BERT has two separate Transformer models: one for the left-to-right direction (the “encoder”) and one for the right-to-left direction (the “encoder”).
* The output of each model is the hidden state output by the final Transformer layer. The two models are pre-trained jointly on a large corpus of unlabeled text. The pre-training task is a simple and straightforward masked language modeling objective.
* The pre-trained BERT model can then be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.
---
### Reference Papers
**1. Attention Is All You Need**
 [](https://arxiv.org/abs/1706.03762) [](https://arxiv.org/pdf/1706.03762.pdf)
[](https://github.com/jadore801120/attention-is-all-you-need-pytorch) [](https://github.com/lsdefine/attention-is-all-you-need-keras)
<details>
<summary><img src="https://img.shields.io/badge/ABSTRACT-9575cd?&style=plastic"/></summary>
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.
</details>
**2. BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingExcerpt of 183,156 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:5325cec92e05ac79, topic:natural-language-processing, readme:natural language processing, readme:machine translation
matched fp:5325cec92e05ac79, topic:transformer