Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052
| Date | Stars |
|---|---|
| 2026-07-24 | 479 |
| 2026-07-25 | 479 |
| 2026-07-28 | 479 |
| 2026-07-30 | 479 |
| 2026-08-06 | 479 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# ByteTransformer: Optimized BERT Transformer Inference on NVIDIA GPUs
## Introduction
ByteTransformer is a high-performance inference library for BERT-like transformers that offers the following features:
* Provides Python and C++ APIs, with the PyTorch plugin allowing users to enhance transformer inference with just a few lines of Python code.
* Supports both fixed-length and variable-length transformers.
* Includes end-to-end architectural-aware optimizations for the padding-free algorithm on BERT routines, including QKV encoding, softmax, feed forward network, activation, layernorm, and multi-head attention.
ByteTransformer has been widely deployed to improve in-house transformer inference serving systems at ByteDance, delivering superior performance over other transformer implementations for both fixed-length and variable-length inputs. The technical details have been published at IEEE IPDPS 2023.
## Cite Us
If you use our library, please cite our research paper.
```
@article{zhai2022bytetransformer,
title={ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs},
author={Zhai, Yujia and Jiang, Chengquan and Wang, Leyuan and Jia, Xiaoying and Zhang, Shang and Chen, Zizhong and Liu, Xin and Zhu, Yibo},
journal={arXiv preprint arXiv:2210.03052},
year={2022}
}
```
## Performance and Speedup
We compared ByteTransformer with PyTorch, TensorFlow, FasterTransformer, and DeepSpeed on an A100 GPU. The benchmark script is available in [benchmark/bert_bench.sh](https://github.com/bytedance/ByteTransformer/blob/main/benchmark/bert_bench.sh).
**1. Standard BERT batch size = 1, average sequence length = 0.6 * maximal, execution time in millisecond:**
| | PyTorch | Tensorflow | FasterTransformer | FasterTransformer with remove padding | DeepSpeed | ByteTransformer |
|------|-------------|----------------|-------------------|---------------------------------------|---------------------|-----------------|
| 64 | 2.93 | 2.46 | 1.05 | 1.23 | 1.17 | 0.90 |
| 128 | 3.18 | 2.6 | 1.10 | 1.43 | 1.28 | 0.97 |
| 192 | 3.18 | 2.81 | 1.26 | 1.43 | 1.40 | 1.36 |
| 256 | 2.81 | 2.9 | 1.35 | 1.55 | 1.51 | 1.43 |
| 320 | 3.11 | 3.24 | 1.63 | 1.66 | 1.84 | 1.69 |
| 384 | 2.87 | 3.43 | 1.64 | 1.64 | 1.95 | 1.72 |
| 448 | 2.99 | 3.61 | 2.26 | 2.35 | 2.23 | 1.86 |
| 512 | 2.89 | 3.74 | 2.28 | 2.43 | 2.37 | 2.00 |
| 576 | 2.99 | 4.03 | 2.51 | 2.59 | 2.70 | 2.19 |
| 640 | 2.99 | 4.54 | 2.85 | 2.83 | 3.17 | 2.23 |
| 704 | 3.21 | 4.67 | 3.16 | 3.44 | 3.32 | 2.47 |
| 768 | 3.33 | 4.88 | 3.26 | 3.63 | 3.46 | 2.51 |
| 832 | 3.78 | 5.39 | 3.75 | 3.87 | 3.97 | 2.80 |
| 896 | 3.86 | 5.81 | 4.08 | 4.95 | 4.37 | 2.86 |
| 960 | 4.02 Excerpt of 8,876 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:980a2c40eedffa19, topic:inference
matched fp:980a2c40eedffa19, topic:gpu
matched fp:980a2c40eedffa19, topic:transformer