Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
This repository deploys YOLOv4 as an optimized TensorRT engine to Triton Inference Server
| Date | Stars |
|---|---|
| 2026-07-24 | 283 |
| 2026-07-25 | 283 |
| 2026-07-28 | 283 |
| 2026-07-30 | 283 |
| 2026-08-06 | 283 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# YOLOv4 on Triton Inference Server with TensorRT  [](https://opensource.org/licenses/MIT) This repository shows how to deploy YOLOv4 as an optimized [TensorRT](https://github.com/NVIDIA/tensorrt) engine to [Triton Inference Server](https://github.com/NVIDIA/triton-inference-server). Triton Inference Server takes care of model deployment with many out-of-the-box benefits, like a GRPC and HTTP interface, automatic scheduling on multiple GPUs, shared memory (even on GPU), health metrics and memory resource management. TensorRT will automatically optimize throughput and latency of our model by fusing layers and chosing the fastest layer implementations for our specific hardware. We will use the TensorRT API to generate the network from scratch and add all non-supported layers as a plugin. ## Build TensorRT engine There are no dependencies needed to run this code, except a working docker environment with GPU support. We will run all compilation inside the TensorRT NGC container to avoid having to install TensorRT natively. Run the following to get a running TensorRT container with our repo code: ```bash cd yourworkingdirectoryhere git clone [email protected]:isarsoft/yolov4-triton-tensorrt.git docker run --gpus all -it --rm -v $(pwd)/yolov4-triton-tensorrt:/yolov4-triton-tensorrt nvcr.io/nvidia/tensorrt:21.10-py3 ``` Docker will download the TensorRT container. You need to select the version (in this case 21.10) according to the version of Triton that you want to use later to ensure the TensorRT versions match. Matching NGC version tags use the same TensorRT version. Inside the container run the following to compile our code: ```bash cd /yolov4-triton-tensorrt mkdir build cd build cmake .. make ``` This will generate two files (`liblayerplugin.so` and `main`). The library contains all unsupported TensorRT layers and the executable will build us an optimized engine in a second. Download the weights for this network from [Google Drive](https://drive.google.com/drive/folders/1YUDVgEefnk2HENpGMwq599Yj45i_7-iL?usp=sharing). Instructions on how to generate this weight file from the original darknet config and weights can be found [here](https://github.com/wang-xinyu/tensorrtx/tree/master/yolov4). Place the weight file in the same folder as the executable `main`. Then run the following to generate a serialized TensorRT engine optimized for your GPU: ```bash ./main ``` This will generate a file called `yolov4.engine`, which is our serialized TensorRT engine. Together with `liblayerplugin.so` we can now deploy to Triton Inference Server. Before we do this we can test the engine with standalone TensorRT by running: ```bash cd /workspace/tensorrt/bin ./trtexec --loadEngine=/yolov4-triton-tensorrt/build/yolov4.engine --plugins=/yolov4-triton-tensorrt/build/liblayerplugin.so ``` ``` (...) [I] Starting inference threads [I] Warmup completed 1 queries over 200 ms* [I] Timing trace has 204 queries over 3.00185 s [I] Trace averages of 10 runs: [I] Average on 10 runs - GPU latency: 7.8773 ms* - Host latency: 9.45764 ms* (end to end 9.48074 ms*, enqueue 1.98274 ms* [I] Average on 10 runs - GPU latency: 7.73803 ms* - Host latency: 9.3154 ms* (end to end 9.33945 ms*, enqueue 2.02845 ms* (...) [I] GPU Compute [I] min: 7.01465 ms* [I] max: 9.11838 ms* [I] mean: 7.79672 ms* ``` ## Deploy to Triton Inference Server We need to create our model repository file structure first: ```bash # Create model repository cd yourworkingdirectoryhere mkdir -p triton-deploy/models/yolov4/1/ mkdir triton-deploy/plugins # Copy engine and plugins cp yolov4-triton-tensorrt/build/yolov4.engine triton-deploy/models/yolov4/1/model.plan cp yolov4-triton-tensorrt/build/liblayerplugin.so triton-deploy/plugins/ ``` Now we can start Triton with this
Excerpt of 9,810 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8b4486718a6d331d, topic:triton-inference-server, topic:tensorrt, desc:inference server