Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
llm-d Router: The intelligent entry point for inference requests
| Date | Stars |
|---|---|
| 2026-07-31 | 272 |
| 2026-08-03 | 278 |
| 2026-08-06 | 278 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
15.0
growth rate 0.00%/day
[](https://goreportcard.com/report/github.com/llm-d/llm-d-router)
[](https://pkg.go.dev/github.com/llm-d/llm-d-router)
[](/LICENSE)
[](https://llm-d.slack.com/archives/C08SBNRRSBD)
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fllm-d%2Fllm-d-router?ref=badge_shield)
# llm-d Router
> [!IMPORTANT]
> **Terminology Change**: The *Inference Scheduler* has been renamed to **llm-d Router**; see [Terminology](README.md#terminology).
> [!IMPORTANT]
> **API & Code Consolidation**: Core Endpoint Picker (EPP) code and the `InferenceObjective` and `InferenceModelRewrite` APIs have been merged into this repository from [Gateway API Inference Extension (GIE)]. The GIE repository now exclusively hosts the `InferencePool` API—an extension of the [Kubernetes Gateway API]—and defines the Endpoint Picker Protocol.
The **llm-d Router** is the intelligent entry point for inference traffic, delivering LLM load and prefix-cache aware routing, request prioritization, and advanced flow control across diverse request formats to fulfill complex serving objectives. It supports a flexible deployment model: it can run in **Standalone Mode** (where a self-managed Envoy proxy runs alongside the EPP in the same pod) or integrate with L7 load balancers—including self-managed instances (e.g., Istio, AgentGateway) and cloud-managed services (e.g., Google Cloud's Application Load Balancer)—via the Kubernetes Gateway API.
The router achieves its intelligence through an **Endpoint Picker (EPP)** that integrates with production-grade proxies (such as [Envoy]) via the [ext-proc] protocol, injecting real-time signals into the data plane to optimize request placement.
<p align="center">
<img src="docs/images/llm-d-router.svg" width="800" alt="llm-d Router Architecture">
</p>
## Core Components and APIs
This repository hosts the following core components:
- **Endpoint Picker (EPP)**: The intelligent routing engine that serves as the "brain" of the router. It evaluates incoming requests against the current state of the [InferencePool], considering factors like KV-cache locality, current load, and priority to make optimal placement decisions. It integrates with L7 proxies via the `ext-proc` protocol.
- **Request Management APIs**: These resources directly influence the EPP's request handling behavior:
- **InferenceObjective**: Configures the EPP's scheduling goals for specific requests, including priority levels and performance targets.
- **InferenceModelRewrite**: Directs the EPP to perform model name rewriting, enabling flexible traffic management for A/B testing and canary rollouts.
- **Disaggregation Sidecar**: A coordination component deployed alongside model servers (typically as a sidecar to the decode worker). It orchestrates complex multi-stage inference lifecycles, such as **P/D (Prefill/Decode)** and **E/P/D (Encode/Prefill/Decode)**, by communicating with specialized encode and prefill workers to manage KV-cache and embedding transfers. For more details, see the [Disaggregation Documentation].
## Modes of Operation
The llm-d Router supports two primary deployment modes as specified in the [Kubernetes Gateway API Inference Extensions]:
### 1. Standalone Mode
A lightweight deployment where a self-managed Envoy proxy runs alongside the EPP in the same pod. This mode is ideal for clusters without Gateway API infrastructure or for basic testing and local evaluations.
### 2. Gateway Mode (Inference Gateway)
The recommended mode for production environments, leveraging the official [Gateway API]. In this mode, the EPP acts as a backend for an `InfeExcerpt of 7,644 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:3d3ae7b37143e207, topic:inference
matched fp:3d3ae7b37143e207, topic:kubernetes