Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Bayesian Optimization as a Coverage Tool for Evaluating LLMs. Accurate evaluation (benchmarking) that's 10 times faster with just a few lines of modular code.
| Date | Stars |
|---|---|
| 2026-07-31 | 290 |
| 2026-08-03 | 290 |
| 2026-08-06 | 290 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
### 🫙 Archive > This has topped the show HN front page [link](https://news.ycombinator.com/item?id=39358406). > This was born as a research project, and I think I have achieved most of what I set out to create. So today (2025/09/14) I'm archiving this repository. > Nowadays I'm mainly working on [`aioway`](https://github.com/rentruewang/aioway), a deep learning algorithm compiler! Check it out. # ☂️ BoCoEL ## Bayesian Optimization as a Coverage Tool for Evaluating Large Language Models  [](https://github.com/rentruewang/bocoel/actions/workflows/release.yaml) [](https://github.com/rentruewang/bocoel/actions/workflows/build.yaml) [](https://github.com/rentruewang/bocoel/actions/workflows/format.yaml) [](https://github.com/rentruewang/bocoel/actions/workflows/typecheck.yaml) [](https://github.com/rentruewang/bocoel/actions/workflows/unittest.yaml)   ## 🤔 Why BoCoEL? Large language models are expensive and slow behemoths, and evaluating them on gigantic modern datasets only makes it worse. If only there is a way to just select a meaningful (_and small_) subset of the corpus and obtain a highly accurate evaluation..... Wait, sounds like Bayesian Optimization! Bocoel works in the following steps: 1. Encode individual entry into embeddings (way cheaper / faster than LLM and reusable). 2. Use Bayesian optimization to select queries to evaluate. 3. Use the queries to retrieve from our corpus (with the encoded embeddings). 4. Profit. The evaluations generated are easily managed by the provided manager utility. To our knowledge, this is the first work aiming to reduce computation costs during evaluation (benchmarking) with a (possibly dynamic) budget. ## 🚀 Features - 🎯 Accurately evaluate large language models with just tens of samples from your selected corpus. - 💂♂️ Uses the power of Bayesian optimization to select an optimal subset of samples for the language model to evaluate. - 💯 Evaluate the corpus on the model in addition to evaluating the model on the corpus. - 🤗 Support for `GPT2`, `Pythia`, `LLAMA` and more through integration with huggingface [transformers](https://huggingface.co/docs/transformers/en/index) and [datasets](https://huggingface.co/docs/datasets/en/index) - 🧩 Modular design. - 🔎 Efficient representation of the corpus / dataset such as N-sphere representation or whitening of the latent space to augment evaluation quality. ## ⭐ Give us a star! Like what you see? Please consider giving this a star (★)! ## ♾️ Bayesian Optimization <img src="https://upload.wikimedia.org/wikipedia/commons/0/02/GpParBayesAnimationSmall.gif" width="30%" align="right"/> Simply put, Bayesian optimization aims to optimize either the exploration objective (the purple area in the image) or the exploitation object (the height of the black dots). It uses Gaussian processes as a backbone for inference, and uses an **acquisition function** to decide where to sample next. See [here](https://distill.pub/2019/visual-exploration-gaussian-processes/) for an a more in-depth introduction. Since _Bayesian optimization works well with an expensive-to-evaluate black-box model (paraphrase: LLM)_, it is perfect for this particular use case. Bocoel uses Bayesian optimization as a backbone for exploring the embedding space given by our corpus, which allows it to select a good subset acting as a mini sn
Excerpt of 5,965 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:754e6ff71941e0a8, topic:llm, topic:language-model
matched fp:754e6ff71941e0a8, topic:evaluation