Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day
| Date | Stars |
|---|---|
| 2026-07-31 | 260 |
| 2026-08-06 | 260 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Neurips 1 LLM 1 GPU Challenge This repository provides a starting point for those who are interested in the [NeurIPS 1 LLM 1 GPU Competition](https://llm-efficiency-challenge.github.io/). It provides detailed clarifications on what a submission looks like exactly, and how it will be evaluated and submitted. At a high level, the key thing you will contribute is a `Dockerfile`, which will be a reproducible artifact that we can use to test your submission. The `Dockerfile` should contain all the code and dependencies needed to run your submission. We will use this `Dockerfile` to build a docker image and then run it against a set of tasks which will be a subset of the [HELM](https://crfm.stanford.edu/helm/latest/) tasks. Your `Dockerfile` will expose a simple HTTP server, which needs to implement 2 endpoints `/process` and `/tokenize`. We will build that `Dockerfile` and expect it to launch an HTTP server. Once that server is launched, we will make requests to it via HELM and record your results. At a high level the flow you should follow to ensure a strong submission: 1. Pick approved LLMs and datasets from [here](https://llm-efficiency-challenge.github.io/challenge) 2. Start with one of [sample-submissions](sample-submissions) and make sure it runs 3. Evaluate it locally on your own 40Gb A100 or 4090, if you don't have funding for either please see the [GPU funding](#gpu-funding) section for some more options 4. Once you have something working you can make a submission on our [Discord Leaderboard](https://discord.com/channels/1124130156336922665/1124134272631054447/1151718598818156645) to see how you fare up against other competitors 5. On the competition deadline make sure you have the final eval Dockerfile you'd like us to run in your github repo, refer to the [timeline](https://llm-efficiency-challenge.github.io/dates) 6. If your entry makes the shortlist, we will work with you to reproduce all of your artifacts with another finetuning Dockerfile ## Contents - [Approved LLM & Dataset](#approved-llm-and-dataset) - [Submission](#submission) - [Evaluate Your Model Locally Using HELM](#evaluate-your-model-locally-using-helm) - [Finetune](#finetune) - [Create your own submission template](#create-your-own-submission-template) - [Discord Leaderboard](#discord-leaderboard) - [Final Leaderboard Submission](#final-eval-submission) - [Evaluating the Final Submission](#evaluating-the-final-submission) - [GPU funding](#gpu-funding) ## Approved LLM and dataset The LLM space has complex licenses which can make it difficult to figure out what's permittable to use in a competition to streamline this process we've shortlisted a few models and datasets we know are safe to use [here](https://llm-efficiency-challenge.github.io/challenge) That said the LLM space is fast moving so if you'd like to use a dataset or model that isn't on our list make sure to ask us about it on [https://discord.gg/XJwQ5ddMK7](https://discord.gg/XJwQ5ddMK7) ## Submission The submission in this repository is a basic implementation of the setting up an HTTP server in accordance to the `open_api` spec. It includes a sample solution built off of [Lit-GPT](https://github.com/Lightning-AI/lit-gpt) and open-llama weights that participants can reference or modify as they see fit. You can use the provided code as a reference or starting point for your own implementation. The `main.py` file contains the simple FastAPI server, and you can modify it to suit your needs. You can find the Lit-GPT submission [here](sample-submissions/lit-gpt/) and the llama-recipes submission [here](sample-submissions/llama_recipes/) with instructions on how to run each locally. Make sure that your final submission has only a single `Dockerfile` and that your weights are not directly included in the repo, they need to be downloaded during docker build or at runtime. ## Evaluate Your Model Locally Using HELM Every submission will be tested against [HELM](https://crfm.stanford.edu/hel
Excerpt of 8,779 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:5b97dbf0c58e16d3, llm:Repository title and description: 'NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day' — indicates focus on LLM efficiency benchmarking and optimization for constrained resources.
matched fp:5b97dbf0c58e16d3, llm:Repository title and description: 'NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day' — indicates focus on LLM efficiency benchmarking and optimization for constrained resources.
matched fp:5b97dbf0c58e16d3, llm:Repository title and description: 'NeurIPS Large Language Model Efficiency Challenge: 1 LLM + 1GPU + 1Day' — indicates focus on LLM efficiency benchmarking and optimization for constrained resources.