Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Enterprise RAG Challenge to test accuracy of different LLM-driven assistants
| Date | Stars |
|---|---|
| 2026-07-31 | 302 |
| 2026-08-06 | 302 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Enterprise RAG Challenge with Annual Reports BEFORE CLONING: Be aware that this repo has around 1,4GB (due to large PDFs contained). ## Round 1 - Sept. 2024 You can find all infos on the first run in the folder [round1](round1). Make sure to check out the [README](round1/README.md) there. <br> ## Round 2 - Feb 2025 👉[Landing page with Results Video and Keynote by Stephan Gillich](https://www.timetoact-group.at/details/enterprise-rag-challenge)👈  You can find an interactive leaderboard with detailed results [here](https://abdullin.com/erc#r2). The data for the test run of the round 2 can be found in the [samples](round2/samples) folder. <br> ### Explanation We have been discussing various approaches for building AI-driven assistants for companies. Most of the discussions focused on the efficiency and accuracy achieved by different approaches: RAG systems with vector databases and Domain-Driven AI Assistants (with Knowledge Maps). In the long run, the technologies don’t even matter that much - they are just the implementation details. The only thing that is important - does the LLM-driven system provide accurate answers or does it hallucinate on the spot. We can measure that! The idea started as a friendly competition challenge between a couple of teams. Anybody interested can bring their own RAG system of choice into the game (or build one from scratch). This RAG should be able to ingest a bunch of public annual reports and answer questions. All questions will be based on information retrieval tasks: “How many people are employed at the company X?”, “Which company has more liquidity?”, “Does the company X invest in green bonds?” By the way, this test represents a real business case that is quite valuable for companies. First of all, here we can measure and compare the accuracy and hallucination rates of different approaches. Additionally, question answering over a large corpus of data maps well to cases like enterprise lead generation or internal customer support. Participants can use any technology they wish to. It could be local, cloud-hosted or even third-party. The only requirements are: - You can upload a test set of public annual reports. - You can upload a list of questions to the system and get a list of answers. - Between steps 1 and 2, there will be a lag of a few hours, so that the systems could ingest the data (e.g. compute embedding vectors or populate knowledge maps). - There is no need to share the source code of the system. If you are interested, below is the link for a sample subset of PDFs for development. Test dataset will obviously contain different PDFs. <br> ### Geeky Details We want to make the competition open and fair to everyone, so we invested extra effort in that. All participants, even TimeToAct will be in the same conditions: 1. We share a list of all annual reports (7496 files and ~46GB) along with company names and file sha1 hashes in [dataset.csv](dataset.csv). These annual reports are public information. **We don't share all 46GBs of these PDFs upfront, but if you really want, you can find them on the Internet.** 2. We share a code that will generate next unpredictable random seed for the competition. It uses public blockchain API. See [gen_seed.py](gen_seed.py) for the implementation details. 3. We share the question generator that will randomly pick a subset of files for the competition. It will also generate random questions for these files. See [main.py](main.py). 4. Question generator uses a deterministic RNG that will work similarly for everybody. Anybody can run this code at the moment of competition. Everybody is guaranteed to get the same list of files and questions for the competition at the same time. Nobody should be able to figure out these files and questions in advance. As soon as we have a list of PDF files, TimeToAct will package them into a zip and share publicly. Everybody can verify sha1 hashes of these files to
Excerpt of 11,825 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:8c8f65a8dd41dcd1, llm:description: 'Enterprise RAG Challenge to test accuracy of different LLM-driven assistants' (repository description)
matched fp:8c8f65a8dd41dcd1, llm:description: 'Enterprise RAG Challenge to test accuracy of different LLM-driven assistants' (repository description)
matched fp:8c8f65a8dd41dcd1, llm:description: 'Enterprise RAG Challenge to test accuracy of different LLM-driven assistants' (repository description)