Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
FlagEval is an evaluation toolkit for AI large foundation models.
| Date | Stars |
|---|---|
| 2026-07-31 | 338 |
| 2026-08-06 | 338 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
 [简体中文](README_zh.md) -------------------------------------------------------------------------------- ### Overview **FlagEval** is an open-source evaluation toolkit as well as an open platform for evaluation of large models. FlagEval aims to cater to three principal evaluation subjects: foundational models, pre-training algorithms, and fine-tuning/compression algorithms. It encompasses four critical evaluation scenarios — Natural Language Processing (NLP), Computer Vision (CV), Audio, and Multimodal, alongside an abundant variety of downstream tasks. You can find more information on our official website [flageval.baai.ac.cn](https://flageval.baai.ac.cn/#/home). We're committed to developing scientific, impartial, and clear benchmarks, methodologies, and tools. Our goal is to enable researchers to thoroughly evaluate the effectiveness of foundational models and training algorithms. In addition, we are exploring the use of AI techniques to enhance subjective assessments, increasing both the objectivity and efficiency of our evaluation processes. FlagEval open-source toolkit now contains follwing sub-projects. ## 1. mCLIPEval [**mCLIPEval**](https://github.com/FlagOpen/FlagEval/tree/master/mCLIPEval) is a evaluation toolkit for vision-language models (such as CLIP, Contrastive Language–Image Pre-training). * Including Multilingual (12 languages) datasets and monolingual (English/Chinese) datasets. * Supporting for Zero-shot classification, Zero-shot retrieval and zeroshot composition tasks. * Adapted to [FlagAI](https://github.com/FlagAI-Open/FlagAI) pretrained models ([AltCLIP](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltCLIP), [EVA-CLIP](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/EVA_CLIP)), [OpenCLIP](https://github.com/mlfoundations/open_clip) pretrained models, [Chinese CLIP](https://github.com/OFA-Sys/Chinese-CLIP) models, [Multilingual CLIP](https://github.com/FreddeFrallan/Multilingual-CLIP) models, [Taiyi Series](https://fengshenbang-doc.readthedocs.io/zh/latest/docs/%E5%A4%AA%E4%B9%99%E7%B3%BB%E5%88%97/index.html) pretrained models, or customized models. * Data preparation from various resources, like [torchvision](https://pytorch.org/vision/stable/datasets.html), [huggingface](https://huggingface.co/datasets), [kaggle](https://www.kaggle.com/datasets), etc. * Visualization of evaluation results through leaderboard figures or tables, and detailed comparsions between two specific models. ### How to use Environment Preparation: * Pytorch version >= 1.8.0 * Python version >= 3.8 * For evaluating models on GPUs, you'll also need install CUDA and NCCL Step: ```shell git clone https://github.com/FlagOpen/FlagEval.git cd FlagEval/mCLIPEval/ pip install -r requirements.txt ``` Please refer to [mCLIPEval/README.md](https://github.com/FlagOpen/FlagEval/tree/master/mCLIPEval/README.md) for more details. ## 2. ImageEval-prompt [ImageEval-prompt](https://github.com/FlagOpen/FlagEval/blob/master/imageEval/README.md) is a set of prompts that evaluate text-to-image (T2I) models at a fine-grained level, including entity, style and detail. By conducting comprehensive evaluations at a fine-grained level, researchers can better understand the strengths and limitations of T2I models, in order to further improve their performance. * Including 1,624 English prompts and 339 Chinese prompts. * Each prompt is annotated using "double-blind annotation & third-party arbitration" approach, divided into three dimensions: entities, styles, and details. * Entity dimension includes five sub-dimensions: object, state, color, quantity, and position; * Style dimension includes two sub-dimensions: painting style and cultural style; * Detail dimension includes four sub-dimensions: hands, facial features, gender, and illogical knowledge. Please refer to [imageEval/README.md](https://github.com/FlagOpen/FlagEval/blob/master/imageEval/README.md) for more details. ## 3. C-SEM C
Excerpt of 6,974 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:a5ee7ae42179d110, llm:Repository description: "FlagEval is an evaluation toolkit for AI large foundation models." (language: Python, no topics).
matched fp:a5ee7ae42179d110, llm:Repository description: "FlagEval is an evaluation toolkit for AI large foundation models." (language: Python, no topics).