Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
WARC + AI - Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections.
| Date | Stars |
|---|---|
| 2026-07-31 | 275 |
| 2026-08-06 | 275 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# WARC-GPT **WARC + AI:** Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections. More info: - <a href="https://lil.law.harvard.edu/blog/2024/02/12/warc-gpt-an-open-source-tool-for-exploring-web-archives-with-ai/">"WARC-GPT: An Open-Source Tool for Exploring Web Archives Using AI"</a>. Feb 12 2024 - _lil.law.harvard.edu_ https://github.com/harvard-lil/warc-gpt/assets/625889/8ea3da4a-62a1-4ffa-a510-ef3e35699237 --- ## Summary - [Features](#features) - [Installation](#installation) - [Configuring the application](#configuring-the-application) - [Ingesting WARCs](#ingesting-warcs) - [Starting the server](#starting-the-server) - [Interacting with the Web UI](#interacting-with-the-web-ui) - [Interacting with the API](#interacting-with-the-api) - [Visualizing Embeddings](#visualizing-embeddings) - [Disclaimer](#disclaimer) --- ## Features - Retrieval Augmented Generation pipeline for WARC files - Highly customizable, can interact with many different LLMs, providers and embedding models - REST API - Web UI - Embeddings visualization [☝️ Summary](#summary) --- ## Installation WARC-GPT requires the following machine-level dependencies to be installed. - [Python 3.11+](https://python.org) - [Python Poetry](https://python-poetry.org/) Use the following commands to clone the project and install its dependencies: ```bash git clone https://github.com/harvard-lil/warc-gpt.git cd warc-gpt poetry env use 3.11 poetry install ``` If you don't want to use Poetry, or are in some context where that doesn't work, you can clone the repo, create a virtual environment, and install the dependencies like this: ```bash git clone https://github.com/harvard-lil/warc-gpt.git cd warc-gpt python3 -m venv env . env/bin/activate pip install . ``` If you choose this method, remove the prefix `poetry run ` from the commands below. [☝️ Summary](#summary) --- ## Configuring the application This program uses environment variables to handle settings. Copy `.env.example` into a new `.env` file and edit it as needed. ```bash cp .env.example .env ``` See details for individual settings in [.env.example](.env.example). **A few notes:** - WARC-GPT can interact with both the [OpenAI API](https://platform.openai.com/docs/introduction) and [Ollama](https://ollama.ai) for local inference. - Both can be used at the same time, but at least one is needed. - By default, the program will try to communicate with Ollama's API at `http://localhost:11434`. - It is also possible to use OpenAI's client to interact with compatible providers, such as [HuggingFace's Message API](https://huggingface.co/blog/tgi-messages-api) or [vLLM](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#using-openai-completions-api-with-vllm). To do so, set values for both `OPENAI_BASE_URL` and `OPENAI_COMPATIBLE_MODEL` environment variables. - Prompts can be edited directly in the configuration file. [☝️ Summary](#summary) --- ## Ingesting WARCs Place the WARC files you would to explore with WARC-GPT under `./warc` and run the following command to: - Extract text from all the `text/html` and `application/pdf` response records present in the WARC files. - Generate text embeddings for this text. WARC-GPT will automatically split text based on the embedding model's context window. - Store these embeddings in a vector store, so it can be used as WARC-GPT's knowledge base. ```bash poetry run flask ingest # May help with performance in certain cases: only ingest 1 chunk of text at a time. poetry run flask ingest --batch-size 1 ``` **Note:** Running `ingest` clears the `./chromadb` folder. [☝️ Summary](#summary) --- ## Starting the server The following command will start WARC-GPT's server on port `5000`. ```bash poetry run flask run # Not: Use --port to use a different port ``` [☝️ Summary](#summary) --- ## Interacting with the WEB UI Once the server is started, the application's web UI should be available on `http://localhos
Excerpt of 8,036 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:96acdff750bf0e58, topic:rag, desc:retrieval-augmented generation, desc:retrieval augmented