Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extraction easy.
| Date | Stars |
|---|---|
| 2026-07-31 | 301 |
| 2026-08-06 | 301 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Webpage to LLM Ready Input Text
Pre-processing webpage before giving it as input to the LLM improves extraction/scraping accuracy especially if you want to extract website and image links, tables required for most scraping operations like scraping an e-commerce website.
Use this library to turn any webpage/url to LLM friendly text. Fully open source alternative to firecrawl and jina reader api.
You can also refer to my other repo [AI-web_scraper](https://github.com/m92vyas/AI-web_scraper) for direct scraping tools that will do `web search` and `scrapes multiple links` with `just a simple query`. It supports multiple LLMs, Web Search and Extracts Data as per your written instructions.
---
### Update for Old Users:
We have switched from Selenium to Playwright for concurrent web scraping support. Kindly install the required playwright dependencies as given below.
---
### Install:
```python
# install llm-reader
pip install git+https://github.com/m92vyas/llm-reader.git
# install playwright dependencies. we are using playwright for async/concurrent web scraping support.
playwright install # to download browser.
playwright install-deps
```
### Import:
```python
from url_to_llm_text.get_html_text import get_page_source # you can also use your own code or other services to get the page source
from url_to_llm_text.get_llm_input_text import get_processed_text # pass html source text to get llm ready text
```
### Get processed LLM input text:
```python
url= <url_to_scrape>
# get html source text
# You can use your own function to get the html source text
page_source = await get_page_source(url)
# get LLM ready input text from html source text
llm_text = await get_processed_text(page_source, url)
print(llm_text)
### or use asyncio ###
# import asyncio
# url = <url_to_scrape>
# # creating a simple function here. View documentation for more parameter details.
# async def get_llm_ready_text(url: str) -> str:
# page_source = await get_page_source(url)
# llm_text = await get_processed_text(page_source, url)
# return llm_text
# llm_text = asyncio.run(get_llm_ready_text(url))
# print(llm_text)
```
---
### Example Usage:
suppose we want to scrape the product name, main product page link, image link and price from the url "https://www.ikea.com/in/en/cat/corner-sofas-10671/" using any openai model.
```python
import requests
from url_to_llm_text.get_html_text import get_page_source
from url_to_llm_text.get_llm_input_text import get_processed_text
url = "https://www.ikea.com/in/en/cat/corner-sofas-10671/"
# get page html source text using this library function or any other means
page_source = await get_page_source(url)
# get llm ready text and pass the text to your LLM prompt template
llm_text = await get_processed_text(page_source, url)
# prompt template
prompt_format = """extract the product name, product link, image link and price for all the products given in the below webpage. The format should be:
{{
"1": {{
"Product Name": ,
"Product Link": ,
"Image Link": ,
"Price":
}},
"2": {{
"Product Name": ,
...
}},
}}
webpage:
{llm_friendly_webpage_text}
"""
# calculate tokens and truncate the llm_text to fit your model context length and your requirements. sometimes you may need only initial part of the webpage.
# below we are manually truncating to 40000 characters. create a seperate function as per your need.
prompt = prompt_format.format(llm_friendly_webpage_text=llm_text[:40000])
api_key = <your openai api key>
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}"
}
payload = {
"model": "gpt-4o-mini",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": prompt
}
]}],
'seed': 0,
"temperature": 0,
"top_p": 0.001,
# "max_tokens": 1024, # if you want to limit the output tokens. this may keep the output json structure incoExcerpt of 8,859 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:522125f1c3d40211, topic:ai-agents, topic:llm-agent
matched fp:522125f1c3d40211, topic:llm
matched fp:522125f1c3d40211, topic:rag