Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Turn any webpage into structured data using LLMs
| Date | Stars |
|---|---|
| 2026-07-24 | 6885 |
| 2026-07-25 | 6885 |
| 2026-07-28 | 6885 |
| 2026-07-30 | 6885 |
| 2026-07-31 | 6894 |
| 2026-08-06 | 6894 |
Today
— stars today
This week
+9 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.13%/day
# LLM Scraper
<img width="1800" alt="Screenshot 2024-04-20 at 23 11 16" src="https://github.com/mishushakov/llm-scraper/assets/10400064/ab00e048-a9ff-43b6-81d5-2e58090e2e65">
LLM Scraper is a TypeScript library that allows you to extract structured data from **any** webpage using LLMs.
> [!IMPORTANT]
> **LLM Scraper was updated to version 2.0.**
>
> The new version comes with **Vercel AI SDK 6** support and updated examples.
### Features
- Supports GPT, Sonnet, Gemini, Llama, Qwen model series
- Schemas defined with Zod or JSON Schema
- Full type-safety with TypeScript
- Based on Playwright framework
- Streaming objects
- [Code-generation](#code-generation)
- Supports 6 formatting modes:
- `html` for loading pre-processed HTML
- `raw_html` for loading raw HTML (no processing)
- `markdown` for loading markdown
- `text` for loading extracted text (using [Readability.js](https://github.com/mozilla/readability))
- `image` for loading a screenshot (multi-modal only)
- `custom` for loading custom content (using a custom function)
**Make sure to give it a star!**
<img width="165" alt="Screenshot 2024-04-20 at 22 13 32" src="https://github.com/mishushakov/llm-scraper/assets/10400064/11e2a79f-a835-48c4-9f85-5c104ca7bb49">
## Getting started
1. Install the required dependencies from npm:
```
npm i zod playwright llm-scraper
```
2. Initialize your LLM:
**OpenAI**
```
npm i @ai-sdk/openai
```
```js
import { openai } from '@ai-sdk/openai'
const llm = openai('gpt-4o')
```
**Anthropic**
```
npm i @ai-sdk/anthropic
```
```js
import { anthropic } from '@ai-sdk/anthropic'
const llm = anthropic('claude-3-5-sonnet-20240620')
```
**Google**
```
npm i @ai-sdk/google
```
```js
import { google } from '@ai-sdk/google'
const llm = google('gemini-1.5-flash')
```
**Groq**
```
npm i @ai-sdk/openai
```
```js
import { createOpenAI } from '@ai-sdk/openai'
const groq = createOpenAI({
baseURL: 'https://api.groq.com/openai/v1',
apiKey: process.env.GROQ_API_KEY,
})
const llm = groq('llama3-8b-8192')
```
**Ollama**
```
npm i ollama-ai-provider-v2
```
```js
import { ollama } from 'ollama-ai-provider-v2'
const llm = ollama('llama3')
```
3. Create a new scraper instance provided with the llm:
```js
import LLMScraper from 'llm-scraper'
const scraper = new LLMScraper(llm)
```
## Example
In this example, we're extracting top stories from HackerNews:
```ts
import { chromium } from 'playwright'
import { z } from 'zod'
import { Output } from 'ai'
import { openai } from '@ai-sdk/openai'
import LLMScraper from 'llm-scraper'
// Launch a browser instance
const browser = await chromium.launch()
// Initialize LLM provider
const llm = openai('gpt-4o')
// Create a new LLMScraper
const scraper = new LLMScraper(llm)
// Open new page
const page = await browser.newPage()
await page.goto('https://news.ycombinator.com')
// Define schema to extract contents into
const schema = z.object({
top: z
.array(
z.object({
title: z.string(),
points: z.number(),
by: z.string(),
commentsURL: z.string(),
})
)
.length(5)
.describe('Top 5 stories on Hacker News'),
})
// Run the scraper
const { data } = await scraper.run(page, Output.object({ schema }), {
format: 'html',
})
// Show the result from LLM
console.log(data.top)
await page.close()
await browser.close()
```
Output
```js
[
{
title: "Palette lighting tricks on the Nintendo 64",
points: 105,
by: "ibobev",
commentsURL: "https://news.ycombinator.com/item?id=44014587",
},
{
title: "Push Ifs Up and Fors Down",
points: 187,
by: "goranmoomin",
commentsURL: "https://news.ycombinator.com/item?id=44013157",
},
{
title: "JavaScript's New Superpower: Explicit Resource Management",
points: 225,
by: "olalonde",
commentsURL: "hExcerpt of 5,385 characters
Read on GitHub
Mish Ushakov · @e2b-dev · Germany
101
4
Timothy Carambat · Mintplex Labs Inc
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:35c3a2fe6f86f000, topic:llm, topic:gpt, topic:llama
matched fp:35c3a2fe6f86f000, topic:browser-automation, topic:playwright, topic:puppeteer