Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Library to scrape and clean web pages to create massive datasets.
| Date | Stars |
|---|---|
| 2026-07-24 | 2271 |
| 2026-07-25 | 2271 |
| 2026-07-28 | 2271 |
| 2026-07-30 | 2271 |
| 2026-08-06 | 2271 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# lazynlp
[](https://zenodo.org/badge/latestdoi/173005932)
[](LICENSE)
A straightforward library that allows you to crawl, clean up, and deduplicate webpages to create massive monolingual datasets. Using this library, you should be able to create datasets larger than the one used by OpenAI for GPT-2.
## Setup
This library uses Python 3.
1. Clone this library and cd into the lazynlp folder:
```
git clone https://github.com/chiphuyen/lazynlp.git
cd lazynlp
```
2. Install dependencies
``
pip3 install -r requirements.txt
``
3. Install the library
``
pip3 install .
``
If you want to uninstall the library, use:
``
pip3 uninstall lazynlp
``
## How to create a massive dataset using lazynlp:
### Step 1. Obtain URLs of the webpages you want to crawl
There are several major dumps of URLs available that you can use.
#### Reddit URLs
This is [the link to all submissions to Reddit by months](https://files.pushshift.io/reddit/submissions/). You can download the raw dump and process to get the links. Keep in mind that each of these dumps is huge (100MB - 1GB).
[@jcpeterson](https://github.com/jcpeterson) is kind enough to provide a list of deduplicated links with at least 3 karma that you can download [here](https://drive.google.com/file/d/1hRtA3zZ0K5UHKOQ0_8d0BIc_1VyxgY51/view?usp=sharing).
There are about 23M URLs from between 2015-06 to 2018-10, of which around 40 - 60 \% are bad URLs (URLs no longer exist or aren't scraper-friendly).
It means that after you've downloaded and cleaned all good URLs from this, you should have approx 10M webpages or 50GB of pure text.
#### Gutenberg
You can download the list of all URLs to US Gutenberg books [here](https://drive.google.com/file/d/1zIVaRaVqGP8VNBUT4eKAzW3gYWxNk728/view?usp=sharing). There are 50K books, which convert to about 14GB of pure text.
You can also run ``lazynlp.get_us_gutenberg_links()`` to get the same list. For example, if you want to get all the Gutenberg URLs and store it in the file ``us_gutenberg.urls``, run the following command. This might take half a day.
``
lazynlp.get_us_gutenberg_links('us_gutenberg.urls')
``
You can download the list of all URLs to Australian Gutenberg books [here](https://drive.google.com/file/d/1C5aSisXMC3S3OXBFbnETLeK3UTUXEXrC/view?usp=sharing). There are 4k books, which convert to about 1GB of pure text.
You can also run ``lazynlp.get_aus_gutenberg_links()`` to get the same list. For example, if you want to get all the Gutenberg URLs and store it in the file ``aus_gutenberg.urls``:
``
lazynlp.get_aus_gutenberg_links('aus_gutenberg.urls')
``
#### Wikipedia
You can download the Wikipedia dumps [here](https://dumps.wikimedia.org/).
### Step 2. Deduplicate URLs
You don't want to download the same URL multiple times. There are two functions that help you deduplicate all URLs:
``
lazynlp.dedup_lines(files, outfold)
``
This function takes in a list of files (in each file, each line is a URLs) and deduplicate each file against all previous files.
Save all the deduplicated files in outfold.
``
lazynlp.dedup_lines_from_new_file(original_files, new_file, outfile)
``
This function allows you to deduplicate a new file against all previously deduplicated files (original_files)
### Step 3. Download the URLs
If you want to download each webpage separately, call:
``
lazynlp.download_page(link, context=None, timeout=None)
``
If you want to download from a file that contains a list of URLs, call:
``
lazynlp.download_pages(link_file, folder, timeout=30, default_skip=True, extensions=[], domains=[])
``
"""
link_file:
file contains links to webpages to crawl. Each line contains one URL.
folder:
folder that you want to contain your downloaded pages.
timeout:
seconds to wait for a page to respond before abandoning it.
default_skip:
set to True if you want to automatically skip all URLs that contain domainsExcerpt of 10,439 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c1beab4ffb63ce7d, topic:nlp, topic:natural-language-processing
matched fp:c1beab4ffb63ce7d, topic:language-model
