Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
| Date | Stars |
|---|---|
| 2026-07-24 | 814 |
| 2026-07-25 | 814 |
| 2026-07-28 | 813 |
| 2026-07-30 | 813 |
| 2026-08-06 | 813 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<img align="left" width="150" style="padding:6px 23px 6px 0px;" src="img/till-logo.svg"> **DataHen Till** is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.
[](https://www.youtube.com/watch?v=D1VBVYTRo8g)
Till was architected to follow best practices that [DataHen](https://www.datahen.com) has accumulated over the years of scraping at a massive scale.
### Till easily integrates with your existing scrapers...
written in languages such as:
<img align="left" height="50" title="Python" style="padding:6px 23px 6px 0px;" src="img/integrations/python.png">
<img align="left" height="50" title="Nodejs" style="padding:6px 23px 6px 0px; " src= "img/integrations/nodejs.png">
<img align="left" height="50" title="Ruby" style="padding:6px 23px 6px 0px;" src= "img/integrations/ruby.png">
<img align="left" height="50" title="Go" style="padding:6px 23px 6px 0px;" src= "img/integrations/go.png">
<img height="50" title="Java" style="padding:6px 23px 6px 0px;" src= "img/integrations/java.png">
or frameworks such as:
<img align="left" height="50" title="Scrapy" style="padding:6px 23px 6px 0px;" src= "img/integrations/scrapy.png">
<img align="left" height="80" title="Puppeteer" style="padding:6px 23px 6px 0px;" src= "img/integrations/puppeteer.png">
<img align="left" height="50" title="Kimurai" style="padding:6px 23px 6px 0px;" src= "img/integrations/kimurai.png">
<img align="left" height="50" title="Colly" style="padding:6px 23px 6px 0px;" src= "img/integrations/colly.png">
<img align="left" height="50" title="Selenium" style="padding:6px 23px 6px 0px;" src= "img/integrations/selenium.png">
<br clear="left"/>
and many more...
# Table of Contents
* [Problems with Web Scraping](#problems-with-web-scraping)
* [Scaling Your Scraper](#scaling-your-scraper)
* [Blocked scraper](#blocked-scraper)
* [Scraper Maintenance](#scraper-maintenance)
* [Postmortem analysis & reproducability](#postmortem-analysis--reproducability)
* [Starting over from scratch when it fails mid-way](#starting-over-from-scratch-when-it-fails-mid-way)
* [Features](#features)
* [User-Agent randomizer](#user-agent-randomizer)
* [Proxy IP address rotation](#proxy-ip-address-rotation)
* [Sticky Sessions](#sticky-sessions)
* [Managing Cookies](#managing-cookies)
* [Request Logging](#request-logging)
* [HTTP Caching](#http-caching)
* [Global ID (GID)](#global-id-gid)
* [Request Interceptions](#request-interceptions)
* [How DataHen Till works](#how-datahen-till-works)
* [Installation](#installation)
* [Certificate Authority (CA) Certificates](#certificate-authority-ca-certificates)
* [Till Integrations](#till-integrations)
* [Python](#python)
* Scrapy
* [Node.js](#nodejs)
* Plain
* Puppeteer
* [Go](#go)
* net/http
* Colly
* [Ruby](#ruby)
* Kimurai
# Problems with Web Scraping
Web scraping is usually easy to get started, especially on a small scale. However, as you try to scale it up, it gets exponentially difficult. Scraping 10,000 records can easily be done with simple web scraper scripts in any programming language, but as you try to scrape millions of pages, you would need to architect and build features on your web scraping script that allows you to scale, maintain and unblock your scrapers.
**DataHen Till** solves the following problems:
## Scaling your scraper
Scraping to millions or even billions of records requires much more pre-planning. It's not simply running your existing web scraper script in a bigger CPU/Ram machine.
More thoughts are needed, such as:
- How to log massive amounts of HTTP requests.
- How to troubleshoot HTTP requests, when it fails at scale.
- How to minimize bandwidth usageExcerpt of 15,271 characters
Read on GitHub
97
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:e5988d53ee857c1c, topic:web-scraping, topic:crawler, readme:web scraping