Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
| Date | Stars |
|---|---|
| 2026-07-24 | 814 |
| 2026-07-25 | 816 |
| 2026-07-28 | 816 |
| 2026-07-30 | 816 |
| 2026-08-06 | 816 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# SiteOne Crawler
SiteOne Crawler is a powerful and easy-to-use **website analyzer, cloner, and converter** designed for developers seeking security and performance insights, SEO specialists identifying optimization opportunities, and website owners needing reliable backups and offline versions.
**Now rewritten in Rust** for maximum performance, minimal resource usage, and zero runtime dependencies. The transition from PHP+Swoole to Rust resulted in **25% faster execution** and **30% lower memory consumption** while producing identical output.
**Discover the SiteOne Crawler advantage:**
* **Run Anywhere:** Single native binary for **🪟 Windows**, **🍎 macOS**, and **🐧 Linux** (x64 & arm64). No runtime dependencies.
* **Work Your Way:** Launch the binary without arguments for an **interactive wizard** 🧙 with 10 preset modes, use the extensive **command-line interface** 📟 ([releases](https://github.com/janreges/siteone-crawler/releases), [▶️ video](https://www.youtube.com/watch?v=25T_yx13naA&list=PL9mElgTe-s1Csfg0jXWmDS0MHFN7Cpjwp)) for automation and power, or enjoy the intuitive **desktop GUI application** 💻 ([GUI app](https://github.com/janreges/siteone-crawler-gui), [▶️ video](https://www.youtube.com/watch?v=rFW8LNEVNdw)) for visual control.
* **Rich Output Formats:** Interactive **HTML audit report** 📊 with sortable tables and quality scoring (0.0-10.0) (see [nextjs.org sample](https://crawler.siteone.io/html/2024-08-23/forever/cl8xw4r-fdag8wg-44dd.html)), detailed **JSON** for programmatic consumption, and human-readable **text** for terminal. Send HTML reports directly to your inbox via **built-in SMTP mailer** 📧.
* **CI/CD Integration:** Built-in **quality gate** (`--ci`) with configurable thresholds — exit code 10 on failure enables automated deployment blocking. Also useful for **cache warming** — crawling the entire site after deployment populates your reverse proxy/CDN cache.
* **Offline & Markdown Power:** Create complete **offline clones** 💾 for browsing without a server ([nextjs.org clone](https://crawler.siteone.io/examples-exports/nextjs.org/)) or convert entire websites into clean **Markdown** 📝 — perfect for backups, documentation, or feeding content to AI models ([examples](https://github.com/janreges/siteone-crawler-markdown-examples/)).
* **Deep Crawling & Analysis:** Thoroughly crawl every page and asset, identify errors (404s, redirects), generate **sitemaps** 🗺️, and even get **email summaries** 📧 (watch [▶️ video example](https://www.youtube.com/watch?v=PHIFSOmk0gk)).
* **Learn More:** Dive into the 🌐 [Project Website](https://crawler.siteone.io/), explore the detailed [Documentation](https://crawler.siteone.io/configuration/command-line-options/), or check the [JSON](docs/JSON-OUTPUT.md)/[Text](docs/TEXT-OUTPUT.md) output specs.
GIF animation of the crawler in action (also available as a [▶️ video](https://www.youtube.com/watch?v=25T_yx13naA&list=PL9mElgTe-s1Csfg0jXWmDS0MHFN7Cpjwp)):
## Table of contents
- [✨ Features](#-features)
* [🕷️ Crawler](#️-crawler)
* [🛠️ Dev/DevOps assistant](#️-devdevops-assistant)
* [📊 Analyzer](#-analyzer)
* [📧 Reporter](#-reporter)
* [💾 Offline website generator](#-offline-website-generator)
* [📝 Website to markdown converter](#-website-to-markdown-converter)
* [🗺️ Sitemap generator](#️-sitemap-generator)
* [🤖 AI assistant (optional)](#-ai-assistant-optional)
* [🌐 Browser rendering (optional)](#-browser-rendering-optional)
- [🚀 Installation](#-installation)
* [📦 Pre-built binaries](#-pre-built-binaries)
* [🍺 Homebrew (macOS / Linux)](#-homebrew-macos--linux)
* [🐧 Debian / Ubuntu (apt)](#-debian--ubuntu-apt)
* [🎩 Fedora / RHEL (dnf)](#-fedora--rhel-dnf)
* [🦎 openSUSE / SLES (zypper)](#-opensuse--sles-zypper)
* [🏔️ Alpine Linux (apk)](#️-alpine-linux-apk)
* [🔨 Build from source](#-build-from-sourcExcerpt of 79,239 characters
Read on GitHub
530
Brian Teeman · United Kingdom
4
2
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:44b2d877d917a9ba, topic:testing, readme:scoring
matched fp:44b2d877d917a9ba, topic:crawler