Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Getting started with Puppeteer and Chrome Headless for Web Scraping
| Date | Stars |
|---|---|
| 2026-07-24 | 2362 |
| 2026-07-25 | 2362 |
| 2026-07-28 | 2362 |
| 2026-07-30 | 2362 |
| 2026-08-06 | 2362 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Getting started with Puppeteer and Chrome Headless for Web Scraping
**Here is a link to [Medium Article](https://medium.com/@e_mad_ehsan/getting-started-with-puppeteer-and-chrome-headless-for-web-scrapping-6bf5979dee3e)**
**Here is the [Chinese Version](https://github.com/csbun/thal) thanks to [@csbun](https://github.com/csbun/)**
[`Puppeteer`](https://github.com/GoogleChrome/puppeteer) is official tool for Chrome Headless by Google Chrome team. Since the official announcement of Chrome Headless, many of the industry standard libraries for automated testing have been discontinued by their maintainers. Including **PhantomJS**. **Selenium IDE for Firefox** has been discontinued due to lack of maintainers.
For sure, Chrome being the market leader in web browsing, **Chrome Headless** is going to industry leader in **Automated Testing** of web applications. So, I have put together this starter guide on how to get started with `Web Scraping` in **Chrome Headless**.
## TL;DR
In this guide we will scrape GitHub, login to it and extract and save emails of users using `Chrome Headless`, `Puppeteer`, `Node` and `MongoDB`. Don't worry GitHub have rate limiting mechanism in place to keep you under control but this post will give you good idea on Scrapping with Chrome Headless and Node. Also, alway stay updated with the [documentation](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md) because `Puppeteer` is under development and APIs are prone to changes.
## Getting Started
Before we start, we need following tools installed. Head over to their websites and install them.
* [Node 8.+](https://nodejs.org)
* [MongoDB](http://mongodb.com)
## Project setup
Start off by making the project directory
```
$ mkdir thal
$ cd thal
```
Initiate NPM. And put in the necessary details.
```
$ npm init
```
Install `Puppeteer`. Its not stable and repository is updated daily. If you want to avail the latest functionality you can install it directly from its GitHub repository.
```
$ npm i --save puppeteer
```
Puppeteer includes its own chrome / chromium, that is guaranteed to work headless. So each time you install / update puppeteer, it will download its specific chrome version.
## Coding
We will start by taking a screenshot of the page. This is code from their documentation.
### Screenshot
```js
const puppeteer = require('puppeteer');
async function run() {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://github.com');
await page.screenshot({ path: 'screenshots/github.png' });
browser.close();
}
run();
```
If its your first time using `Node` 7 or 8, you might be unfamiliar with `async` and `await` keywords. To put `async/await` in really simple words, an async function returns a Promise. The promise when resolves might return the result that you asked for. But to do this in a single line, you tie the call to async function with `await`.
Save this in `index.js` inside project directory.
Also create the screenshots dir.
```
$ mkdir screenshots
```
Run the code with
```
$ node index.js
```
The screenshot is now saved inside `screenshots/` dir.
### Login to GitHub
If you go to GitHub and search for *john*, then click the users tab. You will see list of all users with names.
Some of them have made their emails publicly visible and some have chosen not to. But the thing is you can't see these emails without logging in. So, lets login. We will make heavy use of [Puppeteer documentation](https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md).
Add a file `creds.js` in project root. I highly recommend signing up for new account with a new dummy email because you **might** end up getting your account blocked.
```js
module.exports = {
username: '<GITHUB_USERNAME>',
password: '<GITHUB_PASSWORD>'
}
```
Add anExcerpt of 15,281 characters
Read on GitHub
45
3
2
2
1
1
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:3acc4384492799a9, topic:puppeteer