Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
news-please - an integrated web crawler and information extractor for news that just works
| Date | Stars |
|---|---|
| 2026-07-24 | 2472 |
| 2026-07-25 | 2472 |
| 2026-07-28 | 2472 |
| 2026-07-30 | 2472 |
| 2026-08-06 | 2472 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# **news-please** #
[](https://pypi.org/project/news-please/)
[](http://dx.doi.org/10.5281/zenodo.4120316)
<img align="right" height="128px" width="128px" src="https://raw.githubusercontent.com/fhamborg/news-please/master/misc/logo/logo-256.png" />
news-please is an open source, easy-to-use news crawler that extracts structured information from almost any news website. It can recursively follow internal hyperlinks and read RSS feeds to fetch both most recent and also old, archived articles. You only need to provide the root URL of the news website to crawl it completely. news-please combines the power of multiple state-of-the-art libraries and tools, such as [scrapy](https://scrapy.org/), [Newspaper](https://github.com/AndyTheFactory/newspaper4k), and [readability](https://github.com/buriy/python-readability).
news-please also allows Python developers to use the crawling and extraction functionality within their own program. Moreover, news-please allows to conveniently [crawl and extract articles](/newsplease/examples/commoncrawl.py) from the (very) large news archive at commoncrawl.org.
If you want to contribute to news-please, please first read [here](#contributions).
## Extracted information
news-please extracts the following attributes from news articles. An examplary json file as extracted by news-please can be found [here](https://github.com/fhamborg/news-please/blob/master/newsplease/examples/sample.json).
* headline
* lead paragraph
* main text
* main image
* name(s) of author(s)
* publication date
* language
## Features
* **works out of the box**: install with pip, add URLs of your pages, run :-)
* run news-please conveniently using its [**CLI**](#run-the-crawler-via-the-cli) mode
* use it as a [**library**](#use-within-your-own-code-as-a-library) within your own software
* extract articles from [**commoncrawl.org's news archive**](#news-archive-from-commoncrawlorg)
### Modes and use cases
news-please supports three main use cases, which are explained in more detail in the following.
#### CLI mode
* stores extracted results in JSON files, PostgreSQL, ElasticSearch, Redis, or your own storage
* simple but extensive configuration (if you want to tweak the results)
* revisions: crawl articles multiple times and track changes
#### Library mode
* crawl and extract information given a list of article URLs
* to use news-please within your own Python code
#### News archive from commoncrawl.org
* commoncrawl.org provides an extensive, free-to-use archive of news articles from small and major publishers world wide
* news-please enables users to conveniently download and extract articles from commoncrawl.org
* you can optionally define filter criteria, such as news publisher(s) or the date period, within which articles need to be published
* clone the news-please repository, adapt the config section in [newsplease/examples/commoncrawl.py](/newsplease/examples/commoncrawl.py), and execute `python3 -m newsplease.examples.commoncrawl`
## Getting started
It's super easy, we promise!
### Installation
news-please runs on Python 3.8+.
```bash
$ pip install news-please
```
### Use within your own code (as a library)
You can access the core functionality of news-please, i.e. extraction of semi-structured information from one or more news articles, in your own code by using news-please in library mode. If you want to use news-please's full website extraction (given only the root URL) or continuous crawling mode (using RSS), you'll need to use the CLI mode, which is described later.
```python
from newsplease import NewsPlease
article = NewsPlease.from_url('https://www.nytimes.com/2017/02/23/us/politics/cpac-stephen-bannon-reince-priebus.html?hp')
print(article.title)
```
A sample of an extracted article can be found [here (as a JSON file)](https://github.com/fhamborg/news-please/blob/master/newsplease/eExcerpt of 14,564 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:4ba84efb57dd29a4, topic:crawler, desc:web crawler
matched fp:4ba84efb57dd29a4, topic:nlp
