Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A comprehensive Data and Text Mining workflow for submissions and comments from any given public subreddit.
| Date | Stars |
|---|---|
| 2026-07-24 | 499 |
| 2026-07-25 | 499 |
| 2026-07-28 | 499 |
| 2026-07-30 | 499 |
| 2026-08-06 | 499 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Subreddit Analyzer
This project documents the process of downloading large amounts of `Reddit` submissions and comments using the `Pushshift` API to get interesting insights such as their distribution by weekday, hour and most common used words.
The project is divided in 3 main parts, the ETL process, the NLP pipeline and the generation of charts and insights.
The following are the summaries of the included scripts:
* subreddit_comments.py - A Python script that downloads a fixed amount of comments from the `Pushshift` API.
* subreddit_submissions.py - A Python script that downloads a fixed amount of submissions from the `Pushshift` API.
* subreddit_comments_alt.py - A Python script that downloads comments starting from the newest one to the first one of the specified date from the `Pushshift` API.
* subreddit_submissions_alt.py - A Python script that downloads submissions starting from the newest one to the first one of the specified date from the `Pushshift` API.
* step2.py - A Python script that uses `spaCy` to pass the downloaded comments into a NLP pipeline.
* step3.py - A Python script that generates several charts and insights from the submissions and comments datasets.
## Requirements
This project uses the following Python libraries
* requests - For querying the Pushshift API.
* spaCy - For passing the comments into a NLP pipeline.
* NumPy - For fast matrix operations.
* pandas - For analysing and getting insights from datasets.
* matplotlib - For creating graphs and plots.
* seaborn - For enhancing the style of matplotlib plots.
* wordcloud - For creating the word clouds.
## ETL Process
This project contains 2 ways of downloading submissions and commnets, by specifying a fixed amount or by specifying a target date.
The scripts allows you to specify any number of subreddits you desire.
```python
SUBREDDITS = ["mexico", "python", "learnpython"]
```
After that, you have the chance of specifying the number of submissions/comments or a target date.
```python
# Fixed amount version.
MAX_SUBMISSIONS = 10000
# Target date version.
# Year month and day.
TARGET_DATE = "2019-01-01"
TARGET_TIMESTAMP = datetime.fromisoformat(TARGET_DATE).timestamp()
```
*Note: These scripts use recursion to keep downloading data, it is required to set a new recursion limit. The following 2 lines of code wlil do that.*
```python
import sys
sys.setrecursionlimit(10000)
```
Now that we have our subreddits and targets defined we iterate over all the subreddits names and create their `csv.writer` objects.
```python
for subreddit in SUBREDDITS:
writer = csv.writer(open("./{}-submissions.csv".format(subreddit),
"w", newline="", encoding="utf-8"))
# Adding the header.
writer.writerow(["datetime", "author", "title", "url", "domain"])
print("Downloading:", subreddit)
download_submissions(subreddit=subreddit)
writer.writerows(SUBMISSIONS_LIST)
SUBMISSIONS_LIST.clear()
```
For these scripts I like to recycle a global list to add all the submissions or comments data.
All csv files have the subreddit name as a prefix, this is to avoid overwritting files by accident.
In the next part we have the function that downloads the data, selects specific fields and calls itself if it finds more data.
```python
base_url = "https://api.pushshift.io/reddit/submission/search/"
params = {"subreddit": subreddit, "sort": "desc",
"sort_type": "created_utc", "size": 500}
# After the first call of this function we will use the 'before' parameter.
if latest_timestamp != None:
params["before"] = latest_timestamp
with requests.get(base_url, params=params, headers=HEADERS) as response:
json_data = response.json()
total_submissions = len(json_data["data"])
latest_timestamp = 0
print("Downloading: {} submissions".format(total_submissions))
for item in json_data["data"]:
# We will only take 3 properties, the timestamp, author and url.
Excerpt of 25,788 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:f427993c68e9de87, topic:nlp, topic:spacy