Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Index your Gmail Inbox with Elasticsearch
| Date | Stars |
|---|---|
| 2026-07-24 | 2058 |
| 2026-07-25 | 2058 |
| 2026-07-28 | 2058 |
| 2026-07-30 | 2058 |
| 2026-08-06 | 2058 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
Elasticsearch For Beginners: Indexing your Gmail Inbox (and more: Supports any mbox and MH mailboxes)
=======================
#### What's this all about?
I recently looked at my Gmail inbox and noticed that I have well over 50k emails, taking up about 12GB of space but there is no good way to tell what emails take up space, who sent them to, who emails me, etc
Goal of this tutorial is to load an entire Gmail inbox into Elasticsearch using bulk indexing and then start querying the cluster to get a better picture of what's going on.
#### Prerequisites
Set up [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/guide/current/running-elasticsearch.html) and make sure it's running at [http://localhost:9200](http://localhost:9200)
A quick way to run Elasticsearch is using Docker: (the cors settings aren't really needed but come in handy if you want to use e.g. [dejavu](https://dejavu.appbase.io/) to explore the index)
```
docker run --name es -d -p 9200:9200 -e http.port=9200 -e http.cors.enabled=true -e 'http.cors.allow-origin=*' -e http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization -e http.cors.allow-credentials=true -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2
```
I use Python and [Tornado](https://github.com/tornadoweb/tornado/) for the scripts to import and query the data. Also `beautifulsoup4` for the stripping HTML/JS/CSS (if you want to use the body indexing flag).
Install the dependencies by running:
`pip3 install -r requirements.txt`
#### Aight, where do we start?
First, go [here](https://www.google.com/settings/takeout/custom/gmail) and download your Gmail mailbox, depending on the amount of emails you have accumulated this might take a while.
There's also a small `sample.mbox` file included in the repo for you to play around with while you're waiting for Google to prepare your download.
The downloaded archive is in the [mbox format](http://en.wikipedia.org/wiki/Mbox) and Python provides libraries to work with the mbox format so that's easy.
You can run the code (assuming Elasticsearch is running at localhost:9200) with the sammple mbox file like this:
```
$ python3 src/index_emails.py --infile=sample.mbox
[I index_emails:173] Starting import from file sample.mbox
[I index_emails:101] Upload: OK - upload took: 1033ms, total messages uploaded: 3
[I index_emails:197] Import done - total count 16
$
```
Note: All examples focus on Gmail inboxes. Substitute any `--infile=` parameters with `--indir=` pointing to an MH directory to make them work with MH mailboxes instead.
#### The Source Code
The overall program will look something like this:
```python
mbox = mailbox.mbox('emails.mbox') // or mailbox.MH('inbox/')
for msg in mbox:
item = convert_msg_to_json(msg)
upload_item_to_es(item)
print "Done!"
```
#### Ok, tell me more about the details
The full Python code is here: [src/index_emails.py](src/index_emails.py)
##### Turn mailbox into JSON
First, we got to turn the messages into JSON so we can insert it into Elasticsearch. [Here](http://nbviewer.ipython.org/github/furukama/Mining-the-Social-Web-2nd-Edition/blob/master/ipynb/Chapter%206%20-%20Mining%20Mailboxes.ipynb) is some sample code that was very useful when it came to normalizing and cleaning up the data.
A good first step:
```python
def convert_msg_to_json(msg):
result = {'parts': []}
for (k, v) in msg.items():
result[k.lower()] = v.decode('utf-8', 'ignore')
```
Additionally, you also want to parse and normalize the `From` and `To` email addresses:
```python
for k in ['to', 'cc', 'bcc']:
if not result.get(k):
continue
emails_split = result[k].replace('\n', '').replace('\t', '').replace('\r', '').replace(' ', '').encode('utf8').decode('utf-8', 'ignore').split(',')
result[k] = [ normalize_email(e) for e in emails_split]
if "from" in result:
result['from'] = normalize_email(result['from'])
```
EExcerpt of 10,233 characters
Read on GitHub
29
4
2
2
1
1
1
1
1
Tim Gates · IRESS · Australia
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:41176e31bb0d1a01, topic:tutorial, readme:tutorial