Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Text Mining on the 2019 Mexican Government Report, covering from extracting text from a PDF file to plotting the results.
| Date | Stars |
|---|---|
| 2026-07-24 | 476 |
| 2026-07-25 | 476 |
| 2026-07-28 | 476 |
| 2026-07-30 | 476 |
| 2026-08-06 | 476 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Mexican Government Report Text Analysis
This repository documents the process of extracting text from a PDF, cleaning it, passing it through an `NLP` pipeline, and presenting the results with graphs.
The PDF is the government report of 2019 that was released on September 1st. The PDF is in the data folder.
## Requirements
This project uses the following Python libraries
* `PyPDF2` : For extracting text from PDF files.
* `spaCy` : For passing the extracted text into an NLP pipeline.
* `NumPy` : For fast matrix operations.
* `pandas` : For analysing and getting insights from datasets.
* `matplotlib` : For creating graphs and plots.
* `seaborn` : For enhancing the style of matplotlib plots.
* `geopandas` : For plotting maps.
If you are on Windows I strongly recommend that you use `Anaconda` and install `geopandas` using these commands:
```
conda install -c conda-forge geopandas
conda install -c conda-forge descartes
```
## PDF Extraction
The government report that we will process can be downloaded from the following url: https://www.gob.mx/primerinforme
For your convenience I have added the PDF file in the data folder.
Extracting text from PDF files is a bit unreliable, you lose the original formatting and there are times you don't get any text at all.
Fortunately, this PDF file is simple enough that extracting the text wasn't that complex but I encountered some challenges that I will go in more detail in this section.
For this task, we will use `PyPDF2` which is a well-known PDF library.
```python
reader = PyPDF2.PdfFileReader("informe.pdf")
full_text = ""
```
The page numbers in the PDF are not the same as the reported number of pages, we use this variable to keep track of both.
```python
pdf_page_number = 3
```
We will only retrieve the first 3 sections of the government report which are between pages 14 and 326. The reason for this is that only the 3 first sections contain a substantial amount of text.
```python
for i in range(14, 327):
# This block is used to remove the page number at the start of
# each page. The first if removes page numbers with one digit.
# The second if removes page numbers with 2 digits and the else
# statement removes page numbers with 3 digits.
if pdf_page_number <= 9:
page_text = reader.getPage(i).extractText().strip()[1:]
elif pdf_page_number >= 10 and pdf_page_number <= 99:
page_text = reader.getPage(i).extractText().strip()[2:]
else:
page_text = reader.getPage(i).extractText().strip()[3:]
full_text += page_text.replace("\n", "")
pdf_page_number += 1
```
The previous code block ensures that we get the cleanest possible output. It removes page numbers and builds us the full transcript to a single string.
There's a small issue when decoding the PDF file. We will manually fix all the weird characters with their correct equivalents.
```python
for item, replacement in CHARACTERS.items():
full_text = full_text.replace(item, replacement)
```
This actually took a few hours to complete, I had to manually map all weird characters with their correct equivalents, some examples are:
```python
CHARACTERS = {
"ç": "Á",
"⁄": "á",
"…": "É",
"”": "é",
"ê": "Í",
"™": 'í',
"î": "Ó"
}
```
We will remove all extra white spaces and finally we save the cleaned text into a .txt file.
```python
# This looks weird but that's the most practical way to remove double to quad white spaces.
full_text = full_text.replace(" ", " ").replace(
" ", " ").replace(" ", " ")
with open("transcript_clean.txt", "w", encoding="utf-8") as temp_file:
temp_file.write(full_text)
```
With the transcript cleaned and well encoded we are ready to run it through a `NLP` pipeline.
## NLP Pipeline
For this step we will make use of `spaCy`, this library is very well built and easy to use.
After installing it with `pip` you must install the Spanish model. You can do it by running the following command on your CMD or TermExcerpt of 15,323 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:5c9e09cb4de3bfe1, topic:nlp, topic:spacy