Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
| Date | Stars |
|---|---|
| 2026-07-31 | 1659 |
| 2026-08-05 | 1658 |
| 2026-08-06 | 1658 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Spark Python Notebooks
[](https://gitter.im/jadianes/spark-py-notebooks?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
This is a collection of [IPython notebook](http://ipython.org/notebook.html)/[Jupyter](https://jupyter.org/)
notebooks intended to train the reader on different [Apache Spark](http://spark.apache.org/) concepts, from
basic to advanced, by using the **Python** language.
If Python is not your language, and it is R, you may want to have a look at our [R on Apache Spark (SparkR) notebooks](https://github.com/jadianes/spark-r-notebooks) instead. Additionally, if your are interested in being introduced to some basic Data Science
Engineering, you might find [these series of tutorials](https://github.com/jadianes/data-science-your-way)
interesting. There we explain different concepts and applications
using Python and R.
## Instructions
A good way of using these notebooks is by first cloning the repo, and then
starting your own [IPython notebook](http://ipython.org/notebook.html)/[Jupyter](https://jupyter.org/) in
**pySpark mode**. For example, if we have a *standalone* Spark installation
running in our `localhost` with a maximum of 6Gb per node assigned to IPython:
MASTER="spark://127.0.0.1:7077" SPARK_EXECUTOR_MEMORY="6G" IPYTHON_OPTS="notebook --pylab inline" ~/spark-1.5.0-bin-hadoop2.6/bin/pyspark
Notice that the path to the `pyspark` command will depend on your specific
installation. So as requirement, you need to have
[Spark installed](https://spark.apache.org/docs/latest/index.html) in
the same machine you are going to start the `IPython notebook` server.
For more Spark options see [here](https://spark.apache.org/docs/latest/spark-standalone.html). In general it works the rule of passing options
described in the form `spark.executor.memory` as `SPARK_EXECUTOR_MEMORY` when
calling IPython/pySpark.
## Datasets
We will be using datasets from the [KDD Cup 1999](http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html). The results
of this competition can be found [here](http://cseweb.ucsd.edu/~elkan/clresults.html).
## References
The reference book for these and other Spark related topics is:
- *Learning Spark* by Holden Karau, Andy Konwinski, Patrick Wendell, and Matei Zaharia.
## Notebooks
The following notebooks can be examined individually, although there is a more
or less linear 'story' when followed in sequence. By using the same dataset
they try to solve a related set of tasks with it.
### [RDD creation](https://github.com/jadianes/spark-py-notebooks/blob/master/nb1-rdd-creation/nb1-rdd-creation.ipynb)
About reading files and parallelize.
### [RDDs basics](https://github.com/jadianes/spark-py-notebooks/blob/master/nb2-rdd-basics/nb2-rdd-basics.ipynb)
A look at `map`, `filter`, and `collect`.
### [Sampling RDDs](https://github.com/jadianes/spark-py-notebooks/blob/master/nb3-rdd-sampling/nb3-rdd-sampling.ipynb)
RDD sampling methods explained.
### [RDD set operations](https://github.com/jadianes/spark-py-notebooks/blob/master/nb4-rdd-set/nb4-rdd-set.ipynb)
Brief introduction to some of the RDD pseudo-set operations.
### [Data aggregations on RDDs](https://github.com/jadianes/spark-py-notebooks/blob/master/nb5-rdd-aggregations/nb5-rdd-aggregations.ipynb)
RDD actions `reduce`, `fold`, and `aggregate`.
### [Working with key/value pair RDDs](https://github.com/jadianes/spark-py-notebooks/blob/master/nb6-rdd-key-value/nb6-rdd-key-value.ipynb)
How to deal with key/value pairs in order to aggregate and explore data.
### [MLlib: Basic Statistics and Exploratory Data Analysis](https://github.com/jadianes/spark-py-notebooks/blob/master/nb7-mllib-statistics/nb7-mllib-statistics.ipynb)
A notebook introducing Local Vector types, basic statistics
in MLlib for Exploratory Data AnaExcerpt of 7,620 characters
Read on GitHub
136
The Gitter Badger · Gitter
1
Bitdeli Chef · Bitdeli · United States
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:da6128e2f794ba5e, llm:Repository description and README: 'Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks'; topics include big-data, data-analysis, data-science, machine-learning, mllib, pyspark.
matched fp:da6128e2f794ba5e, llm:Repository description and README: 'Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks'; topics include big-data, data-analysis, data-science, machine-learning, mllib, pyspark.
matched fp:da6128e2f794ba5e, llm:Repository description and README: 'Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks'; topics include big-data, data-analysis, data-science, machine-learning, mllib, pyspark.