Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
TonY is a framework to natively run deep learning frameworks on Apache Hadoop.
| Date | Stars |
|---|---|
| 2026-07-24 | 707 |
| 2026-07-25 | 707 |
| 2026-07-28 | 707 |
| 2026-07-30 | 707 |
| 2026-07-31 | 707 |
| 2026-08-06 | 707 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# TonY
[](https://circleci.com/gh/tony-framework/TonY/tree/master)
[](https://bestpractices.coreinfrastructure.org/projects/5080)
TonY is a framework to _natively_ run deep learning jobs on [Apache Hadoop](http://hadoop.apache.org/).
It currently supports [TensorFlow](https://github.com/tensorflow/tensorflow), [PyTorch](https://github.com/pytorch/pytorch), [MXNet](https://github.com/apache/incubator-mxnet) and [Horovod](https://github.com/horovod/horovod).
TonY enables running either single node or distributed
training as a Hadoop application. This native connector, together with other TonY features, aims to run
machine learning jobs reliably and flexibly. For a quick overview of TonY and comparisons to other frameworks, please see
[this presentation](https://www.slideshare.net/ssuser72f42a/scaling-deep-learning-on-hadoop-at-linkedin).
## Compatibility Notes
TonY itself is compatible with [Hadoop 2.6.0](https://hadoop.apache.org/docs/r2.6.0/) (CDH5.11.0) and above. If you need GPU isolation from TonY, you need [Hadoop 2.10](https://hadoop.apache.org/docs/r2.10.0/) or higher for Hadoop 2, or [Hadoop 3.1.0](https://hortonworks.com/blog/gpus-support-in-apache-hadoop-3-1-yarn-hdp-3/) or higher for Hadoop 3.
## Build
TonY is built using [Gradle](https://github.com/gradle/gradle). To build TonY, run:
./gradlew build
This will automatically run tests, if want to build without running tests, run:
./gradlew build -x test
The jar required to run TonY will be located in `./tony-cli/build/libs/`.
## Usage
There are two ways to launch your deep learning jobs with TonY:
- Use a zipped Python virtual environment.
- Use Docker container.
### Use a zipped Python virtual environment
The difference between this approach and the one with Docker is
- You don't need to set up your Hadoop cluster with Docker support.
- There is no requirement on a Docker image registry.
As you know, nothing comes for free. If you don't want to bother setting your cluster with Docker support, you'd need to prepare a zipped virtual environment for your job and your cluster should have the same OS version as the computer which builds the Python virtual environment.
#### Python virtual environment in a zip
$ unzip -Z1 my-venv.zip | head -n 10
Python/
Python/bin/
Python/bin/rst2xml.py
Python/bin/wheel
Python/bin/rst2html5.py
Python/bin/rst2odt.py
Python/bin/rst2s5.py
Python/bin/pip2.7
Python/bin/saved_model_cli
Python/bin/rst2pseudoxml.pyc
#### TonY jar and tony.xml
MyJob/
> src/
> models/
mnist_distributed.py
tony.xml
tony-cli-0.4.7-all.jar
my-venv.zip # The additional file you need.
A similar `tony.xml` but without Docker related configurations:
$ cat tony/tony.xml
<configuration>
<property>
<name>tony.worker.instances</name>
<value>4</value>
</property>
<property>
<name>tony.worker.memory</name>
<value>4g</value>
</property>
<property>
<name>tony.worker.gpus</name>
<value>1</value>
</property>
<property>
<name>tony.ps.memory</name>
<value>3g</value>
</property>
</configuration>
Then you can launch your job:
$ java -cp "`hadoop classpath --glob`:MyJob/*:MyJob" \
com.linkedin.tony.cli.ClusterSubmitter \
-executes models/mnist_distributed.py \ # relative path to model program inside the src_dir
-task_params '--input_dir /path/to/hdfs/input --output_dir /path/to/hdfs/output \
-python_venv my-venv.zip \
-python_binary_path Python/bin/python \ # reExcerpt of 13,350 characters
Read on GitHub
100
77
31
27
17
14
Gonzalo Gasca Meza · United States
10
9
9
7
6
Kevin Su · @unionai · Taiwan
5
2
Howard Liberty
2
2
2
2
2
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c81cf0c3aadcbf99, topic:deep-learning, topic:tensorflow, readme:distributed training