Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
TensorFlowOnSpark brings TensorFlow programs to Apache Spark clusters.
| Date | Stars |
|---|---|
| 2026-07-24 | 3845 |
| 2026-07-25 | 3845 |
| 2026-07-28 | 3845 |
| 2026-07-30 | 3845 |
| 2026-08-06 | 3845 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
<!-- Copyright 2019 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. Please see LICENSE file in the project root for terms. --> # TensorFlowOnSpark > _TensorFlowOnSpark brings scalable deep learning to Apache Hadoop and Apache Spark clusters._ [](https://cd.screwdriver.cd/pipelines/6384) [](https://pypi.org/project/tensorflowonspark/) [](https://img.shields.io/pypi/dm/tensorflowonspark.svg) [](https://yahoo.github.io/TensorFlowOnSpark/) By combining salient features from the [TensorFlow](https://www.tensorflow.org) deep learning framework with [Apache Spark](http://spark.apache.org) and [Apache Hadoop](http://hadoop.apache.org), TensorFlowOnSpark enables distributed deep learning on a cluster of GPU and CPU servers. It enables both distributed TensorFlow training and inferencing on Spark clusters, with a goal to minimize the amount of code changes required to run existing TensorFlow programs on a shared grid. Its Spark-compatible API helps manage the TensorFlow cluster with the following steps: 1. **Startup** - launches the Tensorflow main function on the executors, along with listeners for data/control messages. 1. **Data ingestion** - **InputMode.TENSORFLOW** - leverages TensorFlow's built-in APIs to read data files directly from HDFS. - **InputMode.SPARK** - sends Spark RDD data to the TensorFlow nodes via a `TFNode.DataFeed` class. Note that we leverage the [Hadoop Input/Output Format](https://github.com/tensorflow/ecosystem/tree/master/hadoop) to access TFRecords on HDFS. 1. **Shutdown** - shuts down the Tensorflow workers and PS nodes on the executors. ## Table of Contents - [Background](#background) - [Install](#install) - [Usage](#usage) - [API](#api) - [Contribute](#contribute) - [License](#license) ## Background TensorFlowOnSpark was developed by Yahoo for large-scale distributed deep learning on our Hadoop clusters in Yahoo's private cloud. TensorFlowOnSpark provides some important benefits (see [our blog](https://developer.yahoo.com/blogs/157196317141/)) over alternative deep learning solutions. * Easily migrate existing TensorFlow programs with <10 lines of code change. * Support all TensorFlow functionalities: synchronous/asynchronous training, model/data parallelism, inferencing and TensorBoard. * Server-to-server direct communication achieves faster learning when available. * Allow datasets on HDFS and other sources pushed by Spark or pulled by TensorFlow. * Easily integrate with your existing Spark data processing pipelines. * Easily deployed on cloud or on-premise and on CPUs or GPUs. ## Install TensorFlowOnSpark is provided as a pip package, which can be installed on single machines via: ``` # for tensorflow>=2.0.0 pip install tensorflowonspark # for tensorflow<2.0.0 pip install tensorflowonspark==1.4.4 ``` For distributed clusters, please see our [wiki site](../../wiki) for detailed documentation for specific environments, such as our getting started guides for [single-node Spark Standalone](https://github.com/yahoo/TensorFlowOnSpark/wiki/GetStarted_Standalone), [YARN clusters](../../wiki/GetStarted_YARN) and [AWS EC2](../../wiki/GetStarted_EC2). Note: the Windows operating system is not currently supported due to [this issue](https://github.com/yahoo/TensorFlowOnSpark/issues/36). ## Usage To use TensorFlowOnSpark with an existing TensorFlow application, you can follow our [Conversion Guide](../../wiki/Conversion-Guide) to describe the required changes. Additionally, our [wiki site](../../wiki) has pointers to some presentations which provide an overview of the platform. **Note: since TensorFlow 2.x breaks API compatibility with TensorFlow 1.x, the examples have been updated accordingly.
Excerpt of 4,759 characters
Read on GitHub
Lee Yang
276
Nvidia
43
20
9
7
Winston Quock · Zillow Group
7
Yuming Wang · @eBay @Apache · China
5
5
3
3
2
1
Zhe Zhang · @ray-project · United States
1
1
Sina Sheikholeslami · KTH Climate Action Centre · Sweden
1
1
1
1
Qingsheng Bao · China
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:74e77af09e66cac0, topic:tensorflow, readme:deep learning framework