Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Real Time Big Data / IoT Machine Learning (Model Training and Inference) with HiveMQ (MQTT), TensorFlow IO and Apache Kafka - no additional data store like S3, HDFS or Spark required
| Date | Stars |
|---|---|
| 2026-07-24 | 419 |
| 2026-07-25 | 419 |
| 2026-07-28 | 419 |
| 2026-07-30 | 419 |
| 2026-07-31 | 419 |
| 2026-08-06 | 419 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Streaming Machine Learning at Scale from 100000 IoT Devices with HiveMQ, Apache Kafka and TensorFLow If you just want to get started and quickly start the demo in a few minutes, go to the [quick start](infrastructure/README.md) to setup the infrastructure (on GCP) and run the demo. You can also check out the 20min video recording with a live demo: [Streaming Machine Learning at Scale from 100000 IoT Devices with HiveMQ, Apache Kafka and TensorFLow](https://www.youtube.com/watch?v=7oVSLt0AZ3M). There is also a blog post with more details about [use cases for event streaming and streaming analytics in the automotive industry](https://www.kai-waehner.de/blog/2019/11/22/apache-kafka-automotive-industry-industrial-iot-iiot/). ## Motivation: Demo an IoT Scenario at Scale You want to see an IoT example at huge scale? Not just 100 or 1000 devices producing data, but a really scalable demo with millions of messages per second from tens of thousands of devices? This is the right demo for you! The demo shows how you can integrate with tens or hundreds of thousands IoT devices and process the data in real time. The demo use case is predictive maintenance (i.e. anomaly detection) in a connected car infrastructure to predict motor engine failures. ### Background: Cloud-Native MQTT and Kafka If you need more background about the challenges of building a scalable IoT infrastructure, the differences and relation between MQTT and Apache Kafka, and best practices for realizing a cloud-native IoT infrastructure based on Kubernetes, check out the slide deck "[Best Practices for Streaming IoT Data with MQTT and Apache Kafka](https://www.slideshare.net/KaiWaehner/best-practices-for-streaming-iot-data-with-mqtt-and-apache-kafka)". ### Background: Digital Twin with Kafka In addition to the predictive maintenance scenario with machine learning, we also implemented an example of a Digital Twin. More thoughts on this here: "[Apache Kafka as Digital Twin for Open, Scalable, Reliable Industrial IoT (IIoT)](https://www.kai-waehner.de/blog/2019/11/28/apache-kafka-industrial-iot-iiot-build-an-open-scalable-reliable-digital-twin/)". In our example, we use Kafka as ingestion layer and MongoDB for storage and analytics. However, this is [just one of various IoT architectures for building a digital twin with Apache Kafka](https://www.kai-waehner.de/blog/2020/03/25/architectures-digital-twin-digital-thread-apache-kafka-iot-platforms-machine-learning). ## Use Case: Anomaly Detection in Real Time for 100000+ Connected Cars Streaming Machine Learning in Real Time at Scale with [MQTT](http://mqtt.org/), [Apache Kafka](https://kafka.apache.org/), [TensorFlow](https://www.tensorflow.org) and [TensorFlow I/O](https://www.tensorflow.org/io): - Data Integration - Data Preprocessing - Model Training - Model Deployment - Real Time Scoring - Real Time Monitoring This project implements a scenario where you can integrate with tens of thousands (simulated) cars using a scalable MQTT platform and an event streaming platform. The demo trains new analytic models from streaming data - without the need for an additional data store - to do predictive maintenance on real time sensor data from cars:  We built two different analytic models using different approaches: - Unsupervised learning (used by default in our project): An Autoencoder neural network is trained to detect anomaly detection without any labeled data. - Supervised learning: A LSTM neural network is trained on labeled data to be able to do predictions about future sensor events. The [digital twin implementation with Kafka and MongoDB is discussed on its own page](infrastructure/kafka-connect/mongodb/README.md), including implementation details and configuration examples. Another example demonstrates how you can [store all sensor data in a data lake
Excerpt of 16,142 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:b2023c697c6d5bc1, topic:kubernetes
matched fp:b2023c697c6d5bc1, topic:tensorflow