Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A serverless architecture for orchestrating ETL jobs in arbitrarily-complex workflows using AWS Step Functions and AWS Lambda.
| Date | Stars |
|---|---|
| 2026-07-24 | 345 |
| 2026-07-25 | 345 |
| 2026-07-28 | 345 |
| 2026-07-30 | 345 |
| 2026-08-06 | 345 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
Table of Contents
=================
* [Introduction](#introduction)
* [The challenge of orchestrating an ETL workflow](#the-challenge-of-orchestrating-an-etl-workflow)
* [Example ETL workflow requirements](#example-etl-workflow-requirements)
* [The ETL orchestration architecture and events](#the-etl-orchestration-architecture-and-events)
* [Modeling the ETL orchestration workflow in AWS Step Functions](#modeling-the-etl-orchestration-workflow-in-aws-step-functions)
* [AWS CloudFormation templates](#aws-cloudformation-templates)
* [Handling failed ETL jobs](#handling-failed-etl-jobs)
* [Preparing your development environment](#preparing-your-development-environment)
* [Configuring the project](#configuring-the-project)
* [cloudformation/gluerunner-lambda-params.json](#cloudformationgluerunner-lambda-paramsjson)
* [cloudformation/athenarunner-lambda-params.json](#cloudformationathenarunner-lambda-paramsjson)
* [lambda/s3-deployment-descriptor.json](#lambdas3-deployment-descriptorjson)
* [cloudformation/glue-resources-params.json](#cloudformationglue-resources-paramsjson)
* [lambda/gluerunner/gluerunner-config.json](#lambdagluerunnergluerunner-configjson)
* [cloudformation/step-functions-resources-params.json](#cloudformationstep-functions-resources-paramsjson)
* [Build commands](#build-commands)
* [The packagelambda build command](#the-packagelambda-build-command)
* [The deploylambda build command](#the-deploylambda-build-command)
* [The createstack build command](#the-createstack-build-command)
* [The updatestack build command](#the-updatestack-build-command)
* [The deletestack build command](#the-deletestack-build-command)
* [The deploygluescripts build command](#the-deploygluescripts-build-command)
* [Putting it all together: Example usage of build commands](#putting-it-all-together-example-usage-of-build-commands)
* [License](#license)
<a name="introduction"></a>
# Introduction
Extract, transform, and load (ETL) operations collectively form the backbone of any modern enterprise data lake. It transforms raw data into useful datasets and, ultimately, into actionable insight. An ETL job typically reads data from one or more data sources, applies various transformations to the data, and then writes the results to a target where data is ready for consumption. The sources and targets of an ETL job could be relational databases in Amazon Relational Database Service (Amazon RDS) or on-premises, a data warehouse such as Amazon Redshift, or object storage such as Amazon Simple Storage Service (Amazon S3) buckets. Amazon S3 as a target is especially commonplace in the context of building a data lake in AWS.
AWS offers [AWS Glue](https://aws.amazon.com/glue/), which is a service that helps author and deploy ETL jobs. AWS Glue is a fully managed extract, transform, and load service that makes it easy for customers to prepare and load their data for analytics. Other AWS Services also can be used to implement and manage ETL jobs. They include: [AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html) (AWS DMS), [Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html) (using the Steps API), and even [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html).
<a name="the-challenge-of-orchestrating-an-etl-workflow"></a>
# The challenge of orchestrating an ETL workflow
How can we orchestrate an ETL workflow that involves a diverse set of ETL technologies? AWS Glue, AWS DMS, Amazon EMR, and other services support [Amazon CloudWatch Events](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html), which we could use to chain ETL jobs together. Amazon S3, the central data lake store, also supports CloudWatch Events. But relying on CloudWatch Events alone means that there's no single visual representation of tExcerpt of 32,582 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c4e5f18f5aee3a92, topic:etl, name:etl, desc:etl
matched fp:c4e5f18f5aee3a92, topic:serverless
matched fp:c4e5f18f5aee3a92, topic:orchestration