Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Enterprise-grade, production-hardened, serverless data lake on AWS
| Date | Stars |
|---|---|
| 2026-07-24 | 482 |
| 2026-07-25 | 482 |
| 2026-07-28 | 482 |
| 2026-07-30 | 482 |
| 2026-08-06 | 482 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# Serverless Data Lake Framework (SDLF)
## Table of Contents
1. [Overview](#overview)
- [Cost](#cost)
2. [Prerequisites](#prerequisites)
3. [Deployment Steps](#deployment-steps)
4. [Deployment Validation](#deployment-validation)
5. [Running the Guidance](#running-the-guidance)
6. [Cleanup](#cleanup)
6. [Authors](#authors)
## Overview
An [AWS Professional Service](https://aws.amazon.com/professional-services/) open source initiative | [email protected]
<img align="left" src="docs/_static/sail-icon.png" width="50" height="44"> The Serverless Data Lake Framework (SDLF) is a collection of reusable artifacts aimed at accelerating the delivery of enterprise data lakes on AWS, shortening the deployment time to production from several months to a few weeks. It can be used by AWS teams, partners and customers to implement the foundational structure of a data lake following best practices.
A data lake gives your organization agility. It provides a repository where consumers can quickly find the data they need and use it in their business projects. However, building a data lake can be complex; there's a lot to think about beyond the storage of files. For example, how do you catalog the data so you know what you've stored? What ingestion pipelines do you need? How do you manage data quality? How do you keep the code for your transformations under source control? How do you manage development, test and production environments? Building a solution that addresses these use cases can take many weeks and this time can be better spent innovating with data and achieving business goals.
To learn more about SDLF, its constructs and data architectures, please visit the following pages:
- [Constructs](https://sdlf.readthedocs.io/en/latest/constructs/)
- [Architecture](https://sdlf.readthedocs.io/en/latest/architecture/)
### Cost
You are responsible for the cost of the AWS services used while running this Guidance. As of December 2024, the cost for running this guidance with the default settings in the eu-west-1 region (Ireland) is approximately $15 per month.
We recommend creating a [Budget](https://docs.aws.amazon.com/cost-management/latest/userguide/budgets-managing-costs.html) through [AWS Cost Explorer](https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) to help manage costs. Prices are subject to change. For full details, refer to the pricing webpage for each AWS service used in this Guidance.
The following table provides a sample cost breakdown for deploying this guidance with the default parameters in the eu-west-1 region (Ireland) for one month.
| AWS service | Dimensions | Cost [USD] |
| ----------- | ------------ | ------------ |
| Amazon S3 | 100,000 PUT, COPY, POST, LIST requests to S3 Standard per month | $3.50 |
| AWS Glue ETL jobs | 10 DPU per job | $0.75 |
| AWS Glue Crawlers | 3 crawlers | $0.50 |
| AWS Lambda | 180 requests per month| $0 |
| AWS Step Functions | 30 workflow requests and 10 state transitions per workflow, per month | $0 |
| Amazon EventBridge | 150 schedule invocations per month | $0 |
| Amazon Athena | 30 queries per month | $0 |
| AWS Lake Formation | - | $0 |
| Amazon SQS | 1 million FIFO queue requests | $0 |
| Amazon DynamoDB | 1 GB data storage size, 1 KB average item size| $1.05 |
| AWS KMS | 5 customer managed Customer Master Keys (CMK) with 2000000 symmetric requests per month | $11.00 |
## Prerequisites
AWS CloudShell can be used to deploy SDLF.
* git
* AWS CLI
* IAM deployment role
## Deployment Steps
The [SDLF workshop](https://sdlf.workshop.aws/) walks you through the deployment of a data lake.
**It is recommended to use the [latest stable release](https://github.com/awslabs/aws-serverless-data-lake-framework/releases) when setting up SDLF.**
**For users of SDLF 1.x, version 1 is still available on the [master branch](https:Excerpt of 7,641 characters
Read on GitHubWould you bet a product on this? Bounded 0–100 and slow moving.
matched fp:c089915aa21a8898, topic:data-engineering, topic:etl, readme:etl
matched fp:c089915aa21a8898, topic:serverless