Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
| Date | Stars |
|---|---|
| 2026-07-24 | 1888 |
| 2026-07-25 | 1888 |
| 2026-07-28 | 1888 |
| 2026-07-30 | 1888 |
| 2026-07-31 | 1889 |
| 2026-08-06 | 1890 |
Today
+1 stars today
This week
+2 stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.11%/day
Petastorm
=========
.. image:: https://github.com/uber/petastorm/actions/workflows/unittest.yml/badge.svg?branch=master
:target: https://github.com/uber/petastorm/actions/workflows/unittest.yml
:alt: Build Status
.. image:: https://codecov.io/gh/uber/petastorm/branch/master/graph/badge.svg
:target: https://codecov.io/gh/uber/petastorm/branch/master
:alt: Code coverage
.. image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
:target: https://img.shields.io/badge/License-Apache%202.0-blue.svg
:alt: License
.. image:: https://badge.fury.io/py/petastorm.svg
:target: https://pypi.org/project/petastorm
:alt: Latest Version
.. inclusion-marker-start-do-not-remove
.. contents::
Petastorm is an open source data access library developed at Uber ATG. This library enables single machine or
distributed training and evaluation of deep learning models directly from datasets in Apache Parquet
format. Petastorm supports popular Python-based machine learning (ML) frameworks such as
`Tensorflow <http://www.tensorflow.org/>`_, `PyTorch <https://pytorch.org/>`_, and
`PySpark <http://spark.apache.org/docs/latest/api/python/pyspark.html>`_. It can also be used from pure Python code.
Documentation web site: `<https://petastorm.readthedocs.io>`_
Installation
------------
.. code-block:: bash
pip install petastorm
There are several extra dependencies that are defined by the ``petastorm`` package that are not installed automatically.
The extras are: ``tf``, ``tf_gpu``, ``torch``, ``opencv``, ``docs``, ``test``.
For example to trigger installation of GPU version of tensorflow and opencv, use the following pip command:
.. code-block:: bash
pip install petastorm[opencv,tf_gpu]
Generating a dataset
--------------------
A dataset created using Petastorm is stored in `Apache Parquet <https://parquet.apache.org/>`_ format.
On top of a Parquet schema, petastorm also stores higher-level schema information that makes multidimensional arrays into a native part of a petastorm dataset.
Petastorm supports extensible data codecs. These enable a user to use one of the standard data compressions (jpeg, png) or implement her own.
Generating a dataset is done using PySpark.
PySpark natively supports Parquet format, making it easy to run on a single machine or on a Spark compute cluster.
Here is a minimalistic example writing out a table with some random data.
.. code-block:: python
import numpy as np
from pyspark.sql import SparkSession
from pyspark.sql.types import IntegerType
from petastorm.codecs import ScalarCodec, CompressedImageCodec, NdarrayCodec
from petastorm.etl.dataset_metadata import materialize_dataset
from petastorm.unischema import dict_to_spark_row, Unischema, UnischemaField
# The schema defines how the dataset schema looks like
HelloWorldSchema = Unischema('HelloWorldSchema', [
UnischemaField('id', np.int32, (), ScalarCodec(IntegerType()), False),
UnischemaField('image1', np.uint8, (128, 256, 3), CompressedImageCodec('png'), False),
UnischemaField('array_4d', np.uint8, (None, 128, 30, None), NdarrayCodec(), False),
])
def row_generator(x):
"""Returns a single entry in the generated dataset. Return a bunch of random values as an example."""
return {'id': x,
'image1': np.random.randint(0, 255, dtype=np.uint8, size=(128, 256, 3)),
'array_4d': np.random.randint(0, 255, dtype=np.uint8, size=(4, 128, 30, 3))}
def generate_petastorm_dataset(output_url='file:///tmp/hello_world_dataset'):
rowgroup_size_mb = 256
spark = SparkSession.builder.config('spark.driver.memory', '2g').master('local[2]').getOrCreate()
sc = spark.sparkContext
# Wrap dataset materialization portion. Will take care of setting up spark environment variables as
# well as save petastorm specific metadata
rows_count = 10
with materialize_dataset(spark, output_url, HelloWorExcerpt of 17,482 characters
Read on GitHub
65
WeichenXu · @databricks
64
Uber · United States
30
21
15
5
5
5
Travis Addair · @uber · United States
4
Praateek Mahajan · NVIDIA · United States
4
Kim Hammar · Imperial College · United Kingdom
3
2
Vivek Panyam · Uber, FB, Google
2
2
Karthikeyan Singaravelan · Visa Inc · India
2
2
2
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:957af6ad00f05114, topic:deep-learning, topic:pytorch, topic:tensorflow