Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Standalone TFRecord reader/writer with PyTorch data loaders
| Date | Stars |
|---|---|
| 2026-07-24 | 902 |
| 2026-07-25 | 902 |
| 2026-07-28 | 902 |
| 2026-07-30 | 902 |
| 2026-08-06 | 902 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# TFRecord reader and writer
This library allows reading and writing tfrecord files efficiently in python. The library also provides an IterableDataset reader of tfrecord files for PyTorch. Currently uncompressed and compressed gzip TFRecords are supported.
## Installation
```
pip3 install 'tfrecord[torch]'
```
## Usage
It's recommended to create an index file for each TFRecord file. Index file must be provided when using multiple workers, otherwise the loader may return duplicate records. You can create an index file for an individual tfrecord file with this utility program:
```
python3 -m tfrecord.tools.tfrecord2idx <tfrecord path> <index path>
```
To create "*.tfidnex" files for all "*.tfrecord" files in a directory run:
```
tfrecord2idx <data dir>
```
## Reading & Writing tf.train.Example
### Reading tf.Example records in PyTorch
Use TFRecordDataset to read TFRecord files in PyTorch.
```python
import torch
from tfrecord.torch.dataset import TFRecordDataset
tfrecord_path = "/tmp/data.tfrecord"
index_path = None
description = {"image": "byte", "label": "float"}
dataset = TFRecordDataset(tfrecord_path, index_path, description)
loader = torch.utils.data.DataLoader(dataset, batch_size=32)
data = next(iter(loader))
print(data)
```
Use MultiTFRecordDataset to read multiple TFRecord files. This class samples from given tfrecord files with given probability.
```python
import torch
from tfrecord.torch.dataset import MultiTFRecordDataset
tfrecord_pattern = "/tmp/{}.tfrecord"
index_pattern = "/tmp/{}.index"
splits = {
"dataset1": 0.8,
"dataset2": 0.2,
}
description = {"image": "byte", "label": "int"}
dataset = MultiTFRecordDataset(tfrecord_pattern, index_pattern, splits, description)
loader = torch.utils.data.DataLoader(dataset, batch_size=32)
data = next(iter(loader))
print(data)
```
### Infinite and finite PyTorch dataset
By default, `MultiTFRecordDataset` is infinite, meaning that it samples the data forever. You can make it finite by providing the appropriate flag
```
dataset = MultiTFRecordDataset(..., infinite=False)
```
### Shuffling the data
Both TFRecordDataset and MultiTFRecordDataset automatically shuffle the data when you provide a queue size.
```
dataset = TFRecordDataset(..., shuffle_queue_size=1024)
```
### Transforming input data
You can optionally pass a function as `transform` argument to perform post processing of features before returning.
This can for example be used to decode images or normalize colors to a certain range or pad variable length sequence.
```python
import tfrecord
import cv2
def decode_image(features):
# get BGR image from bytes
features["image"] = cv2.imdecode(features["image"], -1)
return features
description = {
"image": "bytes",
}
dataset = tfrecord.torch.TFRecordDataset("/tmp/data.tfrecord",
index_path=None,
description=description,
transform=decode_image)
data = next(iter(dataset))
print(data)
```
### Writing tf.Example records in Python
```python
import tfrecord
writer = tfrecord.TFRecordWriter("/tmp/data.tfrecord")
writer.write({
"image": (image_bytes, "byte"),
"label": (label, "float"),
"index": (index, "int")
})
writer.close()
```
### Reading tf.Example records in Python
```python
import tfrecord
loader = tfrecord.tfrecord_loader("/tmp/data.tfrecord", None, {
"image": "byte",
"label": "float",
"index": "int"
})
for record in loader:
print(record["label"])
```
## Reading & Writing tf.train.SequenceExample
SequenceExamples can be read and written using the same methods shown above with an extra argument
(`sequence_description` for reading and `sequence_datum` for writing) which cause the respective
read/write functions to treat the data as a SequenceExample.
### Writing SequenceExamples to file
```python
import tfrecord
writer = tfrecord.TFRecordWriter("/tmp/data.tfrecord")
writer.wrExcerpt of 6,980 characters
Read on GitHub
Vahid Kazemi · United States
36
Praateek Mahajan · NVIDIA · United States
4
2
Ota
2
1
1
1
1
1
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:4e390d10fdf6c94e, topic:pytorch, topic:tensorflow
matched fp:4e390d10fdf6c94e, topic:dataset, readme:dataset