Top AI Repos — open-source AI, indexed and scored
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
Top AI Repos tracks AI repositories on GitHub and answers two different questions about each one: is it moving right now, and would you bet a product on it.
A recurrent neural network for generating little stories about images
| Date | Stars |
|---|---|
| 2026-07-31 | 2960 |
| 2026-08-03 | 2960 |
| 2026-08-04 | 2960 |
| 2026-08-06 | 2960 |
Today
— stars today
This week
— stars this week
This month
— stars this month
Momentum
0.0
growth rate 0.00%/day
# neural-storyteller neural-storyteller is a recurrent neural network that generates little stories about images. This repository contains code for generating stories with your own images, as well as instructions for training new models. <img src="https://github.com/ryankiros/neural-storyteller/blob/master/images/ex1.jpg" height="220px" align="left"> *We were barely able to catch the breeze at the beach , and it felt as if someone stepped out of my mind . She was in love with him for the first time in months , so she had no intention of escaping . The sun had risen from the ocean , making her feel more alive than normal . She 's beautiful , but the truth is that I do n't know what to do . The sun was just starting to fade away , leaving people scattered around the Atlantic Ocean . I d seen the men in his life , who guided me at the beach once more .* [Samim](http://samim.io/) has made an awesome blog post with lots of results [here](https://medium.com/@samim/generating-stories-about-images-d163ba41e4ed). Some more results from an older model trained on Adventure books can be found [here](http://www.cs.toronto.edu/~rkiros/adv_L.html). The whole approach contains 4 components: * [skip-thought vectors](https://github.com/ryankiros/skip-thoughts) * [image-sentence embeddings](https://github.com/ryankiros/visual-semantic-embedding) * [conditional neural language models](https://github.com/ryankiros/skip-thoughts/tree/master/decoding) * style shifting (described in this project) The 'style-shifting' operation is what allows our model to transfer standard image captions to the style of stories from novels. The only source of supervision in our models is from [Microsoft COCO](http://mscoco.org/) captions. That is, we did not collect any new training data to directly predict stories given images. Style shifting was inspired by [A Neural Algorithm of Artistic Style](http://arxiv.org/abs/1508.06576) but the technical details are completely different. ## How does it work? We first train a recurrent neural network (RNN) decoder on romance novels. Each passage from a novel is mapped to a skip-thought vector. The RNN then conditions on the skip-thought vector and aims to generate the passage that it has encoded. We use romance novels collected from the BookCorpus [dataset](http://www.cs.toronto.edu/~mbweb/). Parallel to this, we train a visual-semantic embedding between COCO images and captions. In this model, captions and images are mapped into a common vector space. After training, we can embed new images and retrieve captions. Given these models, we need a way to bridge the gap between retrieved image captions and passages in novels. That is, if we had a function F that maps a collection of image caption vectors **x** to a book passage vector F(**x**), then we could feed F(**x**) to the decoder to get our story. There is no such parallel data, so we need to construct F another way. It turns out that skip-thought vectors have some intriguing properties that allow us to construct F in a really simple way. Suppose we have 3 vectors: an image caption **x**, a "caption style" vector **c** and a "book style" vector **b**. Then we define F as F(**x**) = **x** - **c** + **b** which intuitively means: keep the "thought" of the caption, but replace the image caption style with that of a story. Then, we simply feed F(**x**) to the decoder. How do we construct **c** and **b**? Here, **c** is the mean of the skip-thought vectors for Microsoft COCO training captions. We set **b** to be the mean of the skip-thought vectors for romance novel passages that are of length > 100. #### What kind of biases work? Skip-thought vectors are sensitive to: - length (if you bias by really long passages, it will decode really long stories) - punctuation - vocabulary - syntactic style (loosely speaking) For the last point, if you bias using text all written the same way the stories you get will also be written the same way. #### What can the decoder
Excerpt of 9,098 characters
Read on GitHub
19
3
Jared Lumpe · United States
1
Would you bet a product on this? Bounded 0–100 and slow moving.
matched fp:62348cdb2f8f85c7, llm:Description: 'A recurrent neural network for generating little stories about images' (image captioning/story generation). Language: Python.
matched fp:62348cdb2f8f85c7, llm:Description: 'A recurrent neural network for generating little stories about images' (image captioning/story generation). Language: Python.
matched fp:62348cdb2f8f85c7, llm:Description: 'A recurrent neural network for generating little stories about images' (image captioning/story generation). Language: Python.
matched fp:62348cdb2f8f85c7, llm:Description: 'A recurrent neural network for generating little stories about images' (image captioning/story generation). Language: Python.