Exploring different generative tools (GANs) and combining them with natural language processing neural networks (e.g. CLIP) to create videos with a smooth interpolation between concepts of each sentence and move within that space to generate continuous imagery.
The final tool can be used to create story narratives, and music videos based on the lyrics or the creator's imagination.
The final tool can be used to create story narratives, and music videos based on the lyrics or the creator's imagination.
These images were created using CLIP+DALL-E from the corresponding text. The video interpolates between the space of text embeddings to create a smooth transition between frames.
It was a warm summer day
And Alice was getting bored sitting by her sister
who had her nose buried in a book
he shook his head, then disappeared into the rabbit hole
"I must find out why he's in such a hurry", cried Alice
"When will I reach the bottom of this dreadful hole", she said
This key must fit one of the doors, she said
"But I'm too big to fit through such a little door, maybe the potion on the table will help"
And she drank it
Here is the video connecting the story frames:
Different GANs were used to compare their performance.
BigGAN is trained on natural images such as animals, flowers, etc. Therefore the results look like nature photos, Whereas DALL-E shows a diverse visualization of concepts.
