Generative Video Tests
This week I have been experimenting with different video AI generators to create the artificial fish tank for our installation. While text-to-image generators have become quite sophisticated over the last year, text-to-video option remain comparatively limited.
The majority of commercial text-to-video options seem geared towards businesses seeking sales avatars to reduce production costs. See: https://www.synthesia.io/.
Other text-to-video generators such as https://invideo.io/ai/ also seem designed for presentation-style formats and use stock footage scoured from the internet to generate the content. Meta has the https://makeavideo.studio/ in development. Although the current outputs appear low resolution and no testing app is available online.
The most promising creative AI video generator I could find is https://runwayml.com/. The application enables you to generate video from a text prompt, still image sample, or combined image and description.
I decided to test runway using a starting image of a fish tank as visual guide. To generate the starting image, I used Adobe Firefly and prompt for a ‘front view of a saltwater aquarium with different fish and coral’.
The results were surprisingly good and rival the visual outputs of other generative image applications such as Midjourney.
I selected the first fish tank image as the starting point. On the free runway plan, you can only generate video output in 4sec increments. Here is the first render:
As you can see generative video has some quirks in relation to rendering realistic swimming motion. If you don’t look too closely, your brain seems to fill in partial information and the aquarium scene seems legitimate. On closer inspection, you notice that many of the fish are oddly shaped and emerge from nowhere.
For the next test, I relied wholly on runway to generate the scene from the following text prompt: front view of saltwater aquarium with a different fish and coral.
Overall the output was very good and I particularly like the way in which runway rendered particles in the water. It has a great cinematic feel. However, the fish on the right hand side appears to be composed of two tails.
While runway only generates 4sec increments, it does allow you to extend the scene in further 4sec segments - although the rendering becomes increasingly unstable. I love the way in which everything turns into fish!