Final Aquarium Video
We are finalising our project and getting ready for shipping. This means exporting the final project videos, including our AI fishtank.
We were hoping to test and maybe include some footage from Google’s Lumiere project, but it is yet to launch in the public realm.
Screenshot of examples of short videos generated by Lumiere.
Meta, similarly has an unreleased text-to-video application called Make-A-Video in development. The early research documented online shows some promise, but Lumiere seems to have better resolution overall at this point.
Screenshot of example from Make-A-Video.
To date, the winner in the emerging text to video space (based on available online info) is the OpenAi equivalent Sora. Just look at the generated video quality in the below example:
Screenshot example of text to video out using OpenAI’s Sora.
Based on this, and other examples shown on the Sora research page, it will become increasingly difficult to determine if footage is ‘real’ or AI generated. This will have major impacts for the film industry with job requirements likely shifting to include generative skills and different workflows for staging, special effects and editing. More generally, this shift will mean that cultures embracing generative AI will become avid consumers of simulated representations.
Since all of these applications are still in development and not available for public release, we were limited to current public software only. Unfortunately, we were unable to find any free options that would allow us to export at a high resolution and without watermarks. As such, Runwayml was our choice. We also tested Pika which is gaining traction in the generative AI video arena, but found that Runway was best for allowing multiple extensions and generating video from a starting image. Runway also has other AI functionality including the ability to animate and expand images.
As outlined previously, to create our longer sections of video, we had to export the max. video length (18 sec) and then restart a new video generation from the last video still. We also needed to ensure a (mostly) seamless blend and colour match where there was a noticeable shift between renderings. Over time, you can also see that the video increasingly lost quality and cohesion.
To try and claw some image quality back (and test some more AI tools), we had a look at some of the new AI video enhancers such as HitPaw - although there are a range of the others including VMake and Neural Love. HitPaw’s trial option did seem to produce a relatively good outcome, but there was a high subscription cost to use the product for video export. Most other options available included size restrictions for video content. In the end, we opted for Adobe’s After Effects to upscale the video using the ‘Detail-Preserving Upscale’ option under the Effects panel. This was a good option for us as the Adobe Creative Suite is part of the UTAS School of Creative Arts and Media software options. However, it does also start to reveal that some of the issues regarding accessibility for more commercial or ‘high end” AI tools for generating high resolution outputs.
The final Aquarium video sequence consists of five video sequences using different starting images created in Adobe Firefly or Runway. If you watch the videos you can detect moment when the generative process restarts (every 18 secs) and how the starting image degrades into increasing random fish and blob formations. There are some pretty funny moments… see if you can spot the strange living pink sand and random people walking across the frame ;)