Interactivity, iOS and Occasional Intrigue with Sam Keene

July 24th, 2011

Transparent Video On The iPhone, It Can Be Done… With Flash!

The creative for a client project I worked on recently involved several video loops of actors shot on green screen. The videos needed to overlap, so the actors didn’t look like they were in discreet spaces, but in one seamless environment. The problem with this creative was that the iPhone SDK doesn’t include a video codec with alpha channel (ie animation codec with millions of colors +). I’m not aware of all the CODECs that are allowed but I have used .H264 and also .mov with PhotoJPEG.
I played around with using PNG image sequences in a UIView animation sequence, but the file size was an unacceptable 400MGs. I also had a look into some of the more complex aspects of AVFoundation using AVVideoCompositions with two layers, one for the video, and another for a video with a solid color representing the transparency. The idea was to then extract the color out, turning it into a transparent layer and combinig the layers back into a composition. However I wasn’t even sure if this would leave me with video with a transparent layer outside of the AVComposition, that could be placed on a PNG background, so I’d still be left with the same problem. It then occurred to me to test out Adobe’s AIR 2.7 for the iPhone to export using an .flv. It sounded crazy and I didn’t think it was going to work, but to my surprise it did.

Now I have absolutley no idea how Adobe package up a Flash app for iPhone, but I think it’s amazing that a video format that not only isn’t compatible with, but also has technical functionality that can’t be replicated on the native iOS SDK can work through a third party’s system. So just to share with you some of the results, I was attempting to play four transparent videos, all slightly overlapping with a png image in the background. The playback was unacceptably slow with this many streams. I then tried, two transparent videos with a PNG background and the playback was fine, I didn’t notice any frame grabbing or glitches, and one full screen iPhone video with transparency and a PNG image in the back ground ran fine!

Again, I’d like to express my amazement at being able to play an .flv with transparency on the iPhone. I have no idea how Adobe are able to do it, but I really hope Apple step up their game and include a video codec with alpha channel sometime soon. The only down side to using ARI 2.7 is the lack of support for native UIKit components, meaning you have to build everything in Flash/Flex.

If you want to play around with transparent video on the iPhone with Flash you’ll need to download the trial version of Flash Builder 4.5 from here, and check out this post on setting up your system to export from Flash Builder to your iPhone. You do need to be registered as an Apple dev, as to my knowledge there is no simulator in the Adobe set up, but it was all pretty straight forward (no less painful than having to do it through Apple and XCode).

Here’s the AS3 code, I just created a new Mobile ActionScript Project, as I’ve heard Flex Mobile Projects add unwanted overhead to your app (but don’t quote me on that). You’ll also need to make sure that the info in the project’s XML file (application descriptor file) matches all of your code signing identity info ie com.mycompany.myAppID
One thing to note is that I didn’t use the StageWebView object to place my video. According to Adobe’s docs StageWebView’s shouldn’t overlap with each other, so this wasn’t suitable for this project. The only piece of AIR specific code is the

new File(new File("app:/transparentVids/BB_bg.png").nativePath).url;

which is how we access files from the application bundle, the rest is just standard AS3. Again if you don’t have a need for transparency, I’d suggest you use a StageWebView with H264 CODEC the performance will probably be significantly better particularly on older iOS devices.

package
{
	import flash.display.StageAlign;
	import flash.display.StageScaleMode;
	import flash.display.MovieClip;
	import flash.events.*;
	import flash.media.Video;
	import flash.net.NetStream;
	import flash.net.NetConnection;
	import flash.filesystem.File;
	import flash.net.URLRequest;
	import flash.display.Sprite;
	import flash.display.Loader;
	
	public class Main extends Sprite
	{
		private var ns:NetStream;
		private var ns2:NetStream;
		private var ns3:NetStream;
		
		public function Main()
		{
			super();
			stage.align = StageAlign.TOP_LEFT;
			stage.scaleMode = StageScaleMode.NO_SCALE;
			this.init();
		}
		
		private function init():void
		{
			var bgFilePath:String = new File(new File("app:/transparentVids/BB_bg.png").nativePath).url;
			var fPath:String = new File(new File("app:/transparentVids/myTransparentVid1.flv").nativePath).url;
			var fPath2:String = new File(new File("app:/transparentVids/myTransparentVid2.flv").nativePath).url;
			
			var nc:NetConnection = new NetConnection();
			nc.connect(null);
			this.ns=new NetStream(nc);
			ns.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncErrorHandler);
			ns.play(fPath);
			
			var nc2:NetConnection = new NetConnection();
			nc2.connect(null);
			this.ns2=new NetStream(nc2);
			ns2.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncErrorHandler);
			ns2.play(fPath2);
			
			var nc3:NetConnection = new NetConnection();
			nc3.connect(null);
			this.ns3=new NetStream(nc3);
			ns3.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncErrorHandler);
			ns3.play(fPath);
			
			var nc3:NetConnection = new NetConnection();
			nc3.connect(null);
			this.ns3=new NetStream(nc3);
			ns3.addEventListener(AsyncErrorEvent.ASYNC_ERROR, asyncErrorHandler);
			ns3.play(fPath2);
			
			var loader:Loader = new Loader();
			addChild(loader);
			loader.load( new URLRequest(bgFilePath));
			
			var vid:Video = new Video();
			vid.attachNetStream(ns);
			vid.width = 320;
			vid.height = 120;
			addChild(vid);
			var vid2:Video = new Video();
			vid2.attachNetStream(ns2);
			vid2.y = 120;
			vid2.width = 320;
			vid2.height = 120;
			addChild(vid2);
			var vid3:Video = new Video();
			vid3.attachNetStream(ns3);
			vid3.y = 240;
			vid3.width = 320;
			vid3.height = 120;
			addChild(vid3);
			
		}
		
		private function asyncErrorHandler(event:AsyncErrorEvent):void {
			//handle error
		}
	}
}

9 Comments »

  1. That was an interesting read! I would love to read more posts from you ^^b Bookmarked this page =]

    Comment by jhon — October 4, 2011 @ 9:04 am
  2. You have the monopoly on useful information—aren’t monpoloies illegal? ;)

    Comment by Bobs — November 15, 2011 @ 9:24 am
  3. Love your tutorials my friend! You rock!

    Are you saying that .flv files can actaully be bundled and run inside an app? Or that they can only run when streamed to an app? Or both?

    Comment by Marsman — January 1, 2012 @ 4:20 pm
  4. Love your tutorials my friend! You rock!

    Are you saying that .flv files can actaully be bundled to run inside an app? Or that they can only run when streamed to an app? Or both?

    Comment by Marsman250 — January 1, 2012 @ 6:17 pm
  5. Is there any mobile working link you could show as a demo? I am trying to achieve something that sounds similar. I have a video with transparent background that displays on my website now using FLV and SWF. since flash can’t be displayed on Iphone in safari, video doesnt show. I too looked at png sequence …sync audio …no luck as yet.

    Is what you are suggesting a solution? I would love to take a look at your example through my iphone.

    Please advise.

    Comment by Phillip — January 4, 2012 @ 7:47 pm
  6. this can be achieved via FFmpeg, no flash required… see this post:

    http://www.codza.com/extracting-frames-from-movies-on-iphone

    your technique works because flash does all video decoding and rendering in the AVM, then passed to iOS for display. unfortunately that’s also why as3 for iOS doesn’t have native performance. get a little, give a little ;)

    Comment by starpause — January 10, 2012 @ 8:29 pm
  7. back again, just to comment on the performance of this technique. i’m getting about 4fps on an iPhone4G.

    Comment by starpause — January 12, 2012 @ 12:19 am
  8. Great but, where’s the proof?!

    Comment by Luke Frohling — January 23, 2012 @ 5:46 am
  9. There is a much easier way to implement animations with an alpha channel, have a look at the “APNG” app on the app store for a demo and links to more info.

    http://itunes.apple.com/us/app/apng/id447928057

    Comment by Mo — July 29, 2012 @ 10:29 pm

RSS feed for comments on this post. TrackBack URI

Leave a comment

This work is licensed under GPL - 2009 | Powered by Wordpress using the theme aav1