As Flash Platform devs, the easiest way to get access to the XMP data in a video file created by Adobe Premiere is to make sure you handle the onXMPData callback that NetStream fires (Flash Player 10 only). One way to do this is by extending NetStream and adding a callback handler method that looks like this:

// callback handler inside a NetStream subclass dispatches a custom event that  
// forwards the xmp data found in the callback handler
public function onXMPData( info:Object ):void
{
	dispatchEvent( new XMPEvent( XMPEvent.XMP_STRING, info.data, true ) ); 
}

Next you need to make sure you’re listening for the XMPEvent, then load the MPEG-4 AVC / F4V / FLV media:

// pseudo code
var video:NetStream = new NetStream( connection );
video.addEventListener( XMPEvent.XMP_STRING, xmpHandler, false, 0, true );
video.play( "h264_captions.f4v" );

Now its time to see what XMP looks like. We’ll use the AS3 XMP library that Adobe has up on labs:

private function xmpHandler( e:XMPEvent ):void
{
	// XMPMeta is root data structure for xmp data. Pass in the xmp data
	// we received from the NetStream callback
	var xmp:XMPMeta = new XMPMeta( e.xmp );
 
	 var xml:XML = xmp.serializeToXML();
	 trace( xml.toXMLString() );
}

The output of the previous trace should look something like this (full example xml available here):

As you can see in the example above the much of the XMP information is stored in the Adobe Dynamic Media schema that uses the xmpDM namespace.

XMP itself builds on top of another metadata framework called RDF (Rich Description Framework). You can think of RDF as the raw building blocks or data structures necessary to generically describe data.

The gigantic mess of RDF and XMP namespace elements that need to be mixed and matched can be overwhelming, but keep yer cool–Adobe’s AS3 XMP library makes it relatively simple to parse, modify or create XMP metadata.

First off let’s come to a basic understanding of the core RDF elements used in XMP:

• Description – the basic data structure of RDF, this sucka holds just about anything.
• Bag – an array structure. I just think of these as unordered all purposes arrays.
• Seq – an array structure that’s ordinal by nature.
• li – an item. If yer a Bag or Seq and aren’t just an empty vessel then plenty of these will be present.

Luckily, Adobe’s AS3 XMP library has a similar ActionScript object data model:

• XMPStruct – the core data structure of XMP.
• XMPArray – there are bag and seq flavors, but for the most part these are implicitly derived.

Let’s take a look at the speech-to-text data again and see if we can piece together some of the details.

Now lets see how we could use this information to parse the XMP data and extract values:

private function xmpHandler( e:XMPEvent ):void
{
    var xmp:XMPMeta = new XMPMeta( e.xmp );
    // define a namespace that will allow us to "dot down" into the data
    // the XMP lib provides constants for many of the common XMP schemas
    var xmpDM:Namespace = XMPConst.xmpDM; 
 
    // "Tracks" is a Bag array containing XMPStruct types.
    // Let's take a look at how we can parse out the values in
    // first XMPStruct. NOTE: All XMP arrays start at 1 rather than 0.
    var track1:XMPStruct = xmp.xmpDM::Tracks[1];
    trace( track1.xmpDM::trackName );
    trace( track1.xmpDM::frameRate );
 
    // the second struct contains the array of speech-to-text
    // markers that we're interested in.
    var track2:XMPStruct = xmp.xmpDM::Tracks[2];
    var markers:XMPArray = track2.xmpDM::markers;
    var item:XMPStruct;
 
    for ( var i:int = 1; i < markers.length + 1; i++ )
    {
        item = markers[i]; //  a reference to the current marker of type XMPStruct
 
        // now we can simply look at all of the property values for each 
        // item in the array
        trace( "startTime: " + item.xmpDM::startTime );
        trace( "duration: " + item.xmpDM::duration );
        trace( "name: " + item.xmpDM::name );
        trace( "speaker: " + item.xmpDM::speaker );
        trace( "probability: " + item.xmpDM::probability + "\n");
    }
}

Ok, so that wasn’t too bad was it? Armed with some basic RDF knowledge, the Adobe AS3 XMP lib actually makes it pretty easy to parse out any XMP metadata you’re likely to run into. But we can do better than that, right? Let’s use the XMP lib and our reverse engineered knowledge of the Adobe XMP Dynamic Media schema to create some XMP marker data from scratch.

private function createXMPMarkers():void
{
    // create an xmp object and create a namespace for the Adobe DynamicMedia schema
    var xmp:XMPMeta = new XMPMeta();
    var xmpDM:Namespace = XMPConst.xmpDM;
 
    // create some markers and set some properties
    var item1:XMPStruct = new XMPStruct();
    item1.xmpDM::name = "XMP Standard Schemas";
    item1.xmpDM::startTime = 0;
    item1.xmpDM::duration = 15000;
    item1.xmpDM::type = "Chapter";
 
    var item2:XMPStruct = new XMPStruct();
    item2.xmpDM::name = "Dublin Core schema";
    item2.xmpDM::startTime = 0;
    item2.xmpDM::duration = 9999;
    item2.xmpDM::type = "Index";
 
    var item3:XMPStruct = new XMPStruct();
    item3.xmpDM::name = "XMP Dynamic Media schema";
    item3.xmpDM::startTime = 10000;
    item3.xmpDM::duration = 5000;
    item3.xmpDM::type = "Index";
 
    var item4:XMPStruct = new XMPStruct();
    item4.xmpDM::name = "Specialized Schema";
    item4.xmpDM::startTime = 15001;
    item4.xmpDM::duration = 50000;
    item4.xmpDM::type = "Chapter";
 
    // create an sequenced array to hold all of the markers
    var markers:XMPArray = XMPArray.newSeq();
    markers.append( item1 );
    markers.append( item2 );
    markers.append( item3 );
    markers.append( item4 );
 
    //Create Tracks Struct
    var tracks:XMPStruct = new XMPStruct();
    tracks.xmpDM::trackName = "TOC Markers";
    tracks.xmpDM::markers = markers;
 
    // Add the "Tracks" property to the xmp object and set its value equal to the track variable.
    xmp.xmpDM::Tracks[1] = tracks;
 
    var xml:XML = xmp.serializeToXML();
    trace( xml.toXMLString() );
}

That’s it for now kids – y’all should be XMP parsing and generating machines by now. Look for an upcoming tutorial on creating and using custom XMP schemas / namespaces to store application specific metadata in your media files.

Go fullscreen to see the video in full 1:1 pixel clarity.

Sometimes you’re an idiot and you spend a lot of time reinventing the wheel because you misread one little line of documentation. This is one of those times where I get to be that idiot. I invented a solution for bilinear resampling in Flash Player when I didn’t need to. You can actually get the same results pretty simply using just BitmapData.draw() method with smoothing on (I had read this didn’t work when down sizing), but it requires creating a temp BitmapData object if the source you’re feeding the BitmapData.draw() method is not another BitmapData object (i.e. a DisplayObject). Here’s what that scenario would look like.

            // source in this example is a DisplayObject
            var temp:BitmapData = new BitmapData( sourceWidth, sourceHeight );
            temp.draw( source );
 
            var output:BitmapData = new BitmapData( outputWidth, outputHeight );
 
            var matrix:Matrix = new Matrix();
            matrix.scale( outputWidth / sourceWidth, outputHeight / sourceHeight );
 
            output.draw( temp, matrix, null, null, null, true );
            temp.dispose();

Many thanks to Felix for leaving a comment that made me question myself and go back to the problem with fresh eyes.

Now I can take some consolation in the fact that I actually learned a lot by going down this path. I dug in to Pixel Bender again and learned how to use ShaderJob. To that end, portions of this “tutorial” illustrate how to use a PixelBender shader and a ShaderJob together. ShaderJob, when run with the async flag (false), is as close as you can come to creating new threads in Flash–that’s some pretty neat Flash hackery.

So, if you find yourself reading this, you’re advised to skip down to the code included below and take a look at the embedded filter, Shader / ShaderJob portion.

*End of Update*

If you’re looking to generate high quality scaled bitmaps in Flash Player / AIR, you’ll need to do some form of bilinear or bicubic resampling. Why you ask? Because, Flash Player resampling is, by default, nearest neighbor and that looks, well, like ass (see evidence directly below).

If you dig around long enough on the net you’ll find some AS3 libraries (Java ports) like clevrlib which will do bilinear or bicubic resampling using BitmapData methods like getPixel / setPixel. As seen below, clevrlib definitely decreases jaggies and improves the legibility of text, but its extremely slow–scaling a 960×540 BitmapData object to 640×360 takes around 1.5 – 2 seconds for bilinear resampling and around 3 – 5 seconds for bicubic resampling (mileage will vary based on the type of imagery and size of the picture). Also, the results, in some cases, exhibit noticeable resampling artifacts (check out the fullscreen icon in the right corner of the examples immediately below).

I was looking to implement bilinear resampling in ThumbGenie, but, as the old saying goes–speed kills. I just couldn’t tolerate 2 or 3 seconds to resample an image–there had to to be a better way. After a bit of thought I fell on the answer–PixelBender.

Flash Player 10 PixelBender bytecode is designed to manipulate individual pixels in a bitmap super fast. In fact, it executes fast enough to apply real time filter effects to video. This seemed promising and sure enough after reading through the sdk docs and playing around for awhile (read I spent a Saturday evening trying to wrap my little melon around the issue), I was able to find a built-in PixelBender method that does bilinear sampling. Slap a parameter on that biatch (aka kernel) controlling the desired scale of the output and little ol’ me was rendering silky-smooth, scaled bitmaps out of Flash Player in mere milliseconds (8-13 milliseconds scaling a 960×540 bitmap to 640×360–take that suckaz).

Interested in chasing the same dream? Here’s, the rough outline you’ll need to follow:

1. Create Pixel Bender kernel that does bilinear resampling.
2. Compile Pixel Bender bytecode for Flash Player 10 (.pbj file).
3. Embed PBJ bytecode in AS3 class.
4. Generate BitmapData from a DisplayObject (BitmapData.draw( obj )).
5. Create an AS3 Shader object, instantiate embedded Pixel Bender bytecode, configure Shader.
6. Create an output BitmapData object (this will hold the scaled bitmap we create.
7. Create a ShaderJob – this is a process that will run the Shader and return its output to an object (as opposed to being applied to a DisplayObject wrapped in a ShaderFilter).
8. Run the ShaderJob by calling the ShaderJob.start( true ) method.
9. Do something with the BitmapData thumbnail (display, push to server, or encode and save to disk)

Ouch, that’s pretty painful. And believe me, I did some serious bleeding before figuring this one out–I really had to push the two or three sleep deprived brain cells I have left in order to get the job done. And well, this is just a long-winded way of saying that I’ll be providing y’all the Pixel Bender kernel (.pbk) and bytcode (.pbj) plus the AS3 implementation that I developed for ThumbGenie (just my way of saying thanks for all of the code you Flash / Flex cats so liberally share all of time).

Whew, you made it this far, now its time for the good stuff. Download the PixelBender source / bytecode here and feel free to peruse / use the implementation seen below.

/**
 * Copyright 2009 (c) , Brooks Andrus
 * 
 * Permission is hereby granted, free of charge, to any person
 * obtaining a copy of this software to deal in the Software without
 * restriction, including without limitation the rights to use,
 * copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the
 * Software is furnished to do so, subject to the following
 * conditions:
 *
 * The above copyright notice and this permission notice shall be
 * included in all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 * OTHER DEALINGS IN THE SOFTWARE.
 * 
 */
package com.brooksandrus.utils
{
    import flash.display.BitmapData;
    import flash.display.DisplayObject;
    import flash.display.Shader;
    import flash.display.ShaderJob;
 
    public class BilinearResample
    {
        [Embed ( source="../../../../assets/bilinearresample.pbj", mimeType="application/octet-stream" ) ]
        private var BilinearScaling:Class;
 
        public function BilinearResample()
        {
            super();
        }
 
        public function resampleBitmap( input:BitmapData, desiredWidth:int, desiredHeight:int, cleanup:Boolean = true ):BitmapData
        {
            var aspectRatio:Number = input.width / input.height;
 
            var factor:Number = Math.max( input.width / desiredWidth, input.height / desiredHeight );
 
            // create and configure a Shader object
            var shader:Shader = new Shader();
            shader.byteCode = new BilinearScaling(); // instantiate embedded Pixel Bender bytecode
            shader.data.src.input = input; // supply the shader with BitmapData it will manipulate
            shader.data.scale.value = [factor]; // scale factor. shader params are all stored in arrays.
 
            var outputWidth:int;
            var outputHeight:int;
 
            // determine output bitmap dimensions
            if ( input.width > input.height )
            {
                outputWidth = desiredWidth;
                outputHeight = desiredWidth / aspectRatio;
            }
            else
            {
                outputWidth = desiredHeight * aspectRatio;
                outputHeight = desiredHeight;
            }
 
            // create a bitmap - our shader will return its data (an image) to this bitmap
            var output:BitmapData = new BitmapData( outputWidth, outputHeight );
 
            // shader jobs are wicked cool
            var job:ShaderJob = new ShaderJob();
            job.target = output; // ShaderJob returns to this object
            job.shader = shader; // The Shader assigned to this job
            job.start( true ); // true flag runs the job synchronously.
 
            if ( cleanup )
            {
                input.dispose();
            }
 
            return output;
        }
 
        public function resampleDisplayObject( source:DisplayObject, desiredWidth:int, desiredHeight:int ):BitmapData
        {
            var aspectRatio:Number = source.width / source.height;
 
            var factor:Number = Math.max( source.width / desiredWidth, source.height / desiredHeight );
 
            var input:BitmapData = new BitmapData( source.width, source.height, true );
            input.draw( source );
 
            // configure the shader
            var shader:Shader = new Shader();
            shader.byteCode = new BilinearScaling();
            shader.data.src.input = input;
            shader.data.scale.value = [factor]; // scale factor. shader params are all stored in arrays.
 
            var outputWidth:int;
            var outputHeight:int;
 
            if ( input.width > input.height )
            {
                outputWidth = desiredWidth;
                outputHeight = desiredWidth / aspectRatio;
            }
            else
            {
                outputWidth = desiredHeight * aspectRatio;
                outputHeight = desiredHeight;
            }
 
            var output:BitmapData = new BitmapData( outputWidth, outputHeight );
 
            var job:ShaderJob = new ShaderJob();
            job.target = output;
            job.shader = shader;
            job.start( true ); // true flag runs the job synchronously.
 
            input.dispose();
 
            return output;
        }
    }
}

The simple answer of course is to point him to the “script” tag (let’s just forget code behind support was ever added to Flex). Rather than typing up a long winded response I made a quick and dirty Jing video to “show him around” Flex’s script tag (no deep explanation, I’m just teaching by “doing”).
Go fullscreen to see the video in full 1:1 pixel clarity.

Instructions: Press the “play” button, then select the effect you’d like to see applied to the video.

The ten thousand dollar question is what’s the value outside of gratuitous eye candy? It seems that in some cases you might want to apply a particular stylistic treatment (wouldn’t it be great if the cartoon effect that ships with After Effects would compile for Flash Player). There’s also the opportunity to animate the filters and create some really killer transition effects–anyone with a media player component or video editing / mashup RIA ought to see the potential there. And that’s just the tip of the iceberg, I’m sure there are far grander ideas being born as I write this.

FYI – All of the filters were gathered up out on the interweb (mesh, exchange, pixelero).

Yep, its old news, but its a great example of the type of quick and simple knowledge nugget you can share with Jing and I’ve got a new years resolution to do that a bit more often.

F4V files (otherwise known as MPEG4-AVC / h.264 + a special Flash only file extension) DO NOT SUPPORT CUE POINTS. Unfortunately, no good tooling support for timed text tracks replaces them.

Why F4V is the wrong decision:

• Destroys one of the great promises of MPEG4-AVC / h.264–interoperability.
• No single file deployment (closely tied to interoperability). I certainly imagine a world where a single piece of media can be posted on the web, downloaded and played back through Flash Player, iPhone / iPod, QuickTime, Apple TV, etc. Gone should be the days of providing different files for different browsers, plugins or mobile devices.
• Effectively reduces the usefulness the existing MPEG4-AVC / h.264 ecosystem. The MPEG4 spec urges .mp4 be used as the file extension and many vendors just refuse to work with something that has a different file extension.
• You lose out on much of the existing intelligent rss enclosure handling. Blogging platforms and plugins will recognize the .mp4 extension and auto-generate enclosures (very useful for delivering to feed readers and mobile devices).
• F4V just muddles the codec picture even more. Take a look at any of the encoder dialogs in CS4 and you’ll see a confusing slew of options (flv, f4v, h.264, h.264 blu-ray, etc.). Hell, I even heard an instructor in one of the MAX hands on sessions, urge students to steer clear of h.264 encodings since that was just HD / blu-ray stuff. He actually was pushing flv (pretty bizarre for an AfterEffects class).

I’m not trying to be a jerk, there are definite reasons Adobe may have chosen to use F4V:

• Apple did it (M4V). Yep, Apple screwed the pooch as well.
• It’s easy to associate file extensions with default application handlers (i.e. AMP is the default media player for *.flv and *.f4v files).
• Allows for a unique mime-type (Flash Player gets associated with a specific file type on servers).
• OS file choosers allow filtering by extension.
• F4V clearly establishes a file as compatible with the parts of the MPEG4 spec supported by Flash Player. This has the obvious advantage of visual associations / assumptions and might assist descriptions in documentation and marketing.

I guess I feel like there should be some sort of primary directive: “thou shalt not damage interoperability.” Any time you’re thinking of messing with the spec it should be examined through this lens. As valid as some of the reasoning for using F4V is, it fails, IMHO, when compared to the primary directive.

I was blown away by the morning session on the Flash Platform’s new text engine. It seems the InDesign team has been hard at work on an AS3 framework called TLF that will be released to labs this Friday. TLF provides enormous framework agnostic (works with Flex or Flash) layout capabilities built on top of Flash Player’s low-level APIs. Very cool stuff that definitely builds on Flash Player’s legend as a cutting edge experience-delivery runtime.

It’s areas like this where you really see the enormous payoff from the Adobe / Macromedia merger. Not many people in the world have the deep typographic and layout knowledge necessary to fully utilize the low level text API’s exposed in Flash Player 10. Adobe, however, has deep roots in type with everything from print drivers, to document formats designed for consistent cross-platform type rendering, to best-of-breed layout tooling. There’s scary potential for the platform when you consider they’ve already delivered Pixel Bender shader effects and are sitting on top of serious video tooling / expertise.

Next I sat in a fantastic Pixel Bender lab put on by the AIF crew. These guys ran the best code oriented lab I’ve ever sat in. I wish I could describe code as clearly and efficiently as these cats–I’d be a code super hero, or an even larger pain in the ass to other developers. Regardless, I should note there’s killer tooling built up around Pixel Bender. A shader language with a code hinting IDE, debugging, breakpoints and export of pbj (bytecode Flash Player 10 runs). Slick stuff–the little girlies over at Adobe should be proud.

I also got to sit in another session on XMP with some silly name like, “Use XMP Metadata to Label, Track, and Manage Assets within Creative Suite.” I’m hoping it was the designerish name that scared everyone off–less than 15 people were in the session. That’s seriously disappointing my fellow meta-nerds. Regardless, Gunar (a Sr. PM with Adobe) gave a super articulate overview of what XMP is, it’s alignment with various industry standards and how it provides tangible benefits throughout the create, edit, publish, deploy and consumption lifecycle. Good stuff that should be mandatory viewing for the lot of you (if only so you can mock my meta fetish).

That’s a wrap for MAX day three. If I get a chance I’ll type up some additional thoughts on the long plane ride home.

Ely Greenfield – As the Oracle might say, “he’s got the gift.” I’d love some time in an osmosis chamber with this cat–it’d be guaranteed to take some of the dumb off.

Ryan Taylor. Had a fantastic session on Pixel Bender. Spoke articulately in code (difficult). I’m no expert now, but he broke it down and made it approachable and desirable. I’d go back for more. Check out his blog.

Maria Shriver – keynote was ho hum. Huge production value, but the overall story seemed, well, less than ground breaking. Shriver stole the show–charismatic and raised to lead (I’m an anti-class American, but she was in a league of her own). Wave was undeserving of the spotlight and CoCoMo feels like something that didn’t sell but has been repackaged as a PaaS.

“Adobe devs who get the most applause get an iPod. Loser gets shipped off to Microsoft.”

Good clean fun. ;-)

Good to know. The real question is WTF Google? Why don’t we know what exactly all the requirements are? For instance, I work for a company that would be very interested in knowing how to achieve pass through upload (we encode video to YouTube’s specs and then upload and thereby skip the YouTube encoding step–saving time and YouTube CPU). If you, YouTube, insist on being the 800 lbs. gorilla, its high time you were WAY MORE TRANSPARENT. I want a legit service that’s not so opaque and has real features instead of hacks posted on blogs. Just my two cents. ;-)

Hmph. An onXMPData callback–that piqued my interest. I’ve long been a fan of metadata and XMP in particular (it takes a special type of nerd to have the meta love), so when I saw this error the wheels began to turn immediately. I had been testing a lot of video files lately, but this particular piece of content was created with After Effects CS4. This made me think that the long and slow roll out of XMP into the Adobe suite was finally here and I wanted to know what exactly they were doing.

Fist things first, I fired up Flex, set a breakpoint and loaded the After Effects created video file into my MPEG4-AVC player. The short video below describes what I found inside the onXMPData callback.

But wait, there’s more, using XMP extends beyond standard file info and saving a video file’s editing history. Case in point, Premiere Pro CS4 has a killer “speech to text” transcription feature. The transcription text is stored as XMP within the video file. This allows Premiere and other tools to accurately search for this data within a file (powers the search feature inside of Premiere and Bridge). As a special bonus, Flash Player 10 provides the same data in its onXMPData callback. Check the short video below to see what this XMP data looks like inside of Flash Player.

The implications of XMP were big enough, but the inclusion of the onXMPData callback in Flash Player 10 makes this a monster if you develop for the platform. Imagine how much more information there is at our fingertips; the ability to tap into the data, connect context, establish relationships and, ultimately, to create lean-forward experiences. It leaves, this lil’ ol’ geek giddy as a schoolgirl.

Now if only I can wrangle a few devs at work into writing XMP into the video files that TechSmith tooling creates. I’m talking cursor info, open application info, html links, captioning, callouts, quizing, the works. I’d literally be swimming in data and that my friends would make me one dangerous hombre (at least in my own mind).

Finally, content creation vendors should absolutely be aware that XMP is an open and extensible standard Adobe is attempting to promote in the industry. The spec is published and they even provide an XMP SDK for C++ and Java which allows you to read and inject XMP metadata into a variety of media file formats including JPEG, PSD, TIFF, AVI, WAV, MPEG, MP3, MOV, INDD, PS, EPS and PNG. By the way Adobe, get cracking on providing an AS3 XMP library that we can use in AIR apps. Growl is nice, but a library for reading and injecting XMP would bring down the house for meta-nerds (this is me grovelling).

P.S. In an effort to stuff an insane amount of video into a single post, or just in case you’re interested in learning how Premiere’s speech to text capabilities work check out the Adobe TV screencast below.