Getting Down and Dirty with XMP and AS3

One of the best ways to get started using XMP is to take a look at how Adobe is using it to embed speech-to-text data inside video files using Adobe Premiere.

As Flash Platform devs, the easiest way to get access to the XMP data in a video file created by Adobe Premiere is to make sure you handle the onXMPData callback that NetStream fires (Flash Player 10 only). One way to do this is by extending NetStream and adding a callback handler method that looks like this:

// callback handler inside a NetStream subclass dispatches a custom event that  
// forwards the xmp data found in the callback handler
public function onXMPData( info:Object ):void
{
	dispatchEvent( new XMPEvent( XMPEvent.XMP_STRING, info.data, true ) ); 
}

Next you need to make sure you’re listening for the XMPEvent, then load the MPEG-4 AVC / F4V / FLV media:

// pseudo code
var video:NetStream = new NetStream( connection );
video.addEventListener( XMPEvent.XMP_STRING, xmpHandler, false, 0, true );
video.play( "h264_captions.f4v" );

Now its time to see what XMP looks like. We’ll use the AS3 XMP library that Adobe has up on labs:

private function xmpHandler( e:XMPEvent ):void
{
	// XMPMeta is root data structure for xmp data. Pass in the xmp data
	// we received from the NetStream callback
	var xmp:XMPMeta = new XMPMeta( e.xmp );
 
	 var xml:XML = xmp.serializeToXML();
	 trace( xml.toXMLString() );
}

The output of the previous trace should look something like this (full example xml available here):

As you can see in the example above the much of the XMP information is stored in the Adobe Dynamic Media schema that uses the xmpDM namespace.

XMP itself builds on top of another metadata framework called RDF (Rich Description Framework). You can think of RDF as the raw building blocks or data structures necessary to generically describe data.

The gigantic mess of RDF and XMP namespace elements that need to be mixed and matched can be overwhelming, but keep yer cool–Adobe’s AS3 XMP library makes it relatively simple to parse, modify or create XMP metadata.

First off let’s come to a basic understanding of the core RDF elements used in XMP:

• Description – the basic data structure of RDF, this sucka holds just about anything.
• Bag – an array structure. I just think of these as unordered all purposes arrays.
• Seq – an array structure that’s ordinal by nature.
• li – an item. If yer a Bag or Seq and aren’t just an empty vessel then plenty of these will be present.

Luckily, Adobe’s AS3 XMP library has a similar ActionScript object data model:

• XMPStruct – the core data structure of XMP.
• XMPArray – there are bag and seq flavors, but for the most part these are implicitly derived.

Let’s take a look at the speech-to-text data again and see if we can piece together some of the details.

Now lets see how we could use this information to parse the XMP data and extract values:

private function xmpHandler( e:XMPEvent ):void
{
    var xmp:XMPMeta = new XMPMeta( e.xmp );
    // define a namespace that will allow us to "dot down" into the data
    // the XMP lib provides constants for many of the common XMP schemas
    var xmpDM:Namespace = XMPConst.xmpDM; 
 
    // "Tracks" is a Bag array containing XMPStruct types.
    // Let's take a look at how we can parse out the values in
    // first XMPStruct. NOTE: All XMP arrays start at 1 rather than 0.
    var track1:XMPStruct = xmp.xmpDM::Tracks[1];
    trace( track1.xmpDM::trackName );
    trace( track1.xmpDM::frameRate );
 
    // the second struct contains the array of speech-to-text
    // markers that we're interested in.
    var track2:XMPStruct = xmp.xmpDM::Tracks[2];
    var markers:XMPArray = track2.xmpDM::markers;
    var item:XMPStruct;
 
    for ( var i:int = 1; i < markers.length + 1; i++ )
    {
        item = markers[i]; //  a reference to the current marker of type XMPStruct
 
        // now we can simply look at all of the property values for each 
        // item in the array
        trace( "startTime: " + item.xmpDM::startTime );
        trace( "duration: " + item.xmpDM::duration );
        trace( "name: " + item.xmpDM::name );
        trace( "speaker: " + item.xmpDM::speaker );
        trace( "probability: " + item.xmpDM::probability + "\n");
    }
}

Ok, so that wasn’t too bad was it? Armed with some basic RDF knowledge, the Adobe AS3 XMP lib actually makes it pretty easy to parse out any XMP metadata you’re likely to run into. But we can do better than that, right? Let’s use the XMP lib and our reverse engineered knowledge of the Adobe XMP Dynamic Media schema to create some XMP marker data from scratch.

private function createXMPMarkers():void
{
    // create an xmp object and create a namespace for the Adobe DynamicMedia schema
    var xmp:XMPMeta = new XMPMeta();
    var xmpDM:Namespace = XMPConst.xmpDM;
 
    // create some markers and set some properties
    var item1:XMPStruct = new XMPStruct();
    item1.xmpDM::name = "XMP Standard Schemas";
    item1.xmpDM::startTime = 0;
    item1.xmpDM::duration = 15000;
    item1.xmpDM::type = "Chapter";
 
    var item2:XMPStruct = new XMPStruct();
    item2.xmpDM::name = "Dublin Core schema";
    item2.xmpDM::startTime = 0;
    item2.xmpDM::duration = 9999;
    item2.xmpDM::type = "Index";
 
    var item3:XMPStruct = new XMPStruct();
    item3.xmpDM::name = "XMP Dynamic Media schema";
    item3.xmpDM::startTime = 10000;
    item3.xmpDM::duration = 5000;
    item3.xmpDM::type = "Index";
 
    var item4:XMPStruct = new XMPStruct();
    item4.xmpDM::name = "Specialized Schema";
    item4.xmpDM::startTime = 15001;
    item4.xmpDM::duration = 50000;
    item4.xmpDM::type = "Chapter";
 
    // create an sequenced array to hold all of the markers
    var markers:XMPArray = XMPArray.newSeq();
    markers.append( item1 );
    markers.append( item2 );
    markers.append( item3 );
    markers.append( item4 );
 
    //Create Tracks Struct
    var tracks:XMPStruct = new XMPStruct();
    tracks.xmpDM::trackName = "TOC Markers";
    tracks.xmpDM::markers = markers;
 
    // Add the "Tracks" property to the xmp object and set its value equal to the track variable.
    xmp.xmpDM::Tracks[1] = tracks;
 
    var xml:XML = xmp.serializeToXML();
    trace( xml.toXMLString() );
}

That’s it for now kids – y’all should be XMP parsing and generating machines by now. Look for an upcoming tutorial on creating and using custom XMP schemas / namespaces to store application specific metadata in your media files.

Flex Skins with Illustrator CS4

Looking for a quick intro to building Flex components with Illustrator? Check out this short video.

Go fullscreen to see the video in full 1:1 pixel clarity.

Bilinear Resampling, ShaderJob, Pixel Bender And Flash Player

*Update*

Sometimes you’re an idiot and you spend a lot of time reinventing the wheel because you misread one little line of documentation. This is one of those times where I get to be that idiot. I invented a solution for bilinear resampling in Flash Player when I didn’t need to. You can actually get the same results pretty simply using just BitmapData.draw() method with smoothing on (I had read this didn’t work when down sizing), but it requires creating a temp BitmapData object if the source you’re feeding the BitmapData.draw() method is not another BitmapData object (i.e. a DisplayObject). Here’s what that scenario would look like.

            // source in this example is a DisplayObject
            var temp:BitmapData = new BitmapData( sourceWidth, sourceHeight );
            temp.draw( source );
 
            var output:BitmapData = new BitmapData( outputWidth, outputHeight );
 
            var matrix:Matrix = new Matrix();
            matrix.scale( outputWidth / sourceWidth, outputHeight / sourceHeight );
 
            output.draw( temp, matrix, null, null, null, true );
            temp.dispose();

(more…)

Flex Basics – The Script Tag

Grasping the simple things is often the hardest part to getting up and running with Flex. For example, a friend of mine who I’ve been trying to pull over from the dark side (.Net guy) asked, “I’ve got an MXML Application with a button in it that I’d like to bind to a method–where’s the code behind.”

The simple answer of course is to point him to the “script” tag (let’s just forget code behind support was ever added to Flex). Rather than typing up a long winded response I made a quick and dirty Jing video to “show him around” Flex’s script tag (no deep explanation, I’m just teaching by “doing”).
(more…)

Pixel Bender + Video = Killer Runtime Effects

I’ve been playing with Pixel Bender a bit and noticed that there weren’t really any examples (I could find) that illustrated effects being applied to video. So I gathered together a whole slew of kernels from the interweb, downloaded a copy of Dancing Matt and shoehorned them into a media player that allows you to select and apply the effects during video playback. (more…)