OSMF + Flex Example

I’ve been checking up on OSMF (Open Source Media Framework) periodically, but found that many of their examples / samples don’t work at the moment (their APIs have been in flux). If you’re just looking for a quick way to get OSMF (v.9) up and running in Flex here’s a bit of code that should help.

(more…)

Getting Down and Dirty with XMP and AS3

One of the best ways to get started using XMP is to take a look at how Adobe is using it to embed speech-to-text data inside video files using Adobe Premiere.

As Flash Platform devs, the easiest way to get access to the XMP data in a video file created by Adobe Premiere is to make sure you handle the onXMPData callback that NetStream fires (Flash Player 10 only). One way to do this is by extending NetStream and adding a callback handler method that looks like this:

// callback handler inside a NetStream subclass dispatches a custom event that  
// forwards the xmp data found in the callback handler
public function onXMPData( info:Object ):void
{
	dispatchEvent( new XMPEvent( XMPEvent.XMP_STRING, info.data, true ) ); 
}

Next you need to make sure you’re listening for the XMPEvent, then load the MPEG-4 AVC / F4V / FLV media:

// pseudo code
var video:NetStream = new NetStream( connection );
video.addEventListener( XMPEvent.XMP_STRING, xmpHandler, false, 0, true );
video.play( "h264_captions.f4v" );

Now its time to see what XMP looks like. We’ll use the AS3 XMP library that Adobe has up on labs:

private function xmpHandler( e:XMPEvent ):void
{
	// XMPMeta is root data structure for xmp data. Pass in the xmp data
	// we received from the NetStream callback
	var xmp:XMPMeta = new XMPMeta( e.xmp );
 
	 var xml:XML = xmp.serializeToXML();
	 trace( xml.toXMLString() );
}

The output of the previous trace should look something like this (full example xml available here):

As you can see in the example above the much of the XMP information is stored in the Adobe Dynamic Media schema that uses the xmpDM namespace.

XMP itself builds on top of another metadata framework called RDF (Rich Description Framework). You can think of RDF as the raw building blocks or data structures necessary to generically describe data.

The gigantic mess of RDF and XMP namespace elements that need to be mixed and matched can be overwhelming, but keep yer cool–Adobe’s AS3 XMP library makes it relatively simple to parse, modify or create XMP metadata.

First off let’s come to a basic understanding of the core RDF elements used in XMP:

• Description – the basic data structure of RDF, this sucka holds just about anything.
• Bag – an array structure. I just think of these as unordered all purposes arrays.
• Seq – an array structure that’s ordinal by nature.
• li – an item. If yer a Bag or Seq and aren’t just an empty vessel then plenty of these will be present.

Luckily, Adobe’s AS3 XMP library has a similar ActionScript object data model:

• XMPStruct – the core data structure of XMP.
• XMPArray – there are bag and seq flavors, but for the most part these are implicitly derived.

Let’s take a look at the speech-to-text data again and see if we can piece together some of the details.

Now lets see how we could use this information to parse the XMP data and extract values:

private function xmpHandler( e:XMPEvent ):void
{
    var xmp:XMPMeta = new XMPMeta( e.xmp );
    // define a namespace that will allow us to "dot down" into the data
    // the XMP lib provides constants for many of the common XMP schemas
    var xmpDM:Namespace = XMPConst.xmpDM; 
 
    // "Tracks" is a Bag array containing XMPStruct types.
    // Let's take a look at how we can parse out the values in
    // first XMPStruct. NOTE: All XMP arrays start at 1 rather than 0.
    var track1:XMPStruct = xmp.xmpDM::Tracks[1];
    trace( track1.xmpDM::trackName );
    trace( track1.xmpDM::frameRate );
 
    // the second struct contains the array of speech-to-text
    // markers that we're interested in.
    var track2:XMPStruct = xmp.xmpDM::Tracks[2];
    var markers:XMPArray = track2.xmpDM::markers;
    var item:XMPStruct;
 
    for ( var i:int = 1; i < markers.length + 1; i++ )
    {
        item = markers[i]; //  a reference to the current marker of type XMPStruct
 
        // now we can simply look at all of the property values for each 
        // item in the array
        trace( "startTime: " + item.xmpDM::startTime );
        trace( "duration: " + item.xmpDM::duration );
        trace( "name: " + item.xmpDM::name );
        trace( "speaker: " + item.xmpDM::speaker );
        trace( "probability: " + item.xmpDM::probability + "\n");
    }
}

Ok, so that wasn’t too bad was it? Armed with some basic RDF knowledge, the Adobe AS3 XMP lib actually makes it pretty easy to parse out any XMP metadata you’re likely to run into. But we can do better than that, right? Let’s use the XMP lib and our reverse engineered knowledge of the Adobe XMP Dynamic Media schema to create some XMP marker data from scratch.

private function createXMPMarkers():void
{
    // create an xmp object and create a namespace for the Adobe DynamicMedia schema
    var xmp:XMPMeta = new XMPMeta();
    var xmpDM:Namespace = XMPConst.xmpDM;
 
    // create some markers and set some properties
    var item1:XMPStruct = new XMPStruct();
    item1.xmpDM::name = "XMP Standard Schemas";
    item1.xmpDM::startTime = 0;
    item1.xmpDM::duration = 15000;
    item1.xmpDM::type = "Chapter";
 
    var item2:XMPStruct = new XMPStruct();
    item2.xmpDM::name = "Dublin Core schema";
    item2.xmpDM::startTime = 0;
    item2.xmpDM::duration = 9999;
    item2.xmpDM::type = "Index";
 
    var item3:XMPStruct = new XMPStruct();
    item3.xmpDM::name = "XMP Dynamic Media schema";
    item3.xmpDM::startTime = 10000;
    item3.xmpDM::duration = 5000;
    item3.xmpDM::type = "Index";
 
    var item4:XMPStruct = new XMPStruct();
    item4.xmpDM::name = "Specialized Schema";
    item4.xmpDM::startTime = 15001;
    item4.xmpDM::duration = 50000;
    item4.xmpDM::type = "Chapter";
 
    // create an sequenced array to hold all of the markers
    var markers:XMPArray = XMPArray.newSeq();
    markers.append( item1 );
    markers.append( item2 );
    markers.append( item3 );
    markers.append( item4 );
 
    //Create Tracks Struct
    var tracks:XMPStruct = new XMPStruct();
    tracks.xmpDM::trackName = "TOC Markers";
    tracks.xmpDM::markers = markers;
 
    // Add the "Tracks" property to the xmp object and set its value equal to the track variable.
    xmp.xmpDM::Tracks[1] = tracks;
 
    var xml:XML = xmp.serializeToXML();
    trace( xml.toXMLString() );
}

That’s it for now kids – y’all should be XMP parsing and generating machines by now. Look for an upcoming tutorial on creating and using custom XMP schemas / namespaces to store application specific metadata in your media files.

The Silver Tongue and Lifeless Corpse of Microsoft Silverlight

I work in a predominantly Windows oriented development shop focused around video and there’s been lots of discussion surrounding Microsoft Silverlight and VC-1 over the last year or so. We’ve looked at the technical specs, analyzed the possibilities and engaged in seemingly endless discussions comparing the technology to the Adobe Flash platform. Microsoft has come knocking on our door repeatedly pitching the technology and as a Flash developer I’m surrounded by marketing hype and blogosphere rants, experiments, and outright paranoia at times. I’m always amazed at some of the claims that I see on the web and hear during meetings with their reps–claims which seemingly go unchallenged (maybe it’s just my ignorance that’s holding me back).

One of the oft heard and most annoying claims is that Silverlight will be a ubiquitous runtime quickly. This argument posits that all that it takes to be a ubiquitous runtime is a few killer apps. These killer apps typically are either viral content repositories or social in nature and therefore will quickly reach nearly every web enabled computer on the planet. Microsoft assured us during the Silverlight launch buildup that multiple killer apps would be released with the official 1.0 launch ensuring quick and massive penetration.

This claim that all it takes to become ubiquitous is a couple of killer apps has always struck me as off. Microsoft, and even some folks in the Adobe camp, like to point to YouTube and MySpace as what “made” Flash while conveniently forgetting that Flash Player adoption was at 98% of web enabled computers long before YouTube and MySpace came along.

It’s probably much more accurate to argue that Adobe Flash Player’s high penetration is what enabled the success of MySpace and YouTube. YouTube’s roaring success validated that Flash video had already won on the web despite the fact it supported fewer codecs of lesser quality than did the other embeddable media players.

When Flash MX and Flash Player 6 hit the streets in July of 2002 there were a lot of us who bought into the potential of video when paired with Flash Player’s ubiquity and its strange but wonderful scripting, graphics and animation capabilities. We envisioned a world of custom interfaces embedded seamlessly inside our rich applications and designs. We fought battle after battle just for the opportunity to use the technology. “Back in the day” no one was a believer and video on the web was a bad joke constantly undermined by the big three’s attempts at media player domination.

The point is that 6 years ago we started building the foundation for YouTube and it wasn’t by having a few killer apps, but rather thousands upon thousands of sites with a level of richness and creativity that was unimagined (I’m continually blown away by the creativity and talent that lives and breathes in this community).

Finally, VC-1 seems to be completely dead in the water. It’s my understanding that Microsoft’s strategy of driving adoption from the top down via major media studios and broadcasters has been a major failure. Hell, back in 2002 when Flash Player 6 sported its first video capabilities Windows Media 9 was being hyped like crazy by the softies. Six years later Microsoft has the same codec sporting a new name officially standardized under SMPTE and still can’t find a date to the prom. Instead, industry support has coalesced around h.264. The h.264 tooling and licensing options are available on a variety of different platforms and are an integral part of many media technology stacks.

It’s clear that Apple’s strategy of winning the war by focusing on the content creation tools, dominant portable media playback, cross OS playback / encoding, and media purchasing has soundly beaten Microsoft in the codec / media platform war. Adobe’s decision to include the h.264 codec instead of VC-1 in Flash Player and ditch its proprietary container format should be read as a slap in the face and a clear sign of VC-1′s impotence. In one move they walk away from vendor lock-in strategies and indicate how unappealing Microsoft’s codec offering is.

Microsoft’s track record over the last 10 years lends little comfort. They’ve managed to make a media player that is universally despised and which has progressively gotten worse with each subsequent release. They have a retread codec that’s worn many skirts and names and still can’t get a date. They make a portable media player no one wants and have backed a next generation dvd format that is now officially dead. Let’s see, there’s QuickTime competitor ASF which pales in comparison and has never taken off and it replaced the universally maligned “video for windows” container–AVI. Oh and let’s not forget gems such as Windows Movie Maker and Microsoft Producer. I could go on, but you should be able to connect the dots. I actually feel sorry for the Microsoft employees when I hear them pitch this stuff–it can’t be good for the self-esteem. Note to Microsoft–maybe stop tanking in the media sector and work on making an operating system that people don’t hate.

In the end, perhaps Silverlight will take off, but I doubt it will happen overnight. If it does, it will mean a long, hard, consistent battle in the trenches where winning over the hearts of small shops and individual artists is just as important as pumping up MSDN groupies and stroking ISVs. It will mean building a community of stunning talent and incredible heart (its not a numbers game in my book). It will mean winning over the trust of a massive population trained not to trust new browser installs and increasingly cynical of Microsoft and its aggressive monopolist tendencies as a whole.

If Microsoft is able to overcome all of the hurdles in front of it, Silverlight may become more than a lifeless corpse, but until then it appears they’ll be relying on Jedi mind tricks and their silvery tongue to convince developers and consumers to ignore their repeated missteps and failures in the media sector. Good luck with that MS, you’re going to need it. ;-)