Make your Silverlight applications Speak to you with Microsoft Translator

One of the announcements that happened during the MIX10 conference was the availability of the V2 of the Microsoft Translator API.  This is the engine that powers the translation behind http://www.bing.com/translate and some other Bing-related properties as well.  A lot of research has gone into the engine from Microsoft Research and others.  Language translation isn’t an easy task especially taking into consideration cultural significance of words, etc.  I have heard that the most challenging in machine translation is to Asian languages.  I will admit to not speaking any of them, so I don’t know how well we are performing here – you’ll have to let that team know if they are doing well.

After reading the announcement and working on my translate plugin for the new Seesmic Desktop Platform, I noticed that there was a Speak API.  After reading I saw this literally translates text to a WAV file for platback.  Pretty cool I thought.  I wanted to play around with this in Silverlight so created a simple application to do so:

The Speak translation isn’t available for all the Translator languages (currently 30 languages for text language translation) but does support seven (7) languages: English, German, Spanish, French, Italian, Portuguese, and Russian.  So how is Silverlight talking back to you?

The Translator API comes in 3 flavors: SOAP, HTTP and Ajax.  Now I could have used the SOAP version and used Add Service Reference but I felt for what I was doing this was overkill.  The SOAP API doesn’t return me back super-strongly typed objects, so I saw little value in doing that over the REST-based HTTP methods which I decided to use.  The code is relatively simple.  I first want to translate the text input into the selected language, then pass the translated text to the Speak API and play the results in a MediaElement.

First, we use the Translate method and a WebClient call to accomplish this:

   1: private void TranslateTextToAudio(object sender, RoutedEventArgs e)

   2: {

   3:     if (Languages.SelectedIndex < 0)

   4:     {

   5:         MessageBox.Show("Please select a language first...");

   6:         return;

   7:     }

   8:     WebClient client = new WebClient();

   9:     client.OpenReadCompleted += ((s, args) =>

  10:         {

  11:             if (args.Error == null)

  12:             {

  13:                 DataContractSerializer des = new DataContractSerializer(typeof(string));

  14:                 string responseText = des.ReadObject(args.Result) as string;

  15:                 SpeakIt(responseText);

  16:             }

  17:         });

  18:     client.OpenReadAsync(new Uri(string.Format(TRANSLATE_URI, _appId, HttpUtility.UrlEncode(TextToTranslate.Text), Languages.SelectedValue.ToString(), _currentLang)));

  19: }

Notice the DataContractSerializer use here.  The HTTP API returns serialized objects, so you’ll want to use this method here to deserialize to make it easier.

After we have the translated text, we pass that to the Speak API (notice the call to SpeakIt above) which returns a Stream that is an audio/wav format.   We know that Silverlight’s MediaElement cannot directly play lossless WAV format.  But luckily there is a MediaStreamSource API that enables us to essentially write our own decoders for audio/video.  One of the Silverlight team members, Gilles, created a WAV MediaStreamSource for us to use.  After having that and not worrying about decoding, I could create a new WaveMediaStreamSource with my result Stream and set that as the Source for the MediaElement – here is the resulting SpeakIt() method (TranslatedPlayback is the name of my MediaElement in the application):

   1: private void SpeakIt(string responseText)

   2: {

   3:     WebClient client = new WebClient();

   4:     client.OpenReadCompleted += ((s, args) =>

   5:     {

   6:         if (args.Error == null)

   7:         {

   8:             WaveMediaStreamSource mss = new WaveMediaStreamSource(args.Result);

   9:             TranslatedPlayback.SetSource(mss);

  10:         }

  11:     });

  12:     client.OpenReadAsync(new Uri(string.Format(SPEAK_URI, _appId, responseText, Languages.SelectedValue)));

  13: }

That’s it!  It’s pretty cool (bonus points to you in identifying the obvious pre-filled text being used in my sample).

I’ve almost completed my Seesmic plugin and using the Translator API has made it easier.  I’ve found that a simple wrapper to the HTTP-based methods is going to make things easier for people to use, so I’ve created a Translator Client for Silverlight that I’ll be releasing once I can complete the plugin (waiting on a few things).  This will make it easier for Silverlight developers to quickly consume the API for text and Speech translation.

The code (C#) for the above sample is here: MSTranslatorSilverlightSample.zip.  You will need a Translator Application ID key to run it yourself (put it in the App.xaml resources).  If you just want to see it running really quick, you can view it here: Speak Translator for Silverlight.  You will need Silverlight 4 RC to run the sample.

NOTE: The Translator API actually doesn’t need Silverlight 4, but I just created the app using that base.

Hope this helps!