When using media with Silverlight there are a few things that you should be aware of. First, ensure that the media you are planning on using conforms to the VC-1 specification. Your media files just need to then be accessible via streaming or http-based access for progressive downloads.
The media files for progressive downloads can be anywhere: any web server, Amazon S3 storage, some HttpHandler, whatever – as long as they can be served.
Streaming media is supported via Windows Media Services. If you haven’t set this up before, there are a few things to note with regard to Silverlight and streaming content. Most typical default setups of Windows Media streaming uses Real Time Streaming Protocol (RTSP). It’s actually important to note that while RTSP is the visible configuration, it is actually the Real-time Transport Protocol that is responsible for delivering the bits of audio/video. All of this might not matter to you, but cache it away for later if you wish.
For setting up streaming services, I recommend using Windows Media Services 2008, which is a free add-on for Windows Server 2008 systems. After installing this add-on, you’ll have the ability to set up publishing points and host streams of playlists or single files. What I wanted to point out is one of the common gotchas with Silverlight and streaming in the current versions.
After installing the services, if you start the control panel and highlight the server you will see some options. Looking at the Properties tab you will see more options, and specifically I wanted to call attention to the ‘Control Protocol’ option. By default the only enabled option is RTSP. So basically your streams would respond to a mms://foo link for you and open in Windows Media Player or something that plays back that media.
The problem is that a few people have emailed me about why certain streams don’t work, etc. Some are for various resons, but some are because of this problem here. I get a lot of ‘my stream plays in Windows Media Player but not in Silverlight’ questions. Assuming all other factors are correct (see footnote here as well), the problem is likely that you don’t have HTTP-based streaming enabled. Silverlight basically looks at the mms:// URI and translates that to get the stream over HTTP instead as that is the method supported in current versions. You can see this if you put a sniffer on your web application to monitor the requests. To fix this all you have to do is enable the HTTP control protocol on your server. Keep in mind that if you are using a single server for multiple scenarios, the default port 80 might already be taken by IIS. Right click on the HTTP control protocol and choose properties, then choose the IP addresses and/or the port you want it to operate on. Save those settings and then enable the protocol.
By doing this your streaming server will now also stream over HTTP as well as RTSP, thus enabling different scenarios, but specifically allowing you to use Silverlight to host your streaming content. Windows Media Services is a great tool to host streaming content and is pretty fun to mess around with (as well as intuitive to setup new streams) creating new publishing points for media.
Other Streaming Issues
One of the other most common issues I see in my inbox is ‘my stream doesn’t work and yes it is streamed over HTTP.’ The most common problem with this is that your page hosting the Silverlight content is likely being viewed over a file:// protocol scheme and your MediaElement has an http:// scheme. This is known as cross-scheme and is not allowed in Silverlight. Open the page under an HTTP context and it should render fine as long as it is the same protocol.
Have fun playing with this and hope this helps!
The 
