| Comments

Here’s how it started…

Lisa (my wife) [shouting from office into the kitchen]: Tim, what’s this Amazon charge for $193?
Me [thinking what I may have purchased and not remembered]: Um, don’t know…let me look.

I then logged into my Amazon account to see what order I may have forgotten.  Surely I didn’t order $200 worth of MP3…that’s ridiculous.  Sure enough nothing was there.  Immediately I’m thinking fraud.  I start freaking out, getting mad, figuring out my revenge scheme on the scammer, etc.

Then it hit me: Amazon Web Services account.

The Culprit

Sure enough I logged in and my January 2010 billing account was $193 and change.  Yikes.  Well, I could let the (what has been averaging) $30 or so charge slide under the family CFO radar for a while…but this $193 charge…the chief auditor herself caught that one.

So I panicked.  I needed to figure out where/what the spike was.  I logged into the Amazon Web Services management console (I only use the S3/CloudFront storage in their services right no) to see what was going on.  I see ‘Usage Reports’ and click.  I’m met with essentially a bunch of useless data really.  No offense to Amazon, but really the usage reports weren’t really helpful at all.  First, they gave me a Resource ID which I thought would represent the URI I was looking for.  Nope, Resource ID == Bucket.  And they didn’t even put the bucket name in the report!

For some perspective, here’s essentially what I’m used to – here’s my December 2009 billing statement details:

December 2009 S3 CloudFront Billing

Anyhow, after some hunting it was obvious that I wasn’t going to figure out what bucket objects/unique URIs were causing my spike.  This was primarily because I didn’t have logging turned on at all on my buckets.  I had in the past but really didn’t think I needed it so I turned it off.

I was wrong – go now and enable logging.

While I was searching for a solution to understand my traffic, I was curious for where my traffic was.  Like I said, I’d been averaging (actually *peaking*) at about a $30 charge for the S3 hosting.

NOTE: I use S3 for all my image/screenshot/sample code file hosting.  I’ve invested in S3 for a long time and built my blogging workflow around it with building tools like S3 Browser for Windows Live Writer.

What was interesting was my most usage of my CloudFront data was coming from Hong Kong.  Compare to above the December 2009 billing to this January 2010 billing:

January 2010 Blling Statement

Yeah, that was my reaction too.  I went from roughly 40GB of transfer bandwidth to over 960GB in one month.  I suspected I knew what happened, but needed to confirm before I changed things. 

Implementing Logging for Statistics

The problem was that I didn’t have logging enabled and I was pretty much stuck.  I needed to get some data from the logs before being for sure.  I quickly found S3Stat and it appears to be the de-facto reporting for Amazon S3 log files.  I signed up for the free trial and generated a new access key to give them.

NOTE: They have a ‘manual’ option which means a lot more work.  I simply generated a NEW S3 access key for this specific purpose.  That way I didn’t have to give them my golden key I’ve been using in other places and can shut this off at any time without issue to my other workflows.

24 hours later, I had some reports.  Wicked cool reports.  Here’s a list of what I’m currently looking at:

  • Total hits, total files, total kbytes
  • Hits/files per hour/day
  • Hourly stats
  • Top 30 URIs
  • Top URIs by kbytes used
  • Top referrers (find out who’s using your bits without you knowing)
  • User agents
    Here’s a quick snapshot of one:
    S3Stat sample report image

Wow…honestly…THIS is what I was expecting when I see “usage” data reports.  S3Stat is awesome and you should use that now.  Yes, I’m buttering up to them…but they have a great tool here for $5/month if you are a heavy Amazon S3/CloudFront user.  Amazon frankly should just buy them and integrate this into their management console.  You can see other examples of their report outputs on their site at http://www.s3stat.com

What I also found out is that the tool I use for my desktop usage of S3/CloudFront (outside of my blogger workfow and S3Browser) has S3Stat integration built in!  I use CloudBerry’s S3 Explorer Pro for managing my S3 content.  It’s awesome and you should look at it.  When I look at the logging features in CloudBerry I see this:

CloudBerry S3Stat dialog

And after enabling the logging, within CloudBerry I can view the log data within the tool:

CloudBerry view logging


Wow, this is incredibly helpful and insightful data.  I now know who/how/when my cloud storage data is being used in various ways I can see the data.  S3Stat immediately showed me incredible value within less than 24 hours of enabling it.  I know can confirm the culprit of the burst of usage and plan accordingly.

Now, to be clear I’m not complaining about the cost of cloud storage.  That has been clear to me from the beginning.  Nothing is hidden and I’m not an idiot for not understanding it.  What I did not account for was the popularity of some files…and then the ones that just happened to be the largest.  I could not have personally thought I’d see a 920GB spike in one month of usage…but now I know…and have to alter some plans. 

Hopefully this is helpful for some who are just exploring cloud storage solutions/services.  Make sure you have instrumentation and logging capabilities turned on so you can identify and tune your situations.  For me, S3Stat and CloudBerry are winners for my personal usages.  If you are an Amazon S3 customer, I recommend looking at S3Stat and turning on logging immediately!

| Comments

Over the holiday this past week I got a ping from Scott Cate about some Silverlight media questions, namely a player and Silverlight Streaming.  The gist of the conversation was that the Encoder 2 SP1 templates and the SLS Plugin don’t play nice together (yet).  I know that the team has been testing some updates to the plugin for uploading Silverlight 2 templates to SLS, but for now if you tried to do that you’ll get a random error message that won’t make sense to you (something along the lines of template not found).

After talking with Scott and understanding his scenario more (media formats and CDN hosting – note: CDN=Content Delivery Network - specifically) we, well I, decided it might be a good idea to use an alternate player right now.  Of course I suggested the one that Joel and I worked on, but there are also many other out there.  I should also point out that the Encoder 2 SP1 templates also include a fully functional Silverlight 2 player with source code that you can use.  Scott took my player as a start (not sure if it will be the final one he uses) and we went to chatting about the implementation.  I first told him he could still use SLS and just upload the XAP, etc. but remembered that SLS doesn’t fully support an easy implementation of initParams just yet (it can, but a little workaround-ish right now).  After learning of the initParams, Scott wanted to host the player on his CDN and just pass in the media URI as the param (exactly what our player is built for).

I pointed to the instructions on the SL2VideoPlayer CodePlex site and we started implementing.  So Scott was going to have his pages delivered from his CMS system on his site (let’s call it foo.com) and have the Silverlight application delivered from his CDN (let’s call that mycdn.com).  Then the media was going to be hosted on another CDN (Silverlight Streaming).  We had three networks in play here: foo.com, mycdn.com and Silverlight Streaming.  In his site delivered by his CMS system he’d have the object tag for the player implemented something similar to:

   1: <object data="data:application/x-silverlight-2," type="application/x-silverlight-2" width="640" height="480">
   2:         <param name="source" value="http://mycdn.com/myaccount/VideoPlayerM.xap"/>
   3:         <param name="background" value="white" />
   4:         <param name="initParams" value="m=http://silverlight.live.com/accountid/Bear.wmv" />
   5:                <param name="minruntimeversion" value="2.0.31005.0" />
   6:         <a href="http://go.microsoft.com/fwlink/?LinkId=124807" style="text-decoration: none;">
   7:              <img src="http://go.microsoft.com/fwlink/?LinkId=108181" alt="Get Microsoft Silverlight" style="border-style: none"/>
   8:         </a>
   9:     </object>

Note that the source for the Silverlight application is a full URI to the CDN endpoint of the XAP file.  Also note the initParams which would point to the video file hosted on Silverlight Streaming (this isn’t the real URI, just a sample).  When rendered, we saw the Silverlight default loader (%age orb) and then a white screen – nothing.  WTF?!  Loading up my favorite tool for Silverlight network debugging we saw the XAP being delivered and loaded, but then the media file was not.

I then remembered that I may have forgot to tell Scott that any remotely loaded XAP needs to ensure that it is delivered with the correct content-type of application/x-silverlight-app to ensure that it would load properly.  We double-checked this and ensured it was (of course it was because it was already being loaded, but good to double check).  Then we couldn’t figure out why it wasn’t loading the media.

Aha!  I remembered that there was one snippet of code that was in there that does a check for a query string parameter.  In doing this, the Silverlight application uses the HTML DOM bridge to interact with, well, the HTML DOM.  This is what was causing the problem.  By default, remotely loaded XAP files cannot access the HTML DOM without the author explicitly allowing it.  So we had to add another parameter to the object tag, the enableHtmlAccess param.  I sent it to Scott in an email (after testing myself) and thought we were done.  He said it still wasn’t working.  Frustrated, we kept looking.  Sergey from the network team alerted me to a possibility of some character encoding issues.  Yep, that was it.  Here’s a screenshot from the code that wasn’t working:

Spot the issue?  The copied/pasted param quotes were not in ASCII encoding and thus I’m assuming HTML was ignoring it.  After Scott typed it out rather than pasting, it worked fine.

Phew!  Nothing like banging your head in the weeds of code when it is a little typo!


So if you are looking to host your Silverlight content on a CDN for delivery, make sure two things:

    1. The server delivering your XAP content must deliver with the correct MIME type: application/x-silverlight-app
    2. If there is any code in your XAP that does HTML bridge interaction, you’ll have to add the enableHtmlAccess param to either your object tag instantiation as a param, or to your createObject call if using Silverlight.js.

I hope this helps some!