×

First time here?

You are looking at the most recent posts. You may also want to check out older archives. Please leave a comment, ask a question and consider subscribing to the latest posts via RSS or email. Thank you for visiting!

Here’s how it started…

Lisa (my wife) [shouting from office into the kitchen]: Tim, what’s this Amazon charge for $193?
Me [thinking what I may have purchased and not remembered]: Um, don’t know…let me look.

I then logged into my Amazon account to see what order I may have forgotten.  Surely I didn’t order $200 worth of MP3…that’s ridiculous.  Sure enough nothing was there.  Immediately I’m thinking fraud.  I start freaking out, getting mad, figuring out my revenge scheme on the scammer, etc.

Then it hit me: Amazon Web Services account.

The Culprit

Sure enough I logged in and my January 2010 billing account was $193 and change.  Yikes.  Well, I could let the (what has been averaging) $30 or so charge slide under the family CFO radar for a while…but this $193 charge…the chief auditor herself caught that one.

So I panicked.  I needed to figure out where/what the spike was.  I logged into the Amazon Web Services management console (I only use the S3/CloudFront storage in their services right no) to see what was going on.  I see ‘Usage Reports’ and click.  I’m met with essentially a bunch of useless data really.  No offense to Amazon, but really the usage reports weren’t really helpful at all.  First, they gave me a Resource ID which I thought would represent the URI I was looking for.  Nope, Resource ID == Bucket.  And they didn’t even put the bucket name in the report!

For some perspective, here’s essentially what I’m used to – here’s my December 2009 billing statement details:

December 2009 S3 CloudFront Billing

Anyhow, after some hunting it was obvious that I wasn’t going to figure out what bucket objects/unique URIs were causing my spike.  This was primarily because I didn’t have logging turned on at all on my buckets.  I had in the past but really didn’t think I needed it so I turned it off.

I was wrong – go now and enable logging.

While I was searching for a solution to understand my traffic, I was curious for where my traffic was.  Like I said, I’d been averaging (actually *peaking*) at about a $30 charge for the S3 hosting.

NOTE: I use S3 for all my image/screenshot/sample code file hosting.  I’ve invested in S3 for a long time and built my blogging workflow around it with building tools like S3 Browser for Windows Live Writer.

What was interesting was my most usage of my CloudFront data was coming from Hong Kong.  Compare to above the December 2009 billing to this January 2010 billing:

January 2010 Blling Statement

Yeah, that was my reaction too.  I went from roughly 40GB of transfer bandwidth to over 960GB in one month.  I suspected I knew what happened, but needed to confirm before I changed things. 

Implementing Logging for Statistics

The problem was that I didn’t have logging enabled and I was pretty much stuck.  I needed to get some data from the logs before being for sure.  I quickly found S3Stat and it appears to be the de-facto reporting for Amazon S3 log files.  I signed up for the free trial and generated a new access key to give them.

NOTE: They have a ‘manual’ option which means a lot more work.  I simply generated a NEW S3 access key for this specific purpose.  That way I didn’t have to give them my golden key I’ve been using in other places and can shut this off at any time without issue to my other workflows.

24 hours later, I had some reports.  Wicked cool reports.  Here’s a list of what I’m currently looking at:

  • Total hits, total files, total kbytes
  • Hits/files per hour/day
  • Hourly stats
  • Top 30 URIs
  • Top URIs by kbytes used
  • Top referrers (find out who’s using your bits without you knowing)
  • User agents
    Here’s a quick snapshot of one:
    S3Stat sample report image

Wow…honestly…THIS is what I was expecting when I see “usage” data reports.  S3Stat is awesome and you should use that now.  Yes, I’m buttering up to them…but they have a great tool here for $5/month if you are a heavy Amazon S3/CloudFront user.  Amazon frankly should just buy them and integrate this into their management console.  You can see other examples of their report outputs on their site at http://www.s3stat.com

What I also found out is that the tool I use for my desktop usage of S3/CloudFront (outside of my blogger workfow and S3Browser) has S3Stat integration built in!  I use CloudBerry’s S3 Explorer Pro for managing my S3 content.  It’s awesome and you should look at it.  When I look at the logging features in CloudBerry I see this:

CloudBerry S3Stat dialog

And after enabling the logging, within CloudBerry I can view the log data within the tool:

CloudBerry view logging

Summary

Wow, this is incredibly helpful and insightful data.  I now know who/how/when my cloud storage data is being used in various ways I can see the data.  S3Stat immediately showed me incredible value within less than 24 hours of enabling it.  I know can confirm the culprit of the burst of usage and plan accordingly.

Now, to be clear I’m not complaining about the cost of cloud storage.  That has been clear to me from the beginning.  Nothing is hidden and I’m not an idiot for not understanding it.  What I did not account for was the popularity of some files…and then the ones that just happened to be the largest.  I could not have personally thought I’d see a 920GB spike in one month of usage…but now I know…and have to alter some plans. 

Hopefully this is helpful for some who are just exploring cloud storage solutions/services.  Make sure you have instrumentation and logging capabilities turned on so you can identify and tune your situations.  For me, S3Stat and CloudBerry are winners for my personal usages.  If you are an Amazon S3 customer, I recommend looking at S3Stat and turning on logging immediately!



This work is licensed under a Creative Commons Attribution By license.


3/2/2010 11:33 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Alternatives to cloud storage? none really. Both S3 and Azure are similar pricing models and suit me fine when I anticipate the usage :-). I made a dumb mistake and put 100MB of popular files out there that were mirrored all over the place.
9/29/2010 6:24 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
960 GB is enormous. I use S3 and highest monthly charge I have seen is in the region of $2 so to get up to $193 is incredible. It is concerning to think that the monthly charge can be open ended and at the mercy of some third party hammering away at a bucket.

Thanks for the tip on the stats site. I have logging enabled on my buckets for my combi boiler site but the output is very hard to interpret.
11/27/2010 7:23 PM | # nice
The strategy of accounting for environmental justice seems consistent with the cost-benefit analysis philosophy of Cass Sunstein, administrator of the White House Office of Information and Regulatory Affairs (OIRA). OIRA sets economic analysis
design icons
12/6/2010 4:33 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
very well said.. My first professional programming experience ever was also like that: writing Java code three or four different ways and wondering why it wasn't changing. I figured it was something about the Servlets I didn't understand but of course it was caching. Sometimes you forget to check! apartamentos en alquiler en benidorm
12/10/2010 9:05 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Well thats perfect example of potencial "cloud danger". Everything stored on one place far away...
George @ track a phone
1/6/2011 7:02 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I really enjoyed reading this post, big fan. Keep up the good work and
please tell me when can you publish more articles or where can I read
more on the subject? I have been visiting various blogs for my thesis writing research. I have found your blog to be quite useful. Keep updating your blog with valuable information… Regards
vacatures | interim

1/27/2011 11:28 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
The strategy of accounting for environmental justice seems consistent with the cost-benefit analysis philosophy of Cass Sunstein, administrator of the White House Office of Information and Regulatory Affairs (OIRA). OIRA sets economic analysis..........
Austin Homes For Sale
1/29/2011 1:43 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Make sure to check out typical Vienna Souvenirs Thanks for taking the time to discuss this, I feel strongly about it and love learning more
on this topic. If possible, as you gain expertise, would you mind updating your blog with
more information? It is extremely helpful for me.
1/31/2011 7:41 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
2/15/2011 11:10 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
very creative effort it is more beneficial to users and makes their life more easy. Really, a genuine effort thanks.
Porch Swings
Austin Gutter Company
2/20/2011 12:16 AM | # :





Thank you so much for this amazing article. I loved to read that post. It was very interesting and informative. You are bookmarked.
dbz hentai
3/11/2011 1:32 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Really informative post mate really hope you keep the good work up, as i will be coming back often! Max and Cleo dresses
4/1/2011 12:08 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
ave a good idea on this nice post.I think Get Data Storage Services and only pay for what y
clarinet maintenance
4/11/2011 11:21 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
ome back to this blog for sure! I bookmarked this blog a while ago because of the useful content and I am nev
1,3 dimethylamylamine
4/24/2011 11:17 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
When Blobs attack – understanding cloud storage bursts and viewing logs - Interesting.Thanks for sharing this useful resource with us.San Francisco movers
4/30/2011 12:11 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Good to know that this topic is being covered also in this website & there are a lot of developers working on this segment but this is one of the best innovative idea ever seen.
essay writers
essay writing
4/30/2011 7:03 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs

Appropriée a great way of Being Introduced to this subject matter. It was great catching up is this. Thank you kindly For the introduction.
essay writing write my essay essay writers research paper
5/5/2011 6:29 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
the snapshots are really help for understanding the whole article ...

Thank you for sharing it with us !

Ryan
from Dubai Apartments to Rent
5/6/2011 5:08 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Retail offer cloud storage designed learn exotic dance online appeal to specific market segments. Nasumi and Egnyte are companies that sell file servers in the cloud to the small business market. Its products are based on the top services wholesale cloud storage, essentially a layer of abstraction. Expect an explosion of products in this space as entrepreneurs dream of ingenious new ways of delivering services cloud storage.mill valley waxing
5/11/2011 7:15 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I really appreciate the fact that you approach these topics from a stand point of knowledge and information. This is the first time, I visited at your site and became your fan. You are bookmarked. Please keep on posting.
comfortable office chair
5/13/2011 11:13 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I really enjoyed reading this post, big fan. Keep up the good work and please tell me when can you publish more articles or where can I read more on the subject.Thanks for nice info. It’s useful for me. Can you give me some more information with details? I will wait for your next post.

Cell Phone Spying
6/7/2011 6:52 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
There is a need for a more comprehensive articles. One that can easily and comprehensively understood by the audience. I must say that the writer did it! Its straightforward and honest deliver made a huge impact amongst its readers. Clearly, its unparalleled presentation and mastery is something that we must celebrate and proud of. More catalyst for development will be coming and the followers of this blog feels good to be part of it. Tenant Screening
6/19/2011 10:44 PM | # kool work
Hello! it nice to read this articles The reverse forecast system acheives consistant profits of over 500 GBP per week with low stakes. 30 day money back guarentee .
7 inch touchscreen
6/26/2011 1:32 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs

i love reading posts at your blog and this is really more amazing than all thanks for sharing this.Round Rock Roofing Company
7/6/2011 10:09 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I am really not too familiar with this subject but I do like to visit blogs for layout ideas. You really expanded upon a subject that I usually don’t care much about and made it very exciting. This is a unique blog that I will take note of. I already bookmarked it for future reference.Thank you. Luftfilter für Renault
7/8/2011 12:49 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs

am sure that this post will be very much helpful for people.You make it so simple to this.Thanks![12:48:08 AM] faiza khalid: cape farms
7/8/2011 1:55 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Blobs attack mean what? Can you expound your info.?Thanks anyways.

Dog Walking Calgary
7/11/2011 11:16 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
you can find in the net explaining everything in detail regarding the topic. I thank you for taking your time sharing your thoughts and ideas to a lot of readers out there. Cleaning Companies London

7/13/2011 10:47 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I really loved reading your blog. It was very well authored and easy to undertand. Unlike additional blogs I have read which are really not tht good. I also found your posts very interesting. In fact after reading, I had to go show it to my friend and he ejoyed it as well!
Orange County Properties
7/14/2011 12:37 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I found the perfect place for my needs. Contains wonderful and useful messages. I have read most of them and has a lot of them. Door Lintels To me, he's doing the great work. sea urchin facts
7/14/2011 11:50 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
familiar with this subject but I do like to visit blogs for layout ideas. You really expanded upon a 000-119 subject that I usually don’t care much about and made it very exciting. This is a unique blog that I will take note of. I already bookmarked it for future reference.Thank you
7/18/2011 9:16 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I found the perfect place for my needs. Contains wonderful and useful messages. I have read most of them and has a lot of them. To me, he's doing the great work.
MF Plasterboard Ceiling System
7/26/2011 1:37 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
good What a wonderful piece of information Admiring the time and effort you put into your blog and detailed information you offer! I will bookmark your blog and have my children check up here often. Thumbs up!
mobiltelefoner uden abonnement
boliglån
tærte opskrifter




6/10/2012 1:58 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I found the perfect place for my needs. Contains wonderful and useful messages. I have read most of them and has a lot of them. To me, he's doing the great work.
Cures for Heartburn
6/10/2012 2:01 PM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Good to know, thanks for Info!
Getaways for Couples
7/23/2012 12:04 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Thank you for this article. That’s all I can say. You most definitely have made this blog into something special. You clearly know what you are doing; you’ve covered so many bases. Thanks!chnlove
10/24/2012 1:12 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
Nice post. I study something tougher on totally different blogs everyday. It'll at all times be stimulating to read content material from different writers and apply a bit of something from their store. I’d choose to make use of some with the content on my weblog whether you don’t mind. Thanks for sharing. ppi
Gravatar
10/24/2012 1:16 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I discovered your weblog site on google and check a few of your early posts. Proceed to keep up the very good operate. I simply further up your RSS feed to my MSN Information Reader. Looking for forward to reading more from you afterward! erectionpills.com
7/3/2014 6:25 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
I love your blog, I recommend to all people who love this kind of post to visit.

minicunas
10/20/2014 6:42 AM | # re: When Blobs attack – understanding cloud storage bursts and viewing logs
This blog post interest me a lot because of your good insight about the topic. Its informative with lots of ideas.

 
Please add 7 and 3 and type the answer here:

DISCLAIMER:

The opinions/content expressed on this blog are provided "ASIS" with no warranties and are my own personal opinions/content (unless otherwise noted) and do not represent my employer's view in any way.