We've been monitoring bandwidth over the past few weeks between a server in your environment with various CDNs in order to evaluate your service. The bandwidth we're seeing is about 8-16x slower than servers in other public clouds. Here is a summary of what we are doing and the results:
http://cdn.cloudfiles.mosso.com/c53212/sample.gif (66KB) - 310 msec avg
http://cdn.cloudfiles.mosso.com/c53212/sample.zip (1MB) - 871 msec avg
http://dp71yw0bh3bah.cloudfront.net/sample.gif (66KB) - 373 msec avg
http://dp71yw0bh3bah.cloudfront.net/sample.zip (1MB) - 1 sec avg
http://http.cdnlayer.com/cloudharmony/sample.gif (66KB) - 409 msec avg
http://http.cdnlayer.com/cloudharmony/sample.zip (1MB) - 1.66 sec avg
For the same files with servers in the ec2, gogrid, rackspace, softlayer, and newserver public clouds, on average the 66KB file is downloading in 15-25 msec and the 1MB file in 100-200 msec.
Is there a cap on your bandwidth that might be causing this? Based on these tests, it appears that we are only getting between 5 and 8 mbps downlink from these 3 CDNs compared with 40-80 mbps in the other 5 public clouds, but the latency is about the same.
I'm acknowledging that I see your post. I personally do not have a response to our comparisons with other companies. I will bring it up with the team tomorrow.
The past few weeks we've used mechanical turk to get a few thousand users around the world on various high speed internet connections (mostly residential) to run our speedtest (cloudharmony.com/speedtest). The speedtest downloads 1mb and uploads 512kB from/to various cloud services including terremark. The test order is always random. This should provide a much larger and better sample for comparing Terremark's throughput with other public clouds. Using geoip, we were also able to tag each speedtest with country, state/province and ISP name and will be able to break out the stats accordingly. We're in the process of compiling the data now and will post the results here hopefully later this week.
Here is a sample from the data that we have already processed (about 300 speedtest executions). Since these speedtests are mostly based on residentical broadband connections, the average throughput is much less than the pingdom tests which were run from of Internet data centers. Once we have finished processing all of the speedtests, I'll update these results:
This first table shows the average download throughput (megabits/second) for ALL speedtests US and non-US. In this test, Terremark was 20% slower than newservers (based out of the same data center), and 47% slower than the fastest overall cloud Voxel based out of NY. It is also the 3rd slowest public cloud out of the 21 included in our tests.
This table shows the average download throughput for only speedtests from non-US connections. In this test, Terremark was 15% slower than newservers (based out of the same data center), and 59% slower than the fastest US cloud Voxel based out of NY.
This last table shows the average download throughput for only speedtests from US connections, Terremark was 22% slower than newservers (based out of the same data center), and 47% slower than the fastest overall cloud EC2 US East.
These tests don't show nearly the same amount of disparity between terremark and other public clouds as the pingdom tests did. However, they do continue to show a notable difference between downlink throughput from Terremark and other public clouds including ~20% less than newservers based out of the exact same Miami data center, and ~50% less than EC2 US/East and Voxel. The global tests also continue to show Terremark near the bottom in terms of downlink throughput relative to other public clouds.
I've compiled a few report runs from your URL.. this is a fairly typical report. Our Server is a vCloud Express 2V/8R.. unloaded.. just sampling things.. while overall we've been happy with the service, we've not really done alot of heavy loading (video streaming for Tv stations) because of reports like these..
As noted.. happy with the vCloud Express product for development purposes.. wondering outload .. if the vCloud Express platform is a better marketing tool for the non-shared products :-)
lets keep the thread fresh.. thanks for the input simgar.. Cheers!
Order Service Location Test Type Data Transferred Time (secs) Transfer Rate (Mb/s)
1 Rackspace CDN by LimeLight download 1.00 MB 2.10 3.81
2 Amazon EC2 EU West IE download 1.00 MB 4.22 1.89
3 Amazon EC2 EU West IE upload 512.00 KB 3.50 1.14
4 VISI Cloud Services MN, US download 1.00 MB 1.62 4.95
5 VISI Cloud Services MN, US upload 512.00 KB 2.30 1.74
6 Amazon EC2 US East North VA, US download 1.00 MB 1.51 5.31
7 Amazon EC2 US East North VA, US upload 512.00 KB 3.74 1.07
8 IBM Developer Cloud download 1.00 MB 1.22 6.55
9 IBM Developer Cloud upload 512.00 KB 2.59 1.54
10 Windows Azure Platform download 1.00 MB 1.10 7.26
11 Windows Azure Platform upload 512.00 KB 2.52 1.58
12 Linode VPS Hosting - London London, UK download 1.00 MB 3.72 2.15
13 Linode VPS Hosting - London London, UK upload 512.00 KB 5.11 0.78
14 Google AppEngine download 1.00 MB 1.19 6.70
15 Google AppEngine upload 512.00 KB 1.77 2.27
16 Linode VPS Hosting - Fremont Fremont CA, US download 1.00 MB 2.70 2.96
17 Linode VPS Hosting - Fremont Fremont CA, US upload 512.00 KB 2.97 1.35
18 Linode VPS Hosting - Dallas Dallas TX, US download 1.00 MB 1.46 5.48
19 Linode VPS Hosting - Dallas Dallas TX, US upload 512.00 KB 3.52 1.14
20 Nirvanix SDN download 1.00 MB 5.06 1.58
21 SoftLayer CDN by Internap download 1.00 MB 1.19 6.72
22 CacheFly CDN download 1.00 MB 0.36 22.04
23 Technology Insight Seattle WA, US download 1.00 MB 2.42 3.31
24 Technology Insight Seattle WA, US upload 512.00 KB 2.26 1.77
25 Flexiscale Cloud Computing London, UK download 1.00 MB 3.30 2.42
26 Flexiscale Cloud Computing London, UK upload 512.00 KB 3.85 1.04
27 Linode VPS Hosting - Newark Newark NJ, US download 1.00 MB 4.26 1.88
28 Linode VPS Hosting - Newark Newark NJ, US upload 512.00 KB 2.58 1.55
29 VoxCAST CDN download 1.00 MB 1.73 4.63
30 Highwinds CDN download 1.00 MB 1.52 5.28
31 NewServers: Bare Metal Cloud Miami FL, US download 1.00 MB 6.76 1.18
32 NewServers: Bare Metal Cloud Miami FL, US upload 512.00 KB 4.46 0.90
33 GoGrid CDN by EdgeCast download 1.00 MB 0.73 10.94
34 Amazon S3 US West Northern CA, US download 1.00 MB 3.42 2.34
35 VoxCLOUD Servers New York NY, US download 1.00 MB 1.69 4.74
36 VoxCLOUD Servers New York NY, US upload 512.00 KB 2.83 1.41
37 SimpleCDN download 1.00 MB 1.61 4.96
38 Linode VPS Hosting - Atlanta Atlanta GA, US download 1.00 MB 0.96 8.38
39 Linode VPS Hosting - Atlanta Atlanta GA, US upload 512.00 KB 1.86 2.16
40 Amazon CloudFront download 1.00 MB 0.60 13.39
41 CloudHarmony.com download 1.00 MB 4.32 1.85
42 CloudHarmony.com upload 512.00 KB 3.39 1.18
43 BlueLock vCloud Express Indianapolis IN, US download 1.00 MB 2.26 3.55
44 BlueLock vCloud Express Indianapolis IN, US upload 512.00 KB 3.30 1.21
45 prgmr Xen Servers Fremont CA, US download 1.00 MB 3.46 2.31
46 prgmr Xen Servers Fremont CA, US upload 512.00 KB 2.94 1.36
47 Rackspace Cloud Servers Dallas TX, US download 1.00 MB 1.54 5.20
48 Rackspace Cloud Servers Dallas TX, US upload 512.00 KB 1.96 2.05
49 Joyent Smart Platform CA, US download 1.00 MB 2.28 3.51
50 Joyent Smart Platform CA, US upload 512.00 KB 3.20 1.25
51 Amazon S3 EU West IE download 1.00 MB 2.55 3.13
52 Amazon S3 US East North VA, US download 1.00 MB 1.75 4.56
53 Terremark vCloud Express Miami FL, US download 1.00 MB 0.50 16.00
54 Terremark vCloud Express Miami FL, US upload 512.00 KB 1.74 2.31
55 Amazon EC2 US West Northern CA, US download 1.00 MB 2.73 2.94
56 Amazon EC2 US West Northern CA, US upload 512.00 KB 3.52 1.14
Average Download download 34.00 MB 77.83 3.49
Average Upload download 11.00 MB 65.87 1.34
Disclousure.. we're an ISP coding an Application with the Hosting Controller API and vCloud API for automatic provisioning IaaS.
We're looking to extend the Application to all Cloud IaaS Providers, but we're focused on the Iaas Providers that have good QoS metrics and and good product.
TM has a good product, the QoS metrics have yet to be dumped out despite a post in mid-December that 'they be posted after the holidays', but, given the data being posted, there is alot of upside to be achieve in the infrastructure.
Opinion.. The Terramark product the most polished front end, marketing, UI, etc, the nuts and bolts of the beta operation leave alot of upside potential and our wondering when when the that point in time will be realized. CIR/CBR is something we need to know upfront, not figuring out AS_Paths with CIR estimates from automated software tools the Routing Arbiters.
Looking for feedback from the helpdesk..
Hello! I referred to this post just now in a prior post and just wanted to bump this one up the ladder.
Based on the Post "Posted by vcs-PVieau on 12-22-2009 7:25 PM" back to my query of QoS Metrics.. "We are working on the rate metrics info. We will have that post sometime after the Holidays."
Whith all due respect to my brothers on the support desk, I asusme you were talking about Xmas/NewYears.. but perhaps I am mistaken.
Can you freshen this post with data we can consume?
Thank you for refreshing your request of QOS Metrics. I will forward your request to the right personnel and we will update you back on this thread.
Again, thank you for your patience.
Terremark Cloud Support
I have forwarded the update request to the appropriate teams. Our apologies for the delays. This information is not something support has access to.
We appreciate the feedback and continued updates to this thread.
Cloud Support Team
I have discussed this matter with our CTO. He is personally working with various teams to get the data, as well as reproducing the issue and attempting to correct it.
We will keep you posted.
After finishing compiling results from about 1300 unique speedtests from 115 countries, we came up with the following tables. In the end, Terremark's downlink throughput actually came out fairly decent.
For US-based tests Terremark was 8th of the 18 tested.
For US-based tests Terremark was 11th of the 18 tested.
For all tests (US and non-US) Terremark was 8th of the 18 tested.
98% of these tests were performed from residential high-speed internet connections. The 700 US tests covered most states. Our blog provides a more detailed description of these results: http://blog.cloudharmony.com/2010/02/cloud-speed-test-results.html
The speedtest is also updated to allow you to selectively download and upload to up to 26 different cloud services including terremark and download sizes up to 5MB: http://cloudharmony.com/speedtest
I hope this is helpful
Thanks for the update simgar. I have forwarded this information to the folks who are currently working on the slow bandwidth problem.
simgar: Thanks for the information.
Also, I'd like to clear up any confusion concerning QoS, rate shaping, etc. There are no caps or throttles of any kind on bandwidth for vCloud Express users. If you are experiencing performance issues, it is likely something to do with routing or latency. I have been gathering performance data and working with the network teams to determine if there is a problem and what the solution would be if a problem exists. We will provide updates as soon as we have something definitive.
Thank you for your patience.
Jason LochheadCTO, Cloud ComputingTerremark Worldwide
Jason, May I ask you to reply to my question, thanks!
I need to develop a performance profile for a VM on your Platform. I've got one for the OS level, now I need network level.
Question - What can we expect as customers for CIR/CBR to a VM at the network level? Is that virtual 10Mb Ethernet, or Virtual GigE? ;-)
For example, I config a full up VM and want to point OC-3 (or more) Capacity on east Coast requiring a CIR/CBR in SLA for my client.. who may only need the capacity for a few hours when news breaks or something stupid like a car chase happens.. which brings in alot of traffic, quickly.. a perfect application for cloud resources.. What can we expect for network performance per VM with respect to CIR/CBR and typical network QoS metrics?
Perhaps I should ask the question this way.. I'm willing to pay for performance QoS.. can you offer an SLA on QoS if I'm willing to pay for it? ping me offline if that is appropriate..
The direct question is.. short of our the business to Enterprise, can you do anything for this type of application I have with the vcloud offering?
I have forwarded the request to the appropriate team. This information is not something support has access to. Thank you for your patience and for you post.
Terremark Cloud Support
Since you are not rate limiting, could it be possible that there are some routing issues? We've been doing some intracloud testing to/from terremark vcloud to about 35 other data centers. Of those 35, one of the faster connections we get is between terremark and rackspace cloud and terremar and liquidweb. However, there is a very significant different between inbound and outbound throughput. If I pull a 100 MB file off the rackspace server, I'm able to achieve 75-90 Mb/s (inbound to terremark). However, if I pull the 100 MB file from terremark to the rackspace server (outbound from terremark), I'm only able to achieve 14 Mb/s. In running a traceroute, I see that terremark routes through limelight (llnw.net) to get to rackspace, but rackspace routes through level3 to get to terremark. I get the same results for a server with liquidweb which shows about a the same difference in outbound vs inbound throughput. terremark to liquidweb goes out through alter.net and sprint, while liquidweb pulling from terremark goes through level3. The highest outbound throughput we were able to achieve from terremark to all of the 35 other data centers was consistently never higher than about 14 Mb/s. However, if I push the file to those same servers, I'm able to see much higher throughput. Thus, could the problem be related to routing through level3 which seems to be capped at about 15 Mb/s when there are obviously more optimal routes that could be taken?
Here are a couple sample traceroutes:
Terremark to Liquid Web:
1 10.112.4.1 (10.112.4.1) 2.034 ms 1.616 ms 1.603 ms
2 22.214.171.124 (126.96.36.199) 1.873 ms 1.797 ms 1.686 ms
3 (188.8.131.52) 1.944 ms 1.446 ms 1.966 ms
4 184.108.40.206 (220.127.116.11) 87.343 ms 87.292 ms 87.228 ms
5 t0-0-0-5.br1.mia.terremark.net (18.104.22.168) 1.938 ms 1.933 ms 1.625 ms
6 GigabitEthernet2-0-0.GW9.MIA4.ALTER.NET (22.214.171.124) 37.404 ms 37.026 ms 37.309 ms
7 0.ge-1-3-0.XL3.MIA4.ALTER.NET (126.96.36.199) 43.842 ms 43.999 ms 43.704 ms
8 0.so-6-0-0.XT1.ATL4.ALTER.NET (188.8.131.52) 44.373 ms 44.202 ms 44.051 ms
9 0.so-6-0-0.BR1.ATL4.ALTER.NET (184.108.40.206) 38.200 ms 38.378 ms 38.027 ms
10 sl-st21-atl-6-3.sprintlink.net (220.127.116.11) 38.071 ms 38.266 ms *
11 sl-crs2-atl-0-4-0-0.sprintlink.net (18.104.22.168) 19.369 ms 19.350 ms 19.150 ms
12 sl-crs2-atl-0-0-0-0.sprintlink.net (22.214.171.124) 35.147 ms 35.088 ms 35.063 ms
13 sl-st20-chi-4-0-0.sprintlink.net (126.96.36.199) 33.735 ms sl-st20-chi-0-0-0.sprintlink.net (188.8.131.52) 33.862 ms sl-st20-chi-13-0-0.sprintlink.net (184.108.40.206) 33.732 ms
14 220.127.116.11 (18.104.22.168) 50.961 ms 50.921 ms 50.866 ms
15 22.214.171.124 (126.96.36.199) 40.023 ms 40.036 ms 40.017 ms
16 lw-dc3-core5-ge1-13.rtr.liquidweb.com (188.8.131.52) 40.812 ms 40.941 ms 40.692 ms
17 lw-dc3-dist8-po5.rtr.liquidweb.com (184.108.40.206) 41.065 ms 41.259 ms 40.949 ms
Liquid Web to Terremark:
1 220.127.116.11 (18.104.22.168) 0.909 ms 0.954 ms 0.922 ms
2 lw-dc3-core5-po7.rtr.liquidweb.com (22.214.171.124) 0.903 ms 0.883 ms 0.875 ms
3 lw-dc2-core3-ge2-15.rtr.liquidweb.com (126.96.36.199) 0.468 ms 0.544 ms 0.573 ms
4 lw-eqx-border5-te1-2.rtr.liquidweb.com (188.8.131.52) 9.814 ms 9.850 ms 9.838 ms
5 184.108.40.206 (220.127.116.11) 6.581 ms 6.632 ms 6.666 ms
6 18.104.22.168 (22.214.171.124) 6.654 ms 6.598 ms 6.639 ms
7 te-3-3.car3.chicago1.level3.net (126.96.36.199) 6.580 ms 6.572 ms 6.607 ms
8 ae-32-56.ebr2.Chicago1.Level3.net (188.8.131.52) 14.869 ms 14.833 ms 14.818 ms
9 ae-3-3.ebr2.Atlanta2.Level3.net (184.108.40.206) 42.850 ms 42.825 ms 42.797 ms
10 ae-2-2.ebr2.Miami1.Level3.net (220.127.116.11) 53.128 ms 53.101 ms 52.874 ms
11 ae-24-52.car4.Miami1.Level3.net (18.104.22.168) 41.508 ms 42.276 ms 42.383 ms
12 DATA-RETURN.car4.Miami1.Level3.net (22.214.171.124) 39.256 ms 39.169 ms 39.221 ms
13 t9-1.gw1.mia.terremark.net (126.96.36.199) 38.918 ms 39.381 ms 39.435 ms
14 188.8.131.52 (184.108.40.206) 38.970 ms 39.670 ms 39.342 ms
15 (220.127.116.11) 39.458 ms 39.577 ms 39.155 ms
16 18.104.22.168 (22.214.171.124) 38.903 ms 39.251 ms 39.147 ms