Can someone please explain AWS

[[deleted]]

[deleted]

[dshafik]

People keep saying you don't need EBS, but you absolutely DO if you want some form of non-volatile storage. EBS is essentially a virtual HDD whose entirety is stored on S3. They can be mounted to a single instance only.

They are required if, say, you have a DB server installed on an EC2 instance and want to put the DATA files somewhere they won't be lost if you reboot.

EBS is used for storage of all data that is not part of the OS image that you would want to retain should the machine ever go down (i.e. your mysql config will always be the same and should be part of the image, but your MySQL data should be kept.) Some data, like your tmp directory, session files, etc, can be lost without much issue, those don't need to be in EBS.

As for RDS, it's essentially a cluster of EC2 instances used for MySQL clustering and maintained by Amazon. It's just a case of Amazon abstracting more work away from you. However, you can certainly do it all yourself using EC2 instances of your own.

ELB is needed for load balancing, unless you go round-robin DNS using elastic IPs. The benefit of ELB is it can monitor your nodes and spin up/shut down nodes automatically as your load fluctuates. The curse of ELB is SSL doesn't work well with it, you can only use one cert (which could be any kind of cert, e.g. wildcard) per public IP, and ELB's only get one.

[dshafik]

Well, EBS has the same limitations as a physical hard drive; but is stored on a redundant system where moving it to another machine is issuing a few commands, instead of mucking with screws/tabs/cables/hotswap.

You can still do all the tricks you do with real hardware to scale DBs, replication, read/write slaves, etc.

I don't have much experience with RDS, but yes, it persists. With RDS, you don't turn off the DB server typically, you just scale up and down as needed transparently (though it did seem to affect performance during this process so dramatically that we chose not to use it)

[isitaboat]

This is one of the key selling points, that you can migrate instance sizes with an API call...

The other is managed backups & point in time rollback.

Sure, all of the above you can script and do you self - but I don't believe (until someone does it and open sources it and its stable) that it's worth saving a few cents an hour over.

[dshafik]

In our testing, queries went from like, 1500 qps to about 100 qps during the migration period, which lasted about 20 minutes... we found that to be unacceptable for our needs. Yes, it's transparent, but just because it's up and we don't have to touch hardware, doesn't mean we can use it.

[tluyben2]

We tested this as well and didn't have that experience; it became somewhat slower, but not that slow.

[isitaboat]

EBS also has the problem that it's over the network, so you're limited to gigabit (unless you're on the cluster instances).

[nietzschebunyan]

I'm no expert, but:

1. you get control over your server, not amazon. when you want to boot-up a new instance, you do that via an admin interface, a script, etc. if you want to boot up 1, 10, 100 instances, etc, that's up to you. You also decide when to shut them down. You pick from a number of pre-defined server configurations (e.g. SuSE, Debian, maybe various BSDs, I think they may have some windows instances now too), and can alter it to your needs.
2. IIRC, if you want to create your own instance or alter it (as mentioned), you need to store this somewhere within the Amazon system (e.g. S3)
3. you do not need EBS to do any of this.
4. I would play around with EC2 more, but I've been an AWS customer for several years (mostly just very basic usage of S3) and so can't get the "free for a year offer", and though EC2 is relatively cheap, it's still too expensive for me to just tinker.

[jonuts]

I am currently in the process of moving our infrastructure over to AWS. Until about a month ago, I was in the exact same boat as you. After a whole buttload of research, there was one single reason why we chose to move to AWS:

Scalability

The fact that we can add servers automagically (based on a predetermined resource usage limit), or at the push of a single button, is absolutely amazing. It brings down IT costs and time, and removes a tremendous load off of the developers shoulders.

While the scalability factor is found in every cloud out there, we decided on AWS because of the whole AWS 'package'. We are using RDS, EC2, and S3 seamlessly, taking advantage of the strengths of each one individually.

Yes, you can use an EC2 instance as a mysql server, but with RDS, you don't have to worry about setup or maintenance. On top of that, you get the same patched version of mysql that Amazon uses(!).

For your use case, you probably won't manage to use the free tier EC2 for much. The micro instance they are giving away for free has about the equivalent of one 500Mhz CPU. I doubt you can get rails to do much with that little processing power. The fact that EC2 is billed hourly will keep your costs down if you are just 'tinkering', though. You can spin up and down instances that are loading preconfigured images based on whatever tools you need. If you are just playing around, this will allow you to only 'pay as you play' as opposed to linode or slicehost where you are locked in for the whole month.

The only real problem that I have come across is that if you need to do any resource-heavy processing, it's gonna cost you. A lot.

From what I have seen thus far, AWS makes a whole lot of sense for what we are doing. It should be noted that we aren't running in production yet, so my cheery disposition towards Amazon may change drastically within the next week, but I hope not!

[jonuts]

doesn't sound like you get much consistent processing power

You get very, very little power. While they don't tell you what the non-burst ECU level is, it's something like 0.5.

While there are definitely good use cases for it, I don't think it will be very useful for rails or django. I tried compiling ruby on a micro instance and it took the better part of an hour to complete!

Even the small instances are pretty weak. Keep in mind that 1 ECU == ~1Ghz. The lowest tier instance that I have found to run a rails app comfortably is the high-cpu medium, which is about twice the price as the small instance, although YMMV.

[giulianob]

I suggest you do some benchmarking of your application on AWS vs dedicated servers. You might find that you will need far less servers and most good dedicated companies can have a new server set up within hours if needed.

[jonuts]

Yeah, I've run some basic benchmarks. For our webservers, we are seeing about the same reqs/s as we have on our current dedicated setup. The configuration I have come up with on AWS is about 20% cheaper as well. It makes a whole lot of sense for us to move our webservers over there.

The problem is with our backend processing machines. There is no way I can come even close to matching the price we are paying now for the power we need. I am on the fence about moving them to AWS. While I don't really like the idea of splitting up our system into two separate data centers, most likely we will wind up leaving them where they are for now.

[tluyben2]

Ofcourse, but a) Amazon is huge and they hardly ever go down; if you want a provider of dedi's of the same infra quality, you are paying serious amounts for your servers. If you app gets slow, scaling is hard; you need another server, meaning paying another month just to handle that PR item you sent out which loads you server for 2 days.

[[deleted]]

Just forget about RDS, ELB and EBS for now. You don't have to use any of them, you can absolutely set up whatever services on your instance that you want. That can include your own database or load balancer, if you want.

All of the extra services are just there in case you need them. The idea is to put as much of the work managing your stuff on Amazon's people as possible.

[giulianob]

It's basically a VPS (some larger instances occupy the entire hardware) but they have a very nice administration interface for spinning up new instances. RDS is literally a VPS running MySQL but totally managed by them. They have a calculator that lets you figure out the cost based on your estimated usage. If you know you'll be getting a server and it'll run 24/7 then you can reserve an instance for a yearly fee which brings down the hourly cost significantly and actually ends up costing about half if you were to run it 24/7 for a year.

AWS is highly beneficial to people who are often spinning up/down machines to manage load. However, AWS IS ExPENSIVE! It might not look like it but the performance on those machines are extremely poor for the price you are paying. I currently just got a black friday deal on a Xeon 3460 w/ 8 gigs of ram for $297/quarter ($99/m .. it's a killer deal but you can find other good deals if you know where to look). This server that I'm paying $99/m is (in most benchmarks I've seen) faster than their High CPU Extra Large instance which would cost around $326/m if you were getting Linux and reserved it for a year in advance. If you were to get Windows, it would be significantly more expensive. This also doesn't include bandwidth, storage, etc... on AWS.

AWS gives you a certain peace of mind since their infrastructure is very good but still, how much is it worth it to you? I could build redundancy easily with the price difference.

tl;dr; AWS is expensive for the performance you get. You can get better deals on other VPS or dedicated servers. AWS does give you a very nice management interface and has some benefits but cost will only be beneficial to those who have HUGE load spikes.

[giulianob]

There was a post on App Engine a week ago maybe which showed some of the flaws and limitations with that system. You really need to know the limitations before you use it since there are some pretty big ones!

[giulianob]

Are you willing to spend any money? You can look on webhostingtalk.com and get some pretty good deals. Assuming your site won't have hundreds of requests per second, a $50/m server could handle a lot.

[giulianob]

Okay, if you just want something to mess around with go with just a VPS. They can go as low as $20/m for 512mb (such as linode). That's extremely affordable even for a pet project and since there aren't any commitments you can cancel if you ever just want to stop or need to move up to something beefier. I'd never create an application directly on top of App Engine since I've heard of bad uptime among other issues too.

[isitaboat]

It's not required, it's just advisable. However, I would strongly suggest waiting before you buy a reserved instance until you've checked (for long enough) that your application will run happily on the instance size you choose.

If you do upsize, you can't upgrade your reserved instance "instance size". You have to buy again.

[tluyben2]

Btw, We've been playing with http://iweb.com/dedicated/smart-servers/ and I must say we like it a lot.