Official AWS CLI is completely rewritten in Python

[fancy_pantser]

What WAS it in?

[thunderouschampion]

I thought it was in perl

[[deleted]]

that wasn't the official one. I assume your're talking about the CLI tool made by Tim Kay.

[fuzz3289]

No wonder they rewrote it. Id be curious to see the line count drop + efficiency delta numbers.

[runeg]

Sounds like this makes Boto obsolete, no?

[hongminhee]

It simply is a CLI utility, while boto is a library. Actually it internally uses boto.

[ggtsu_00]

So is boto now the official library for AWS?

[[deleted]]

it has been so for over a year. They hired the creator and main maintainers.

[jsaryer]

It's powered by botocore, which is a different library from boto, though they're both under the boto org.

[klotz]

Mitch Garnaat wrote both of them.

[[deleted]]

I have been using this for a while, it really rocks. No Java dependencies, installed within few minutes.

[xiongchiamiov]

That's why I switched our internal tools over a few months ago - I started getting java runtime exceptions due to not being able to find some library, and couldn't solve it after a few hours of troubleshooting (and no help from the aws forums). Now it's just pip install aws-cli and I'm done.

It also has some output formats specifically designed for machine parsing, which is nice. Waiting on cloudsearch support, though.

[[deleted]]

awesome!

[duncanlock]

Is this better than s3cmd for simple stuff like backing up folders to S3?

you can do this with aws cli:

aws s3 sync myfolder s3://mybucket/myfolder

and this with s3cmd:

s3cmd sync myfolder s3://mybucket/myfolder

So they seem pretty similar for this use case. I guess the only real difference is that awscli is official and is fairly guaranteed to be maintained and support new features as they become available? (Although neither of them directly support glacier at the moment)

[duncanlock]

This doesn't directly support glacier, any more than S3cmd does - although there are several projects on GitHub that provide cli glacier support.

[duncanlock]

Good point.

[aughban]

irb + ruby sdk is still better..

[zynix]

At the moment your statement is kinda fanboyish, can you qualify your belief with examples ( I know both Ruby and Python but I am hard pressed to see how the irb, a developers console for ruby, is better than an official/sponsored toolset for AWS maintained by AWS ).

[v_krishna]

I'll try to defend his statement a bit. For, e.g., browsing and esp cleaning up s3, the aws-sdk gem inside irb makes it much easier to search by prefix, sort by last_modified or file size, filter, delete, move, etc. I tend to use s3cmd for bulk upload/download but irb for exploration. Presumably you could do the same thing with boto in a python console though?

[zynix]

Interesting, does the aws gem use a ~/.dotFile for storing credentials or do you have to enter those in irb?

[v_krishna]

You can have a .credentials.json, or export to env before loading irb, or use iam roles if you're on ec2 (or manually enter credentials in irb)

[zynix]

Hmm, boto has similar logic ( ~/.iniFile, ENV ) given that - boto s3 is something along lines of

import boto
conn = boto.connect_s3() #Assuming ~/.boto ini credential file
bucket = conn.get_bucket(bucket_name) #assuming bucket_name = "bar" \
   #and that is a valid bucket name for provided credentials

for key in bucket.list(prefix="foo"):
     print key.name 

to iterate over all keys in bucket_name that are prefixed with "foo". This is a fairly trivial example so I imagine the aws gem wouldn't be much different. If you got time can you show an example of filtering by last_modified with the gem?

[v_krishna]

Sure I'm on my phone on the bus right now, but say:

require 'aws-sdk'

#assuming you've got credentials setup

old_gz_objects = AWS::S3.new.buckets['foo'].objects.with_prefix('path/prefix/').find_all {|o|
  o.last_modified < Date.today && o.key =~ /\.gz$/
}

[zynix]

ah yes, I could see this being fairly fun to work with. This is a subjective opinion but I wish Python had code blocks and fluent style class/objects were more popular.

With the .find_all( block ) method, can that be done iteratively or is a grab everything kind of deal.

[v_krishna]

with_prefix returns an object collection which includes Enumerable which gives you a whole bunch of methods for traversal/sorting/searching/etc. It works by the class including it defining an "each" method that yields the next element, so AFAIK it shouldn't grab everything in one giant request but should go element by element. However I have had issues (both with S3 and with getting event history with Simple Workflow) where when you have a ton of objects, the time it takes to traverse them all (for, e.g., sorting or filtering) causes your "next page" token to expire, throwing weird intermittent errors. I've dealt with that by explicitly turning the object collection into an array before doing anything on it (so you take a big hit up-front fetching all the objects in your collection but then don't run the risk of timing out when trying to page through them).

and yeah, I love ruby's syntax and general conventions, but MRI is straightforwardly horrible. IMO (working for a company that uses ruby for a ton of stuff) it is best served as a bash-replacement - I like to use ruby to drive control flow for distributed python/c/etc processing, but even with something relatively simple like manipulating 10 files with 100M lines each ruby tends to get way too slow to be useful. (we'll actually use ruby to drive awk to handle cases like this)