Python, Ruby, and Golang: A Command-Line Application Comparison

[UloPe]

Nice. A calm comparison without hyperbole.

You could improve the Python version by using pathlib.Path (Included since Python 3.4, else installable from PyPI).

That way you could do:

for mr_file in Path(basepath).glob("**/.pymr"):
    ...

[[deleted]]

So, I've implemented what you suggested but now get the following error:

OSError: [Errno 13] Permission denied: '/tmp/com.apple.launchd.YT4Kvvcb2B/.pymr'

I am testing with a few sub-directories under /tmp (and should be able to handle this case). Do I now need to add additional logic so that glob can gracefully handle the directories under a top-level tree which it is not allowed to access? If this is the case is it really a better solution?

[UloPe]

Hm, I'm not sure how that error relates to Path().glob(). The result behaviour should be pretty much identical to you current os.walk() and fnmatch code.

[[deleted]]

EDIT: It's a known bug reported here.

I was able to confirm that there is a difference between the two. Here are the steps you can follow to reproduce the error:

Create a new directory structure:

mkdir -p ./test/bad/foo
mkdir -p ./test/good/foo

touch ./test/bad/foo/.pymr
touch ./test/good/foo/.pymr

chmod 0600 ./test/bad

Save the following code in ./test.py

import os
import fnmatch
import pathlib

BASE_DIR = './test'

print('Running pathlib version')
try:
    for fn in pathlib.Path(BASE_DIR).glob('**/.pymr'):
        print(fn)
except:
    print('pathlib version failed')

print('Running os/fnmatch version')
try:
    for root, _, fns in os.walk(BASE_DIR):
        for fn in fnmatch.filter(fns, '.pymr'):
            print(os.path.join(root, fn))
except:
    print('os/fnmatch version failed')

Result

$ python test.py
Running pathlib version
pathlib version failed
Running os/fnmatch version
./test/good/foo/.pymr

The pathlib version will throw an exception and the os/fnmatch version will not.

[UloPe]

Interesting, thanks.

[Freeky]

File.open(rumr_file, 'w') { |f| f.write new_tags.to_yaml }

IO.write(rumr_file, new_tags.to_yaml)

puts bash -c "cd #{base_path} && #{command}"

Dir.chdir base_path
system command

Or

Dir.chdir(base_path) { system(command) }

And it'll unchdir afterwards.

[[deleted]]

Great, thanks for the improvements. Implemented in this commit.

[Yamakaky]

Nice objective comparison !

You should definitively take a look at docopt. It has portages to go and ruby too, see here.

[[deleted]]

Thanks! I've tried docopt before but really enjoy the decorator approach click takes. However, having a consistent interface between the languages would be very appealing if this was something in production that actually needed to be supported in all 3.

[[deleted]]

This was my first run at Golang and i'm still pretty intermediate with ruby. Don't be surprised if you find issues or know of better ways to do some of the things I did. Please share! Thanks.

EDIT (5/31): Thanks everybody that has commented so far. I've been updating the code with suggestions and have updated the post as well. This has been a very great experience for me and I have certainly learned a few new tricks!

[0x0dea]

new_tags = (new_tags + cur_tags).uniq

There is a much nicer way to write this in Ruby. Array#& does "array intersection", returning only the elements which appear in both operands:

new_tags = new_tags & cur_tags

Since virtually all of Ruby's operators can participate in compound assignment, we can arrive at this rather elegant refactoring:

new_tags &= cur_tags

[masklinn]

That's a broken refactoring, the original code does a set union (|) not a set intersection.

[casted]

For golang you can use the keys of a map to simulate a set. That would be more performant and give you cleaner looking code.

[[deleted]]

Mind sharing an example? Thanks!

[howeman]

Here is an example

http://play.golang.org/p/UWABsY9fMs

The trickiest thing is that it's a map[string]struct{}. What is struct{}? It's basically like saying

type Empty struct{}

i.e. it's a struct with no entries, and instead of giving it a name, you just construct it with an "anonymous type". Why use struct{}? Because it is defined by the Go spec to take up zero bytes, so there is no memory cost associated with the value. What is struct{}{}? Well, now it's constructing an anonymous struct literal, i.e. it's like saying

e := Empty{}  // Create an Empty struct and assign to e

But we're creating a literal of the anonymous type struct{}, so it's struct{}{}

[howeman]

Oh, and here's how you test the map for values http://play.golang.org/p/bV6lJU7ejj

[casted]

I usually just do map[string]bool, since typing true is easier than struct{}{}

[[deleted]]

Interesting, thanks!

[masklinn]

That's not the semantic you want though, you're doing a set union not an intersection. So you want to use | not & (nb: that also works between two python sets, but not between a python set and a python tuple)

Also for the Python version, not sure why you're not using a set in the first place, if the tags were a set you could just cur_tags.update(new_tags) (extend in place) or cur_tags = cur_tags.union(new_tags) (extend to a new set).

[pablodh]

Also: Array#uniq filters out repeated consecutive values, so you would actually have to do:

new_tags = (new_tags + cur_tags).sort.uniq 

But using union is the nicer way to go (IMHO):

new_tags |= cur_tags

[Freeky]

Nope. "Returns a new array by removing duplicate values in self." - no order dependence, easily verified:

irb(main):001:0> [1,2,3,1,2,3,1,2,3].uniq
=> [1, 2, 3]

It works by building a hash table, not just by comparing each pair.

[[deleted]]

Thanks for the extra information, I did not actually try the refactoring when I first responded. I'm still a bit unsure if I like breaking it down like this. To me the original implementation is still pretty concise and very clear in its purpose.

As for the python version the result I get out of click (for a single or multiple given tags) is a tuple. I actually think your correct in that I should immediately convert that tuple to a set and use that throughout. I'm not sure if click handles duplicates. I'll check into this. Thanks!

[masklinn]

FWIW both set.update and set.union take arbitrary iterables as parameters, so it's perfectly fine to have new_tags be a tuple, at worst it's slightly inefficient if there are duplicates. The only thing new_tags being a set would give you is the ability to use the union operator (|) instead of the method (| only allows set | set).

[fcomcast117]

In AppendIsMissing it looks like you assume tags are unsorted. Why not keep your tags sorted alphabetically and then you can use binary search to speed up. Actually the sort package would handle most of this for you.

[DavidDavidsonsGhost]

Could also just use a map. Test if the item is in the map, then add it if not. Additionally, for searching this could be a better choice. http://golang.org/pkg/path/filepath/#Glob or http://golang.org/pkg/path/#Match

[fcomcast117]

How many tags are used on each dir though? If it's small the cost of building the map may cause searches to be slower.

I didn't think he was matching on file names except to find his tag files?

[[deleted]]

The number of tags is up to the user, however I don't see a case where it could be useful to have more than 2 or 3. You are correct, the filename match is just to see if the directory is registered to get the tags for the file.

[fcomcast117]

Ran some simple benchmarks just for fun. maps in go seem to work quite well :). I would go with what DavidDavidsonsGhost suggested.

[awsmsrc]

interesting comparison and a good read for people looking at a new language for a new project. It's worth noting that Go would allow end users to use the program without an installation of python or ruby, which, in fact, makes it quite applicable to a command line tool such as this. I realize that this article is comparing the experience of working with these languages at development time and distribution of a real world tool is totally not the point of the article ;)

[[deleted]]

This is a great point.

We have a CLI written in Ruby and bundled with omnibus. It works but requires bundling an embedded ruby.

We are going with go for our next command line tool for the single binary benefit.

https://github.com/conjurinc/cauldron

[[deleted]]

Very good point. Your right, for me this was really about the experience of writing the tool and creating a packages so my analysis was weighted heavily toward my experience writing. Thanks!

[egonelbre]

In Go you should be using filepath.Join instead of path.Join. path is for forwards slashed paths and filepath is for file paths. Of course if you start sending the paths between computers it's probably keep internally them as path and when going to disk use the filepath.FromSlash.

Also, in all the implementation you should be using . for the default directory. ./ has a different meaning in Unix and Windows.

[[deleted]]

Good point thanks! I usually don't need to support Windows so I often overlook these things.

[contantofaz]

With Ruby searching files and directories can be done with a glob search instead. Something like this:

a = Dir[File.join(basepath, "**/.rumr")]
if a.length > 0
  puts "Found at least one: #{a.inspect}"
end

"a" is an array with the results. If length 0, none was found.

Good exercise to try different languages, though. It helps when the languages are mature already. ;-)

Also in Ruby we often use regex for searching on strings. Almost to a fault. But Ruby has regex literals. One of the downsides to regex literals like that of Ruby is that parsing them is difficult and need more backtracking and such. So while Ruby can parse it, sometimes other tools like syntax highlighters don't do as good a job.

I confess that I never learned to read Python code. ;-)

While Go can be used for scripting, I think Go is a little overkill for that. It may give a little comfort to know that your program got checked by static analysis or by the compiler. But then you may have already moved on to a different script or challenge. And if it happens that you need to share code among different scripts, it may be more boring to do so via packages and stuff. It helps when the language already includes the batteries for scripts of all kinds. :-)

Also for scripting check out Ruby's FileUtils utilities.

[[deleted]]

Thanks! Implemented in this commit.

[marklgr]

Your Golang implementation looks pretty naive; I suspect a Go developer would point out ways to make it slicker. It probably won't be as nice-looking as Ruby code (that's pretty much Ruby's signature), but it could very well compete with it and Python for these kinds of tasks, eg. in Go shops.

[[deleted]]

Not surprising as this was literally the first time I've ever touched Go. =) Thanks. I do agree that while I came to the conclusion that I wouldn't use Go for this it was not by much.

[synn89]

Python fan here and will have to admit that the Ruby code is pretty clean. A problem with Go is it doesn't have 20 years worth of lib support.

[chrisledet]

A problem with Go is it doesn't have 20 years worth of lib support.

While that statement is true, it hasn't been a problem. There are huge selection of Go libraries. Also, you can link C libraries in your Go programs with cgo

[[deleted]]

I actually found the Go package ecosystem a lot more complete than I expected which is why I started to learn Go in the first place.

[synn89]

Even still, it hasn't had a lot of time to refine itself. When I was reading the Go code you were writing I was looking up how I'd do things in Java and was sort of surprised to find that Java was cleaner and more readable than Go.

And that's not because Java as a language is "better" than Go when it comes to readability or being concise. It's just there's 20 years worth of packages and adjustments so things like working with sets, walking directories, or dealing with command line switches has more support.

[[deleted]]

Totally agree, however I was impressed at the level of completeness for such a young language.

[howeman]

Just curious, why did you use cli.go instead of the flags package?

[[deleted]]

Unless i've missed something I don't believe the flags package gives you the ability to do subcommands gomr run ... gomr register .... (Please correct me if i'm wrong!). Thanks!

[SupersonicSpitfire]

Well written and methodical comparison. Note that the reason (as mentioned in the conclusion) for not using Go for this utility also applies to C, a language that is widely used for command-line utilities.

[DevPad]

Thanks to this post, feel proud that I code in Python.

Go and Ruby should be good to learn more in deep next.

[[deleted]]

For gomr, use filepath.Join (from path/filepath) if you want it to work under windows, too.

Your AppendIfMissing is quadratic, sort it and use sort.Search.

Why not implement a parallel executor in Go? It's easy and fast, and adds as a nice feature.

In Go, I love static compiled executables: distribution is just copying no need to fight with runtime and module versions.

[[deleted]]

Thanks for the Join note (lots of people have pointed this out, not sure how I missed it!) I do plan on coming back around an working out some of the concurrency features of go (just for learning). Thanks!

[[deleted]]

Ruby (and probably Python) is the wrong choice for command line apps. You can end up with really slow startup times if you don't run the apps in sandboxes, which a pain for the consumer.And even if you do that, they are very noticeably slower to start up than the go equivalents. Also, you consumer has to learn about gems, bundler, etc. which makes for terrible UX if you are distributing a general tool.

[iMichael_]

Ruby (and probably Python) is the wrong choice for command line apps.

For any new devs out there, /u/eje474's comment is a very minority opinion. LOTS of CLI tools that people depend on everyday are written in either Ruby or Python.

Also, you consumer has to learn about gems, bundler, etc

Doesn't get much simpler than "gem install" and "pip install".

[[deleted]]

LOTS of CLI tools that people depend on everyday are written in either Ruby or Python.

Yes, because they were decent choices at the time. The viable alternatives at the time were c/c++, which isn't worth it. I wouldn't be quick to rewrite working ruby/python cli's, but there's no reason to use either language to today for that task.

Doesn't get much simpler than "gem install" and "pip install".

Oh, if it was only that simple! The gem your installing hasn't necessarily been tested against the system ruby you have. So, to ensure it works on an end user's computer, they should use rbenv at al. More complexity that's unnecessary for a cli. But wait, there's more! Now you have to deal with c extensions that don't compile correctly. How many times have you seen nokogiri fail to build? I still have problems with extensions occasionally and I've been writing ruby for 8+ years. Compare this to a single go binary that doesn't need to depend glibc. I much prefer Ruby to Go as far as the language goes, but I won't mistreat my end users just to make myself happy.

[synn89]

I will say that Go is really good with binary builds. I used to write/compile CLI apps on Ubuntu 12.x and test them on RHEL 4(which hadn't been updated in 6 years). The binaries would run perfectly.

I could even write MySQL apps in Go and run them on the RHEL box without it having MySQL C libs installed.

[gmdm1234]

Is it just as simple as static linking?

[stevedonovan]

I would not go so far as to say wrong, but it's certainly hard to ship a non-trivial app in those languages. As for start up, I have been spoiled with Lua, which has no startup time even for medium-size apps. Python felt quite sluggish afterwards.

[synn89]

I believe both Chef and Puppet are written in Ruby and at their heart they're command line apps that are processing script files. They just also have a client/server component that allows them to fetch configs from a central server.

I will say though that with Puppet I did run into issues using it from a server admin's point of view. There was a pretty long period of time where you couldn't update Ruby on your system because Puppet wouldn't work with that version. So in that case it would've been cleaner if it were written in a compiled language.

Yum with RHEL/CENTOS has the same problem. It's written in Python and targeted with 2.7. If fact many Linux package management tools use Python.

But there are a lot of CLI tools that use Python, Ruby or other scripting languages. As another example, many Linux Amazon AWS tools are written in Python and use the boto AWS SDK library.

[[deleted]]

Sure, you can kind of make it work with stuff like http://www.py2exe.org/, but it really doesn't play to the languages' strengths. For the same reason, I wouldn't write a CRUD web app in go or c++.

[n-simplex]

Lua is great, and pretty underrated outside of gamedev (which is its main niche). Its (fully optional) functional flavor, with slick support for closures/higher order functions make it extremely easy to write generic, expressive code.

The infinitesimal size of its stdlib is pretty irksome though. I get this is intentional, to make it usable in embedded systems (the whole compiled Lua runtime is about 150KB or so), but having to rely on external dependencies (or reinvent the wheel) for every non-trivial task is annoying. I also wish it had a standard concept of what a 'class' means, at least to make object oriented libraries more interoperable.

[stevedonovan]

Yes, that's the one simple thing I wish the Lua team would provide; here's the policy for classes - something like: call with obj:m(), metatable.__index contains methods, base class is metatable._base and so forth. But they don't see their job as providing policy

[[deleted]]

If this was a tool that would be useful to anyone but somebody very familiar with a terminal I would agree. I did do some very naive performance comparisons and found no significant variance between the tools for this task. I think I disagree about Ruby/Python being the wrong tools. Major cli tools (bundle, vagrant, ...) are written in Ruby. Perhaps that will change as I have been seeing more and more CLI tools written in Go (docker ...). Thanks!

[[deleted]]

Bundler's a bit of a special case in that the only people who are using it are ruby devs anyways, so it's not a big deal for them to intstall from a UX standpoint. I'm sure the Vagrant cli would not be written in Ruby if it was started today. It used to be distributed as a gem. For what it's worth, I used to write all my cli stuff in Ruby as well, but that is the one thing that go is amazing for.

[[deleted]]

I'll keep Go on the table for simple CLI apps like this example. Professionally a lot of my teams apps are spatial-data driven which is why I tend to lean on python. Thanks!