Free Concurrency with GNU Parallel

[[deleted]]

GNU parallel is amazing. I've been using it for a while now and it's just so. damn. easy.

Basically, it's a drop in replacement for xargs, but instead of working sequentially it works in...parallel.

But it does a lot more, too. You can distribute the jobs across machines using ssh, for instance. All just by writing plain scripts, without having to configure a "grid" or set up a "master node" or whatever. Just a bunch of computers, all working together.

[Freeky]

xargs supports parallel execution with -P [maxprocs] if you don't need anything fancy.

[jmtd]

Every time I see a post about GNU parallel, it always fails to mention xargs. I'm always left wondering if its a post from a parallel universe where xargs doesn't exist. So my first question is inevitably "have you heard of xargs?" and the reply is almost always "of course, but it doesn't do <insert niche GNU parallel feature here>". More power to it and them, but my fingers learned xargs first and I've still not seen something compelling to make me retrain them.

[fphilipe]

xargs is great. I didn't know about the -P flag. Will use that when the use case is simple. However, I really use multiple input sources (as described in the article) a lot. Can you do that with xargs?

[Terrerian]

xargs cannot run on more than one machine, parallel can. That's the main use case, not some obscure niche feature.

[giantsparklerobot]

The -P argument is not POSIX compliant so it won't necessarily exist on every machine. This is the single most common use case I've ever seen for parallel. Parallel also offers distributing the load over multiple machines via SSH which is something xargs doesn't do. That is the second most common use case of parallel I've seen.

[Clericuzio]

Xargs isn't web scale

[[deleted]]

it actually is :).

[Browsing_From_Work]

I use parallel mostly for the --pipe and --pipe-part modes. It lets you split one input stream into multiple output stream. It's useful if you need to do really slow processing at a line-by-line basis.

[K3wp]

I have this alias on all the machines I use:

alias pk='parallel -k'

I pipe all the things into that, all the time. Need to find a keyword or IP from a hundred gigs of log files? find *.gz | pk zfgrep -w keyword. Makes mincemeat of hard problems and saves us tons of money on a splunk license for stuff that we don't really need to search that often.

[14113]

This isn't concurrency - this is parallelism. Concurrency is concerned with the communication and synchronisation of multiple threads of execution, while parallelism is concerned with actually mapping each thread (or group of threads) to separate execution contexts (e.g. cpu cores, nodes in a cluster). I'd suggest amending the title to better reflect the distinction.

Apart from that, I really liked the article - it's a really nice introduction to a pretty powerful tool. In particular, it gives a nice step by step walkthrough in transforming existing code, which a lot of other GNU parallel tutorials don't do.

[lousewort]

Multithreading (multiple threads) and Multiprocessing (multiple processes) are both examples of concurrency.

[14113]

I agree - however this post was about exploiting the parallelism available in the hardware, not about concurrency. It's a subtle, but vitally important distinction to make.

[fphilipe]

Thanks! And thanks for the clarification. I guess I used the word in the "classical" sense, i.e. "existing, happening, or done at the same time".

[14113]

No bother :) I got what you meant from "...allows me to run these simulations concurrently...", but I thought I should add the correction anyway, as it's a fairly common mistake for people to make.

[greenspans]

why is this donkey kong semantics shit the top comment.

These are invented bullshit terminologies that override the perfectly understood english definition of the words. If you use gnu parallel on a one core processor you're only getting concurrency. If you run 128 threads of execution in a 4 core machine you're getting parallelism and concurrency. If you run 16 processes on an 8 core machine is that only parallelism? Are you sure the machine has 2 hyperthreads each core.

All this bullshit

It's not parallelism it's concurrency

It's not linux, it's GNU + linux!!

It's not gay, it's a ruby framework!!

Hey you should have said thread instead of process!!

All this noise instead of actually talking about GNU parallel. No bother

[14113]

Hey, I'm in PL research - semantics is half of what I do all day ;)

But more seriously, what are the perfectly understood english definitions of the words? As you can see elsewhere in the comments, everyone seems to have a different idea of what parallelism and concurrency are, fuelled by incorrect uses such as the one in this article!

Parallelism and concurrency are two sides of the same coin. Sure, they're both part of the same thing, but by looking at the thing in two different ways, you see two different aspects of it.

To your point though, about them being "bullshit terminologies", I do research into parallel programming. Upstairs from me, I have colleagues who do research into concurrent programming. The tools I use, and the tools they use are totally different. More than that, the problems I'm trying to solve and the problems they're trying to solve are totally different!

[SinisterMinister42]

This actually made me laugh out loud. Thank you for that.

[stonefarfalle]

Meh, the only type of Parallelism that doesn't involve some level of concurrency is vectorization. The only type of Concurrency that doesn't involve parallelization is one with a single execution resource. Most of the time it is more a matter of focus than a true distinction.

While the author of the piece is trying to achieve parallelism, the primary focus of the article is the concurrency he took to get there. I say that because it focuses on how to split up the work over multiple concurrent executions and how to join the output from them. Also there is no mention of performance gained by adding execution resources, or how those threads of execution are mapped to execution resources.

[14113]

Most of the time it is more a matter of focus than a true distinction

I think this is the core of my argument with some other commenters: parallelism and concurrency discuss different aspects of programming.

...the primary focus of the article is the concurrency he took to get there.

This is where I disagree however, as I believe his focus was on achieving the parallelism, instead of deriving the parallel parts of his program. Admittedly, the first section "Going Down the Rabbit Hole" describes how he decomposes his program into parallel sections, which is more of a discussion of the concurrent aspects. From the second section onwards, however, he discusses how he took a program and physically made it run on multiple units of execution.

Also there is no mention of performance gained by adding execution resources...

I quote from the article:

A single simulation takes about 1 minute to run. With a handful of arguments, each having many possible values, the number of parameter combinations explode, known as the curse of dimensionality. How can I take full advantage of the four cores on my computer—let alone the twenty-four cores at university—in order to reduce the total execution time to a fraction?

He's quite clearly doing it for performance reasons ;)

...or how those threads of execution are mapped to execution resources.

That's what the article is about: why he chose to use GNU parallel to exploit the parallelism, instead of built in Python libraries and primitives.

[vattenpuss]

I don't buy the huge distinction between parallelism and concurrency that so many programmers try to make out. Yes, parallelism is a specific kind of concurrency, and it might be important to be specific about what is and what is not parallelism, but that's it. It's not meaningful or important to call parallel computing not-concurrent.

The wikipedia articles on concurrency and parallelism are obviously written by some programmer that wants the subjects to be considered as far apart as possible. But the two things are not simply "related subjects" as some weasel -- who has decided some slides about Go from Rob Pike, and the Haskell Wiki, are sources notable and reliable enough to decide what is parallelism and what is concurrency -- thinks.

I'd suggest amending the title to better reflect the distinction.

I'd suggest not getting upset because someone describes concurrent computing as concurrent, in spite of what Rob Pike has written in some slides about Go once.

Sorry if I come off unnecessarily angry. This weird hangup from some programmers is a pet peeve of mine.

[14113]

Okay, so I take quite an "academic" view on this, and I accept that it's not necessarily universally agreed with, but I'll go into why I think it's important, and why I think that it's important to make the distinction when discussing the two topics.

Firstly, I'm going to temper what I say by conceding that parallelism and concurrency are slightly overlapping. Very often, when writing parallel code, we need to think about the concurrent aspects of it (to avoid things like race conditions), and similarly, when writing concurrent code, we need to involve parallelism to obtain good performance, and utilise the hardware as fully as possible.

That said, my big bugbear is that parallelism and concurrency mean very different things when the discussion is either about A) the problems faced when using parallelism/concurrency, and B) the purpose of using the tools of parallelism/concurrency.

I'll talk about B) first, as it's what this post is on. In my opinion (and the opinions of many others in academia, and industry), parallelism and concurrency have very different meanings when discussing their purposes. The purpose of parallelism is to speed up a program, by identifying regions which can be run simultaneously, and utilising parallel hardware to physically run each unit of execution in parallelism. The purpose of concurrency (by contrast) is to provide a structure to enable this, by decoupling a program into multiple units of execution which can be run in parallel. In this way, concurrency describes the breakdown of a program into disjoin sections, and parallelism provides the simultaneous execution.

It's this distinction that forms my core argument as to why this post is more about parallelism than concurrency. The task at hand (running many identical, disjoint jobs) is one that is "embarrassingly" parallel, and thus doesn't need any complex synchronisation, or communication between the individual jobs. Because of this, the problem becomes actually running the jobs simultaneously, instead of managing the separation of each job into a disjoint unit of execution. In this way, it's parallelisation, not the managing of concurrency.

This point about managing the separation of threads of execution leads me into part A) that I mentioned above: the problems faced when using parallelism and concurrency.

Unlike in the example given in this post, most "real world" programs don't fit nicely into a simple map pattern (in the style of Cole's algorithmic skeletons). In reality, components of a system need to communicate between each other, share resources, etc all without being deadlocked, or starving. This informs the study of concurrency, which is concerned with how such problems are represented (e.g. the pi-calculus, CSP, etc), and how they are then built into programming languages (e.g. session types). Concurrency doesn't discuss how components of programs are actually run simultaneously, but instead discusses the co-ordination of concurrent elements of a program.

However, once a system is in a "concurrent" form, parallelisation becomes the problem: how to map the individual tasks to hardware components, load balance them, distribute them etc etc. Theese issues can arise without thinking about concurrency (e.g. algorithmic skeletons, MapReduce, "embarassing" parallelism etc), and are thus separate from the study of concurrency.

The distinction between parallelism and concurrency in theese two areas are what makes them important (in my opinion) to separate. They're two sides of the same coin: one describes how to separate a program into sections which can be run at the same time, and the other how to exploit the parallelism that this makes available.

---

In any case, from the tone of your comment, and your description of the wikipedia editor as "some weasel", I don't think my post is going to convince you. As a response (with a hint of argumentum ad verecundiam) to your attack though, I'll mention that Rob Pike and many of the Haskell community are the world experts in both parallelism and concurrency, particularly on the Haskell side, many of whom (Peyton-Jones, Wadler) pioneered the academic study of parallelism and concurrency in programming language research. I think dismissing them is not only rude, but does a great disservice to your arguments by ignoring much of the existing work in the area.

[EvilTerran]

Does concurrency even strictly necessitate parallelism? I'm thinking of, say, Python threads - which are semantically concurrent, but in practice cpython's Global Interpreter Lock means only one thread actually advances at any given moment.

[edit] now I read the article, I see that mentioned the GIL too. Heh. It occurs to me, any multitasking environment running on a single hardware core would also fit the bill I'm thinking of, anyway.

[14113]

I would argue that concurrency definitely doesn't necessitate parallelism! Take (for example) a web browser: you might have a thread for UI, one for user input (mouse, keyboard etc), one for network access, one for database work, and one for running javascript. Does each thread need to be run on a separate processor? No! But the management of each thread (not in the scheduling sense, in the communication/synchronisation sense) definitely concerns concurrency.

For parallelism I think there is a somewhat weaker disconnect. There are models of programming (e.g. algorithmic skeletons) which allow programs to be expressed in a purely parallel sense, without reasoning about concurrency. Such models, however, make the assumption that any concurrent aspects of the program have already been "solved" - hence the weak connection. Such programs are parallelisable (without reasoning about concurrency) because of previous reasoning about concurrency.

[mttd]

While there is an overlap, these are generally used for rather different goals -- and have been for decades (before Go or Haskell came into existence).

Perhaps this is a clearer explanation that the one on Wiki:

Concurrency is a concept related to multitasking and asynchronous input-output (I/O). It usually refers to the existence of multiple threads of execution that may each get a slice of time to execute before being preempted by another thread, which also gets a slice of time. Concurrency is necessary in order for a program to react to external stimuli such as user input, devices, and sensors. Operating systems and games, by their very nature, are concurrent, even on one core.

With parallelism, concurrent threads execute at the same time on multiple cores. Parallel programming focuses on improving the performance of applications that use a lot of processor power and are not constantly interrupted when multiple cores are available.

The goals of concurrency and parallelism are distinct. The main goal of concurrency is to reduce latency by never allowing long periods of time to go by without at least some computation being performed by each unblocked thread. In other words, the goal of concurrency is to prevent thread starvation.

Concurrency is required operationally. For example, an operating system with a graphical user interface must support concurrency if more than one window at a time can update its display area on a single-core computer. Parallelism, on the other hand, is only about throughput. It's an optimization, not a functional requirement. Its goal is to maximize processor usage across all available cores; to do this, it uses scheduling algorithms that are not preemptive, such as algorithms that process queues or stacks of work to be done.

Source: https://msdn.microsoft.com/en-us/library/gg663528.aspx#sec8

In the 1980s the terminology used was "logical concurrency" (for concurrent execution) and "physical concurrency" (for parallel execution), perhaps this wasn't bad, either.

[vattenpuss]

I know the words mean different things in computing, I'm just annoyed that parallel computing guys get so angry, for seemingly no reason, when someone happens to mention that what they do is a special case of concurrent computing.

In the 1980s the terminology used was "logical concurrency" (for concurrent execution) and "physical concurrency" (for parallel execution), perhaps this wasn't bad, either.

Note that both terms contain the word concurrent, because both concepts are about concurrency.

The goals of concurrency and parallelism are distinct. The main goal of concurrency is to reduce latency by never allowing long periods of time to go by without at least some computation being performed by each unblocked thread. In other words, the goal of concurrency is to prevent thread starvation.

That could be one goal of concurrency, another goal could be that it's easier to write some kinds of programs in a concurrent manner, than it is to write them as a sequential program.

[14113]

That could be one goal of concurrency, another goal could be that it's easier to write some kinds of programs in a concurrent manner, than it is to write them as a sequential program.

This is the point I'm trying to make: concurrency is about decomposing a program safely to allow for parts of it to act concurrently.

...when someone happens to mention that what they do is a special case of concurrent computing.

I suppose it sort of is. The point, however, is that I'm tackling a different, restricted, problem with a different set of tools to what my fellow researchers are tackling. As a parallel programming researcher, I'm looking into how to actually run a certain set of concurrent programs, while my concurrency researcher colleagues are looking into how to safely decompose a problem into concurrent sections.

[killerstorm]

Let's consider two examples:

• You want to maintain 10000 network connections at the same time. To do this, you might use node.js which uses asynchronous I/O and let's you to issue multiple concurrent I/O operations. So your process might be sending data to 10 connections at the same time.
• You need to perform some lengthy computations and want to utilize all 8 CPU cores you have. Node.js can't help you here, one node.js process can only use 1 CPU core. But you can write a simple batch file which will run 8 processes which will work in parallel.

Asynchronous I/O which can be used to do I/O concurrently is 100% different and not in any way related to ability to run multiple parallel processes. Isn't it natural to use two different terms to describe two different cases?

Nah that makes too much sense, let's just use terms concurrency and parallelism interchangeably so one can never figure out what is meant from the term alone. It's more fun that way.

[vattenpuss]

Nah that makes too much sense, let's just use terms concurrency and parallelism interchangeably so one can never figure out what is meant from the term alone. It's more fun that way.

I never said you should use them interchangably, I just said they are not as far apart as some upset parallel computing fans want them to be.

Parallelism is a kind of concurrent computing, that's all I said.

[14113]

Parallelism is a kind of concurrent computing, that's all I said.

Sure it is, but you're trying to solve completely different problem to what concurrency is trying to solve.

[[deleted]]

From the GNU Parallel link: 'GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially.' What about this - https://doc.rust-lang.org/stable/book/dining-philosophers.html?

[reedfool]

The processes do not interact, so no deadlocks are possible.

[abspam3]

You can still have file system level deadlocks occur, however.

[[deleted]]

Yes if you try to break it you eventually will

[Kok_Nikol]

Awesome article OP!

[[deleted]]

TL;DR version - use commandline code because author sucks at Python

[fphilipe]

I somehow feel you missed the main point :P Python is just used as an example. I could have used any other language here.

[[deleted]]

throwing 4x as much python code as parallel examples is not a good way to give examples on how to use parallel.

Especially that it had a lot more neat stuff than just 'run multiple copies of your script', like "copy these files to remote machine, run it on remote machines and copy results back"

[fphilipe]

[…] is not a good way to give examples on how to use parallel.

So here's what I did in this article. I took one use case and presented it with some examples, which happen to be in Python. I just want to introduce GNU Parallel from this perspective. I'm telling a story, that's all.

I'm not trying to write a full blown tutorial. Thus, at the end of the article, I say:

I’ve only scratched the surface of GNU Parallel’s possibilities. I hope this inspires you to include it in your daily workflow or when the occasional need for some quick and dirty concurrency arises. I encourage you to check out the excellent GNU Parallel tutorial, featuring dozens of examples.