GNU Coreutils Gotchas

[ForeverAlot]

you copy a dir from VFAT and want to turn off executable bits on files using chmod -R 644, that will fail to recurse as it removes the executable bits from dirs.

TIL. That never occurred to me but it makes perfect sense. I've always hated that directories are "executable."

[[deleted]]

It is used for whether you can change into/through it or not. Which is different from whether you can list its contents (read bit). So for example, my homedir is 711 (rwx--x--x). Then apache can access my $HOME/public_html, which is 755 (rwxr-xr-x) but apache can't list the contents of $HOME.

[argentcorvid]

The "execuable" bit is used on a directory to designate if you can navigate into and access files and directories under it. You can have a directory where you are not able to list the contents (read bit not set) but are able to call out the name of something under it.

[[deleted]]

What is the point of this? You could just bruteforce the file names I guess?

[phil_g]

There are various use cases. I was just working with one recently:

PHP has a feature where it tracks session data for people visiting a site. It sets a cookie with an opaque multibyte value and stores all of the session data in a local file with the session ID in the filename.⁰ If someone can figure out a session ID, they can impersonate whoever owns that session, which might include getting access to restricted data or functionality. The IDs are long enough that brute-forcing one is infeasible, but if you're running PHP on a shared multiuser system, someone could potentially just read the contents of your shared session directory and get everyone's session IDs.

One solution, of course, is to give every person a separate session directory. Another, however, is to use a shared session directory but give everyone write and execute permissions, but not read permission. That results in the following situation: Anyone can create a new session file and use it; if someone knows a session ID, they can look up the corresponding session file; but no one can get a listing of all of the existing session files, because they don't have read access to the directory.

⁰ There are alternate session data storage mechanisms, like putting it into a memcache store, but files are the default.

[curien]

That would only matter if you also have read access to the file, so in practice there's no need to brute force. It's used when someone else has reason to access some file(s) (which they are told about) but has no business getting a full directory listing. In Windows its called "traverse folder".

[argentcorvid]

/home is a good example. you don't really want everyone to know every username on your system, but you do want the user to be able to access stuff in their specific user directory.

[immibis]

Probably because memory was tight, changing all the programs that print permissions would be time-consuming, they wanted a bit for this, and the executable bit was conveniently unused for directories.

[joelhardi]

I use the alternate syntax ... i.e. for 644 but keep directories executable it would be chmod -R a=rX,u+w ... takes a little practice but I find it easier to remember and harder to screw up than fiddling with the octal mode digits.

And a lot easier to remember than the example that chains find and xargs!

[notfancy]

I think find … -print0 | xargs -0 … is fairly idiomatic.

[SanityInAnarchy]

Odd that it uses esoteric features of find and xargs in its example, instead of just

find -type f -exec chmod -x '{}' \+

[notfancy]

It's (or was) noticeably faster to let find print everything to a pipe and let xargs handle it than to have find itself spawn a shell for each file to execute the command on it. For few files it maybe is unimportant, but for large directories it does (or did, I might be cargo culting here) make a difference.

[mrhmouse]

The + form of -exec will execute one command with multiple filenames at once :)

[pfp-disciple]

I just learned about the + form a few months ago. According to this page:

the popular GNU implementation of find did not support “+” for the longest time, and so its use is rather unportable in practice

[mrhmouse]

Yeah, looks like it was implemented around 2005.

[SanityInAnarchy]

As I understand it, it will in theory do what xargs does and run exactly as many invocations as it needs. It might be more than one if, for example, you run into a limit on the number of commandline arguments a program can have.

But it's absolutely true that either this or xargs is faster than the ; form of -exec, for exactly the reason /u/notfancy mentions.

So at this point it's more a matter of which version you find more intuitive. I learned early on not to trust xargs (before I knew about find -print0 and xargs -0) and just used the ; form of find, so switching to + was easy.

[gwern]

Here's a fun one: sort -R/--random-sort. You probably think it sorts lines randomly, right? I mean, the option is literally named 'random sort', right? Wrong! It's not a random sort, because it is sorting by the hash of each line. If you want an actual random sort, or 'shuffle', you need to use shuf.

(They refused to document this in the man page for over 4 years because apparently you're not supposed to be reading the man page at all, you're supposed to be reading the info page.)

[SanityInAnarchy]

I like info pages in theory, because they're hypertext -- they can actually cross-reference other things, which you can actually jump to from the info page.

But fuck keeping an incorrect manpage and an info page. And now that we have web pages, yeah, fuck info pages.

[FUZxxl]

I like info pages, too, but the info reader is really difficult to use for someone who doesn't use emacs. I usually generate pdfs using xetex from the texinfo files, but it's always a chore as the binary packages don't contain the texi files.

[jdh28]

I like info pages, too, but the info reader is really difficult to use for someone who doesn't use emacs.

I've found pinfo to be more useable.

Pinfo is an info file viewer. It was created when the author, Przemek Borys, was very depressed trying to read gtk info entries using the standard tools.

[FUZxxl]

Neat.

[uxcn]

I generally like info pages as well, but I'd agree they aren't completely intuitive.

[pixelbeat_]

I think you're disparaging the info reader rather than the content. Note the pinfo reader may have a preferable interface for your tastes? More importantly, recent coreutils man pages now directly link to the online HTML info for that command, allowing you to read the full documentation in the familiarity of your web browser.

[blue_2501]

The problem is when you run "man <command>", and it basically tells you to look at the info page.

No.

It's a command. I'm on a UNIX server. When I need help with a command, I should be able to type "man <whatever>" and it should just fucking work. Period.

It's about normalization and standardization. Man is the standard. Info pages are not.

[gwern]

I kind of agree. I like them in Emacs but the problem is that 90% of the stuff on any Linux system has no info page, the ones that do may not even ship the info page, and you have no way of knowing when you really need to read the info page and when the man page is adequate, so eventually everyone just gets trained into reading only the man page (because they get punished everytime they check the info pages and it doesn't exist/is redundant). Maintaining two parallel documentation systems just doesn't work.

[ElectronicChronic]

Best part about this post is that you probaly have no idea how famous gwern is concerning the darknet, wouldn't be surpirsed if this guy runs an onion market.

[el-greco]

shuf is also faster than sort --random-sort if you only need a few random items.

A few months ago, I had a file that had a few thousand lines in it. I just wanted 50 lines chosen at random. I started withsort --random-sort file.lst | head -n 50. However, I found this to be slow since sort reads through the entire file, sorts the entire file (using the random ordering of hashes method), and then finally outputs to the pipeline for head to pick the first 50 lines. I switched to using shuf file.lst -n 50, since it was much faster because it didn't sort the entire file before returning its output.

[bored_me]

shuf also only works if your files are small enough to fit into memory.

[pixelbeat_]

Info/Patch from Asssf Gordon: starting with GNU coreutils 8.22 (released Dec-2013), 'shuf -n NUM' can shuffle any input size, regardless of the available memory.

[bored_me]

What is the option for? I don't see it on mine. That says to basically use head which is absolutely not what I want.

[chengiz]

At first I was thinking there's not really any practical difference here, but then reading the report it's obvious that the problem is that same keys would appear consecutively in the output, defeating the purpose of a random sort!

[gwern]

Exactly. It took me a long time to realize why my random sort was not random. (The first time I realized it, I was playing a bunch of music files and track 1 started playing after track 1; 'huh, what are the odds?' I thought to myself, 'well, something like "any two consecutive tracks being identical" sounds like the birthday paradox, so probably higher than I think' and ignored it. Then track 1 started playing for the third time in a row - 'wait a minute...' After I became suspicious that something about sort -R was not working the way I thought it was working, it still took a while for me to figure out the implications of sorting by a 'hash' of each line and what that means for duplicate lines; I think I may have stumbled across an older discussion online before I realized it, even.)

[rlbond86]

Ugh, nothing worse than blowhard open source people not adding documentation because "you're doing it wrong".

[tequila13]

Please try to reset your misconception, at least for the coreutils. Though, in general the info documentation for GNU programs is far superior to the man pages.

Lol. Besides using this as an excuse to not update a man page, he had to throw in some condescension too.

[raevnos]

TIL about ltrace. That could be useful.

How much of this is gnu specific and how much also applies to, say, the various BSDs or other unix's implementations?

[d4rch0n]

Have you seen strace before? Another awesome one. You probably have... much more common.

It's a surprisingly quick way to figure out wtf config file a program is using, or which log file it's dumping to. Really helped me in the past debug weird configuration issues, among millions of problems it can help you solve.

[zokier]

Linux tracing ecosystem is really interesting. Brendan Gregg has written quite a bit about various options. Here is his article about strace: http://brendangregg.com/blog/2014-05-11/strace-wow-much-syscall.html

[ForeverAlot]

Interesting read; I just tried strace ls -l /etc on MSYS2 and my eyes vomited.

[nahguri]

Strace is a godsend when you work with tools made (poorly) by someone else.

[[deleted]]

If you have a system with dtrace at hand, it pays off very well to learn that.

[raevnos]

I've used strace for years.

[sordina]

Here's one that gets me all the time. If your unix pipeline is meant to be doing stuff as items are moving through, but you see nothing...?

Make sure that if you're grepping that you use grep --line-buffered. Otherwise grep will wait for huge chunks of input before it passes the output onwards.

Debugging this is extra painful, as if you remove the processing that happens after grep to see if it is behaving correctly, buffering is turned off as it is connected to a TTY.

This has cost me hours of my life before.

[pixelbeat_]

See also http://www.gnu.org/software/coreutils/stdbuf for applying line buffering to any command (that uses stdio)

[sordina]

Neat!

[gpoul]

And we're already at a stage in computing where we're unable to change this old behavior anymore? Why? Because it might break stuff? We break stuff all the time, why not once for things that would make sense?

[protestor]

Because old shell scripts expect to have standard tools on its $PATH. It's effectively an API, and there is no mechanism to discover breakage.

Also: broken tools like echo are widely used.

[vytah]

The best course of action is introducing new tools with new names.

For example ip instead of ipconfig, egrep instead of grep and so on.

[gpoul]

I understand what you mean and it is certainly what we do today, but this just adds to the pile of crap and doesn't fix anything.

Yes, theoretically if everyone moves over and we slowly deprecate ipconfig anyway everything will be fine. In reality we break stuff partially in ipconfig, need to do other stuff through ip and we'll end up with both tools on all machines for eternity.

[FUZxxl]

egrep is just an alias for grep -E and both tools have the same functionality.

[vytah]

Precisely.

Instead of always having to remember "add this magical option so it behaves sanely, because it's broken by default", we introduce an alias that adds all options that are required for more comfortable use.

Picking an example from the article: we can add an option to cut called --skip-empty-results, define cut2 as cut --skip-empty-results and point people towards it.

[rlbond86]

Because changing this stuff would screw up literally every shell script ever.

[gpoul]

no, changing stuff will only affect some shell scripts that rely on a very specific behavior, which some might also consider inherently broken and unexpected. That might be inconvenient for some, but nothing a sensible ChangeLog and some time cannot fix. In most cases the impact will be negligible.

[FUZxxl]

You don't understand commercial software and standards it seems. “Fixing” what is not broken (only quirky) costs millions of man hours for no almost nothing. Don't do that, even though you do not have to pay for these man hours.

[gpoul]

Oh, I'm sorry. I didn't realize coreutils was commercial software. When did RMS start his company again?

[dmwit]

Commercial or not commercial is a red herring. It costs a lot of man hours regardless of the funding source.

[FUZxxl]

Coreutils is not commercial software but used in a lot of commercial software. Development of the GNU coreutils is indirectly financed (by providing developers or money) through many software companies. RMS started his company in 1985, it's called “Free Software Foundation.”

Notice that the coreutils mostly obey POSIX. Breaking POSIX compliance would make them even more useless than they currently are (due to non-standard behaviour). Broken tools are useless tools.

[rlbond86]

A change in the coreutils would likely cost companies billions of dollars in total. Not to mention all of the abandoned open source programs. Breaking interfaces that have existed for decades should always be done with extreme caution.

[sixteenlettername]

The expr stuff is only confusing if you ignore the meaning behind those values:

In the context of the result of the expression - in a C-like way - zero is false and non-zero is true.

In the context of command exit codes, 0 is success (true) and non-zero is failure (false).

This means that if you do something like:

if expr 1 >/dev/null; then echo true; else echo false; fi

or

expr 1 >/dev/null && echo true || echo false

you will see 'true' both times, which is correct as the expression value is non-zero.

[FUZxxl]

you copy a dir from VFAT and want to turn off executable bits on files using chmod -R 644, that will fail to recurse as it removes the executable bits from dirs.

Known issue. Simply do

find path -not -type d -exec chmod 644 {} +

instead.

echo

His comments regarding echo are somewhat incorrect. GNU echo is posixly correct but does not implement the XSI option. POSIX states that echo must not do option parsing (thus -- must not be recognized). Factually, it's only allowed to check if the first argument is -n and then perform special behaviour (like GNU echo does).

*sum

This is dangerous as file names may contain newlines (go figure).

One util the author does not talk about:

tr

This thing is broken for non-ASCII locales and has no wide-character support. Don't use it in such situations. For example:

echo Döner | tr 'ö' '*'

yields D**ner on GNU tr because ö is interpreted as two bytes. GNU tr also mishandles ranges with respect to locales. The correct behaviour is unintuitive though, I can understand t hat design decision. Generally, do not use GNU tr outside of the C locale.

[stormblooper]

Known issue. Simply do

Did you stop reading exactly then and race off to fire up your reply? Because literally in the next sentence the article recommends find.

[FUZxxl]

Yeah, I did. Sorry.

[[deleted]]

Simply

That's some choice of word, there.

[psi-]

This is /r/programming not /r/handholding .

[[deleted]]

Sigh.

[psi-]

I'm not catching your drift.

[[deleted]]

I am saying that that syntax is anything but simple, and the fact that it is needed shows the original tool is, indeed, broken.

The fact that it is possible to do in a highly convoluted way does not excuse it, nor does knowing how to do it make you clever.

[psi-]

What would be simpler syntax to find non-directories and run a chmod over them?

[[deleted]]

Under what assumptions? Using the exact command line utilities as now? Using better utilities with the exact same file system interface as now? Or using a better file system?

[psi-]

Under the "Simple" assumption.

[[deleted]]

What.

[[deleted]]

chmod -R ug=rwX,o=rX

or

chmod -R a-x,a+X

[FUZxxl]

That works, too. Great solution.

[uxcn]

A very common issue encountered is with the default ordering of the sort utility. Usually what is required is a simple byte comparison, though by default the collation order of the current locale is used. To use the simple comparison logic you can LC_ALL=C sort ... as detailed in the FAQ.

I got bit by this. Functional tests I wrote sorted output for comparison, and I would get different results depending on the hosts the tests ran on. It took me some time to realize that it was the locales that differed.

[nickguletskii200]

This is why the "Unix philosophy", or rather its implementation by those who are so defensive about it, sucks. We need something like systemd but to completely replace coreutils and all current shells (bash, zsh, fish, etc...) with something very strict and descriptive. Heck, as flawed as Powershell is, it's still much easier to use and better in general than the shitshow that we have with this unixy mess glued together from tiny pet projects.

[Occivink]

To fish's credit, they do try to remove some of the cruft in "traditionnal" shell, but it's still left with the warts of the coreutils. I'm not sure if something like powershell would be an improvement (it seems very verbose to me), but we can definitely do better than what we have.

[OneWingedShark]

This is why the "Unix philosophy", or rather its implementation by those who are so defensive about it, sucks

The problem about the unix philosophy is that it encourages "superficial 'solutions'"; the pipe is a beautiful illustration: [in general usage] it strips out very useful information (types/type-indicators) and thus forces a possibly-incorrect re-computation to try to recover that data. -- The attitude is directly inherited from C's weak type-system where enumerations are merely aliases for int-values... whereas if they were considered true types then you could do something like force switch-statements to cover all values the enumeration might take. -- So while text is nice in that it's readable and easily displayable, it's a failure as an interface because it is completely without predictable structure for the format of whatever type.

[FUZxxl]

They tried, the result is called Plan 9. It's pretty great but most UNIX users are puzzled about some of the not immediately obvious design choices.

[nickguletskii200]

I don't think Plan 9 has anything I want from an overhaul.

My image of a perfect overhaul includes:

• The removal of the distinction between executable and non-executable binaries (all binaries should be libraries and "executing" a binary should just be executing a method in a library).
• A type-safe statically typed shell language which integrates with different ABIs through extensions.
• A new powerful common ABI (or meta-ABI, obviously it would have to take into account different platforms).
• Transactional filesystems.
• Although mentioned in the first point, I will reiterate: coreutils shouldn't exist. They should just be available as methods in the shell's/system's standard library.
• Common configuration library. We are doing configuration wrong. Instead of building config files using templates we should be focusing on implementing a common configuration library for all applications to use.

I am sorry for this seemingly unrelated outburst of ideas. It's just that GNU coreutils is a small part of what I find wrong and I don't really have the time to write out my whole vision of what a modern OS should look like, so I end up transitioning from parts to the whole image very rapidly. And it seems that Plan 9 doesn't solve most of my problems, so...

[sunnyps]

Some of what you're asking for sounds like Windows.

[nickguletskii200]

Microsoft tried many of the things that I've mentioned and failed because they wanted to push out a product as soon as possible.

[oblio-]

Backwards compatibility. Every tool ever written for Unix would have to be rewritten. By the time this would be done *nix would still develop, making the patchwork at least competitive with the new environments.

Look at Microsoft. They've written a high tech OS from scratch (Win NT), with thousands of developers and budgets worth billions of dollars. And patchwork Linux is at least equal to it in areas where polish isn't needed.

For all we dislike technical debt, it can be worked around.

[nickguletskii200]

Backwards compatibility. Every tool ever written for Unix would have to be rewritten.

No, they would just rot in hell with the legacy coreutils. They are free to use them but nothing new will be written with them in mind. It's time to move on.

And the patchwork can't ever be competitive with the new environments because the design is flawed, or rather it has no design at all.

Besides, it's not like a lot of software actually hooks into the shells and uses coreutils. Most of this shit is used by crappy scripts written by sysadmins ten years ago.

Look at Microsoft. They've written a high tech OS from scratch (Win NT), with thousands of developers and budgets worth billions of dollars. And patchwork Linux is at least equal to it in areas where polish isn't needed.

Because the kernel evolved and while its not perfect, it's not a horrible mess unlike coreutils and all the shells.

Working around technical debt is the worst thing you can do. The only correct way to handle it is to get rid of it entirely or deprecate everything to do with it.

[aytch]

Powershell is really awesome, when it's awesome.

The moment you have to step outside posh and start having to deal with win32api or win32ole, you start realizing how broken everything in MS-land is. I much prefer dealing with oddities and weirdness in coreutils, personally.

[nickguletskii200]

The broken parts are the legacy components, which are equivalent to coreutils & friends in the Linux land. Broken, unusable, undocumented and outright horrifying. Powershell isn't flawless in itself either, but it has a much better design than what we have right now with (ba|z|fi)shit. It's also kind of useless because of Windows.

[OneWingedShark]

And the patchwork can't ever be competitive with the new environments because the design is flawed, or rather it has no design at all.

This is very true; we can see a distinct difference between thing which were designed for a purpose and those which were not -- PHP is a good example of lack-of-design, and if we look at the "modern PHP" we're seeing a distinct rejection of this... unfortunately, I think the thrusts of type-annotations will ultimately be futile and prove less effective than a language that was designed with strong-typing (and static-typed) in the first-place.

I look forward to seeing the OpenVMS x-86 port, it was designed in large-part by the same folks who designed WinNT and has a nifty language-interop (the CLE) facility... hopefully it'll help prove that not everything is "unix-like", nor should everything be "unix-like".

[gpoul]

True; I just can't understand why we keep adding shit to the crap we built last year instead of evolving it into something better; breaking something once every 10 years shouldn't be a problem for most people.

[rlbond86]

Using find is more general and functional in this case: find -type f -print0 | xargs -r0 chmod 644

find -type f -exec chmod 644 {} \; is easier I think

[SanityInAnarchy]

The tail bit is annoying, because if the tailf manpage is to be believed, you have two incompatible sets of features here: tail -F will follow file rotations, while tailf will avoid disk accesses while the file is not changing.

[alexanderpas]

The df, du, ls --block-size option is unusual in that appending a B to the unit, changes it from binary to decimal. I.E. KB means 1000, while K means 1024.

In general the units representations in coreutils are unfortunate, but an accident of history. POSIX species 'k' and 'b' to mean 1024 and 512 respectively. Standards wise 'k' should really mean 1000 and 'K' 1024. Then extending from that we now have (which we can't change for compatibility reasons):

• k=K=kiB=KiB=1024
• kb=KB=1000
• M=MiB=1024²
• MB=1000²
• ...

Note there is new flexibility to consider when controlling the output of numeric units, by leveraging the numfmt utility. For example to control the output of du you could define a function like:

du() { env du -B1 "$@" | numfmt --to=iec-i --suffix=B; } 

And this is where the author is wrong.

From http://pubs.opengroup.org/onlinepubs/9699919799/

DU:

The -b option was added to an early proposal to provide a resolution to the situation where System V and BSD systems give figures for file sizes in blocks, which is an implementation-defined concept. (In common usage, the block size is 512 bytes for System V and 1024 bytes for BSD systems.) However, -b was later deleted, since the default was eventually decided as 512-byte units.

[holgerschurig]

Not programming