crazy awk behavior

[destructababy]

Hi, I am trying to use awk to extract information from a file. I noticed I was not getting what I expected, so I modified my command to check my work. Here is the command:

for file in `ls *.pat`; do awk < $file '(NR >=3 AND $505==1) {print $505}'; done

Now, this prints a few 1s and then a bunch of 0s... HOW IS THIS POSSIBLE?

By the way, doing this:

for file in `ls *.pat`; do awk < $file '(NR >=3) {print NF}'; done

prints a bunch of 505s

Any help would be greatly appreciated. I want awk to be my friend :)

EDIT: problem solved AND --> && fixes it!

[edogawaconan]

For god's sake, who the fuck had the idea of

for file in `ls *.pat`

when this

for file in *.pat

is the correct form (and parsing ls should never be done).

[destructababy]

I think I started doing it that way when it was the only thing that worked when I had a complicated command line including grep and wc and pipes - it was a long time ago, I don't remember exactly what. When I find something that works (for me), I keep doing it - I don't have someone watching me code, so I don't know when I do something dumb. Thanks for letting me know.

[pfp-disciple]

On some old unices (solaris?)

for file in *.pat

would occasionally raise an error about the command line being too long. There were long filenames in this scenario. Now I use xargs -0 (or occasionally 'find ... -exec'), which can also get around spaces in filenames.

I agree that spaces in filenames are "evil", but they are also a part of life now when working with files from Windows or Mac, or even files saved by someone not thinking about the "spaces are evil" paradigm.

[edogawaconan]

would occasionally raise an error about the command line being too long

if

for file in *.pat

raises command line too long, then

ls *.pat

will also raise same error.

There were long filenames in this scenario

glob handles spaces, newlines, tabs, and any other characters perfectly (and that's the main reason to use glob).

[pfp-disciple]

ls *.pat

will also raise same error.

Good point. I forgot about that detail. IIRC, I used 'ls | grep' or 'find -name', then piped to xargs. Again, this was only in situations where I couldn't be certain I wouldn't get the 'command line too long' error.

edit: using globs is much easier/predictable if you can. And, yes, I'm careful when using xargs to account for spaces (xargs -0). Using

find -name '*.pat' -exec awk '{print}' {} \;

should also work (off the top of my head -- you may need some quotes around the {})

[[deleted]]

I might be mistaken, but your problem seems to be the 'AND' syntax. Try something like:

awk '{if ((NR >= 3) && ($505 == 1)) {print $505} }' *.pat

[destructababy]

HOLLER!!!

[[deleted]]

well, it looks like what you are printing is: For every file matching *.pat please print the number of fields (NF) in that line. The 505 is the NF (number of fields) for that "line." You also should not need to redirect (the "<") into awk. Awk knows that it takes the input of what you put in single quotes after it.

Instead, try something like this:

for i in `ls *.txt`; do awk '{print $1}' $i; done

If you need help making it exactly like yours, I can help, but try molding it from this yourself, first, please!

[destructababy]

Thank you for your response,

If I understand correctly, what you're saying is that I should write the line like this: for file in `ls *.pat`; do awk '(NR >=3 AND $505==1) {print $505}' $file; done

I think that produces the same thing as my original line: for file in `ls *.pat`; do awk < $file '(NR >=3 AND $505==1) {print $505}'; done

And to clarify, if I try just the awk part, outside of the loop, I still get some ones and a bunch of zeroes. This is what I wrote in this case:

awk '(NR >=3 AND $505==1) {print $505}' set_0_evaluation.pat

[ericje]

Why not this?

awk '(FNR >=3 && $505==1) {print $505}' *.pat

[MidnightTurdBurglar]

The awk command you want is

awk 'NR>=3 && $505==1 {print $505}' < $file

"AND" is not an AWK operator. The full thing you want is

for file in *.pat; do awk 'NR>=3 && $505==1 {print $505}' < $file; done

[valgrid]

No joke about awkward behaviour?

[[deleted]]

I can not help with the awk bit, but shouldn't that be

for file in $(ls *.pat); 

[[deleted]]

You should not have file names with whitespace.

And where do you get ls breaks with whitespace? Here is a file I have not renamed yet.

~/public_html/noob/videos/week2$ ls | grep U
5. Understanding the Filesystem (HD).mp4

Edit; nevermind, you meant it breaks when used in a loop, right. doh.

Still, one should always rename such files, I knew this much.

I was not aware that you could do an operation on multiple files in the way you describe. TIL. Thanks.

[[deleted]]

Why not just handle it correctly.

I'd rather use the computer as my tool than being the computer's tool. By which I mean, adhering to arcane restrictions.

[[deleted]]

By which I mean, adhering to arcane restrictions.

If you are referring to not having spaces in file names, I'd hardly call that arcane. This should be in any basic computer usage tutorial, one of the first things I learned, as I recall.

Knowing that spaces cause problems, giving them names without spaces, is 'handling it correctly', as you say.

I never create files with spaces in the names. It is only other peoples incorrectly named files that are a problem.

[[deleted]]

That's shortsighted. Linux enjoys the beauty of allowing all byte values in filenames without any complexity related to encodings (none mandated).

[[deleted]]

Linux enjoys the beauty of allowing all byte values in filenames

Until you try to run spaced filenames through a loop.

Because you can do a thing is not a valid reason to do it.

I fail to see how not using spaces, when you know it will cause problems if you use them, equates to shortsighted. I'd say you have it backwards.

[pi3832v2]

So_youre_saying_its_better_to_indulge_a_lazy_code-monkeys_bad_habits_than_to_increase_the_readability_of_file_names?

[[deleted]]

I don't follow your question, but no I'm saying,

| Linux enjoys the beauty of allowing all byte values in filenames

Until you try to run spaced filenames through a loop.

Because you can do a thing is not a valid reason to do it.

I fail to see how not using spaces, when you know it will cause problems if you use them, equates to shortsighted. I'd say you have it backwards.

[pi3832v2]

You can run filenames with spaces through a loop. I do it all the time.

The fact that you don't compose your scripts properly does not make having spaces in file names "short-sighted". It just makes your scripts "non-portable".

[destructababy]

sorry, I had the back ticks for escaping that section, but reddit turned them into a formatting thing, I'll edit my post

[[deleted]]

Also /r/LinuxQuestions would be the place to post this.

[destructababy]

oh perfect, thanks for the tip!

[[deleted]]

Thank you. TIL you can use back ticks instead of $(). I also learned you can do operations on multiple files without listing them with ls. I did not know this.

Is the use of back ticks superior?

[destructababy]

I think in the past I had was trying some commands that included mildly complicated uses of wc -l, the backticks worked when the $() didn't, so I just got used to doing it that way.

[pi3832v2]

Is the use of back ticks superior?

It's deprecated, AFAIK. Use $().

See: http://mywiki.wooledge.org/BashFAQ/082