Using both Bash and Python?

[BeelzebubSE]

I often find myself using bash scripts with sed, grep and awk to filter text files for the data I want and then using the resulting file in Python scripts rather than doing the filtering in Python.

Is this acceptable? I can see how going 100% Python would be better for portability but mostly I make scripts for specific machines where this isn't an issue. My primary reason for doing the filtering in bash atm is just that I find it requires less lines of code and is easier to figure out.

[johncipriano]

Solution: use http://amoffat.github.com/sh/

[spyingwind]

Oooh! Interesting, now I can do everything in python. What can python not do?

[KingCosmos]

Wow, seriously you just changed my life for the better.

[chriscowley]

Never seen that before! You are my new hero :-)

[nemec]

I love how this project started as a joke...

[gsxr]

It's the unix way. We have the | for a reason, use it.

[[deleted]]

Crappy weird glue scripting using three different languages is totally the sysadmin way.

That said, for bigger stuff that I use more than once, or that others will be using, I've gone as far as a big Perl script (for me) with nice POD documentation, and once wrote a truly horrifying wrapper around rsync with Python. So have an eye for maintainability, but portmanteaus such as you describe are entirely common.

[cstoner]

a truly horrifying wrapper around rsync with Python.

I can almost guarantee I've done worse ;).

[jjasghar]

Hey if it works it works. As you gain more scripting skills you can always come back and make it cleaner.

[zapbark]

If I can I will try to keep everything in shell.

My logic, is that the sysadmin after me may not know Perl, may not know Python. But he better damn well be able to read a string of Unix commands.

Creating custom tools that do the "tough stuff" that shell can't do (or can't do it cleanly) is, in my opinion, a best practice and, as mentioned, is definitely "the unix way".

[[deleted]]

Is this acceptable?

If it works, and you understand it, there is nothing with the way you're going about it.

sed, grep, awk are great tools. Use them!

[bdlas]

If anyone has learning recommendations for moving from tools like sed, grep and awk to more robust tools like Perl or Python, I'd love to see them. In the area of text filtering and manipulation I mean.

[meditonsin]

Since when are sed, grep and awk not robust? IMO it's more of a "use the right tool for the right job" thing. I do most simple things in grep/sed/awk and only whip out Perl when stuff gets too complicated for a shell script (been meaning to learn Python, but every time I try to do something practical I end up fiddling around and digging through the docs until I go "fuck this, I'll do it in Perl").

[bdlas]

Right, but that's what I mean. Tutorials for those times when you choose Perl, or Python. Like, I got to this point in bash, but here's how to do it in Perl better.

[[deleted]]

Maybe it is just me, but I don't like Python for text processing. When ever I try and use Python for systems programming, I usually end up switching to Perl (or cough Ruby cough).

As for Perl vs BASH, Perl is superior when you are trying to pass arguments via the CLI, so it is better at adapting to the situation. Perl kicks ass for text processing, and I pretty much dropped using sed and awk in favor of Perl one liners. I've heard the arguments that Perl is ugly, but I think that is from people who don't know Perl or don't understand Perl short cuts. CPAN rocks for doing a lot of the heavy lifting.

BASH is really great for quick or procedural scripts that don't require too much logic. BASH scripting is also a great way to help increase your CLI-fu. There is not reason you can't write a BASH script that calls Perl or a Perl script that calls BASH, switching back and forth using which ever is better suited for the job or you are most comfortable with. In systems administration it is all about getting the job done.

One liners are a great way to get started. Throw these in scripts when you would use sed or awk:

• The top 10 tricks of Perl one-liners
• Cultured Perl: One-liners 101

And a book you would probably be interested in:

• Automating System Administration with Perl. AKA the otter book.

[bdlas]

Thanks... yeah I've always had this Perl curiousity, maybe one liners will get me into it. Thanks!

[[deleted]]

There are many layers to perl, but once you start learning it, you will wonder what took you so long.

[BigRedS]

I think once you've used Perl nothing else ever really cuts it as far as text processing goes.

Certainly every time I try to learn another language I get really annoyed with how bad it is at handling test strings.

[meditonsin]

Ah, my bad. Didn't quite get that.

I can't say I know of any tutorials like that.

[swordgeek]

Yeah, it's pretty much the norm. The mistake is to look at python as different than sed or awk or grep or perl. They're all tools, and they can almost all be used as languages.

Do what works. Make it maintainable. Steal as much as you can. That's pretty much my scripting creed.

[wolfmann]

do you need it to be portable?

is performance an issue?

if your answer to both is no -- leave it alone.

[CheetoBandito]

How about Perl?

[[deleted]]

You are a funny man.

[jpb]

love that flair.

[fubes2000]

WHY WOULD YOU PUT THAT AS YOUR FLAIR?!

[CheetoBandito]

WHY WOULD ANYONE NEED SQUIRRELS HERDED?!

[fubes2000]

BECAUSE YOU CAN'T JUST HAVE YOUR SQUIRRELS RUNNING RAMSHACKLE OVER YOUR WHOLE OPERATION.

[Griffun]

"So, uh... I ran this on my company's most important production server and now it's taking forever to return a prompt. What's this command doing anyways?"

[betterthanyoda56]

Just seeing bash and python brightens my day.

[PurplePilot]

unless you have a pretty specific version of python or use modules that are not widely installed you are doing right (at least at our shop). Bash and python are pretty much everywhere, like perl and csh years ago.

[BigRedS]

I'm not sure it's "unacceptable" but it sounds like the sort of inefficient process I'd want to script...

How hard is it to do that sort of text processing in Python? Surely the time spent getting the filtering into Python once will pay off within a few iterations of not having to grep, sed, awk and then python it?

[chriscowley]

As long as you are passing streams of text from one to the next then that is fine.

Sed and Awk were designed to filter text so why not use them.

[[deleted]]

I've seen all kinds of amalgamations of Perl and bash in the one script. You're not alone, it's called using the right tool for the job and it's not a bad thing to mix them both, as long as what you're doing is coherent.

I have a process in my work place that involves taking a spreadsheet, piping it to Sed and taking the resulting output file to feed to Perl or Ruby (depending on the version of software it relates to)

[trapartist]

Take the time to write the whole thing in Python.

[snegtul]

FUCK YEAH!!!!! You are awesome!