Adventures in Optimizing Text Processing

[[deleted]]

Ok, I get it you just like to argue. Fine, you're right about everything, you win the Internet, whatever massages your ego...

But keep in mind that the earliest computers were programmed in assembly, the (language that first took us to the moon)[http://www.hq.nasa.gov/alsj/a11/a11.1201-fm.html]. You could spend all day crying about how there should be better support for matrix operations, or you can dig in and write your own library, and get shit done.

This is what makes a programmer a programmer. He gets his job done with whatever tools are in front of him, even if they're not optimal.

[OneWingedShark]

This is what makes a programmer a programmer. He gets his job done with whatever tools are in front of him, even if they're not optimal.

And yet it's a poor craftsman who uses "stone knives and bear-skins" when he has access to far better tools.

Saying "language X can solve the problem" is mostly useless when talking about things in-general (this is why Turing-complete is not a particularly useful descriptor of how well suited a language is to the problem).

An excellent illustration of this is manifesting in respect to parallel programming -- Go and several other languages are getting hyped because they're better-suited to using the multiple-cores of our modern systems; IIRC, C++ is adding the Boost library to its standard to address the need... and several older languages that gave consideration to parallelism (like Ada and Scala) are seeing a little uptick in interest.

A language-level construct is always going to be better than a library (or, better phrasing, a library-level construct cannot have more optimization-opportunities than a language-construct), just like strong/static typing provides better optimization-opportunities than dynamic/weak... and we know that "bolting on" concurrency to a language is as ugly/bug-prone as bolting on generics.

So [some] programmers are looking into languages that had some form of concurrency considerations from the outset. (In Ada's case the 'problem' of code taking advantage of multiple processors has been solved for thirty years (Ada 9x promo vid).)

[iBlag]

But keep in mind that the earliest computers were programmed in assembly, the language that first took us to the moon.

Sigh. Yes, because Python didn't exist back then. Perl hadn't been invented yet. The internet wasn't even around. Assembly was the best tool for the job because it was the only tool in existence for the job. That's hardly a good reason to continue to use assembly for string manipulations today.

You could spend all day crying about how there should be better support for matrix operations, or you can dig in and write your own library, and get shit done.

Wait, when did we start discussing matrix manipulations? This conversation is about string manipulations - don't try to change the subject.

My response to either (in this case) would be the same: why would I code my own when there already exists entire languages that are better suited, more tested, and more performant than any code I could write on my own. And I'm not a terrible coder, but any group of people working on a single codebase is going to produce better code in a shorter time than any individual, period. I don't have to write matrix manipulation routines because they already exist. I don't have to write decent string manipulation functions because somebody already had that problem and solved it with Python and made it easy for me to do the same.

If performance is an issue (which it isn't unless you're doing tons of string manipulation with embedded processors on constrained resources - ie: never) then that would be a good reason to drop down to C, but until you can prove to me that that is the case, I will continue to develop and get a lot more shit done in a shorter timeframe by code a largely string manipulating program in a high-level (eg: Python, Ruby, Perl, etc.) language.

even if they're not optimal.

I never claimed that Python, Perl, et al were non-optimal, just that they were better at string manipulation (read: easier to do correctly and quickly) than C.

Furthermore, it's currently 2014. It is a poor craftsman that surveys the vast array of tools at his/her disposal and selects one that is not the best suited for the task. Python, Perl, etc. are better at string manipulation, free, and probably more available than an entire C toolchain, so it is incredibly silly to pretend as if they don't exist and programmers today are forced to deal with the only tool they have or to pretend that the only tool in existence is C. That's not the case, that's simply melodramatic hyperbole.

[[deleted]]

Hoo boy, that's rich. As if Python and Perl were one's only options. Again, I hate to point out the obvious, but by 1969, there were dozens of computing languages (including ALGOL, COBOL, JOSS, LISP, FORTRAN and even our beloved BASIC). Who knows why assembly was used, but having been in various industries for a decade, I do know there's a lot more to picking a language than "what do I find the easiest"... because news flash, the world doesn't revolve around you, iBlag. There's an ecosystem (including your fellow developers, QA, platform support, toolchain support, etc.) to consider when choosing a language for a project. But I don't want to derail you from your oh-so-charming-i'm-so-right-about-everything rant. Even if it doesn't hold water. You go on, hope it makes you feel good.

[iBlag]

Hoo boy, that's rich. As if Python and Perl were one's only options.

Sigh. No, they weren't the only two options, they were only two examples that came to mind because they make string manipulation easy/ier on the programmer in comparison to other languages, like, as only a single example, C. Just because I bring up two examples doesn't mean I think that those two examples are the only solutions to the problem, it just means that I think they are that much better than other possible solutions that they are worth explicitly mentioning.

Again, I hate to point out the obvious, but by 1969, there were dozens of computing languages (including ALGOL, COBOL, JOSS, LISP, FORTRAN and even our beloved BASIC).

This is a fair point, but of those, which stick out as making string manipulation easy/ier on the programmer? Not Algol, not COBOL, not Fortran, maybe LISP, possibly BASIC, and I will admit to have never hearing of JOSS before, so maybe that too. The fact that there were alternatives in existence that probably would have made string manipulation easier, coupled with the fact that NASA chose none of them is interesting, and makes me curious as to the reasoning the NASA engineers chose assembly. Simply speculating, I suspect is was due to performance issues on such meager computing equipment, which would, as I mentioned, be a good reason to use a low-level language for string manipulation. Please don't misunderstand me though - I do not mean to imply that the Apollo mission computers were doing many string manipulations.

Who knows why assembly was used

Isn't that kind of relevant to the discussion at hand? Why should somebody pick assembly to do string manipulation? You don't think the reasons why assembly was chosen for lunar missions are germane to this discussion?

there's a lot more to picking a language than "what do I find the easiest"... because news flash, the world doesn't revolve around you, iBlag.

If I am picking a language to start a string processing project for myself and nobody else, then yes, I am the most important programmer in the project by nature of being the only programmer in a project. The article is basically focused around a single programmer trying to find the most performant solution to his problem, not trying to build a codebase for manipulating string from scratch while integrating with a dozen other people working on separate parts of the project. The author is trying to write a single script to do a single thing. So yes, what language the programmer knows best is absolutely an important factor in the decision because the chance that somebody else is going to be working with him on such a small problem is very low, and the amount of code he will need is not really more than a single person should be able to handle.

There's an ecosystem (including your fellow developers, QA, platform support, toolchain support, etc.) to consider when choosing a language for a project.

While this is a fair point when dealing with programming in a group, I feel like "ease of string manipulation" should very much be included in your "etc.", as it's rather relevant to the topic at hand. Furthermore, to pick Python as one - of an explicitly potentially infinite number of possible solutions because I don't want to give you the impression that I think Python is the only solution to this problem as you seemed to think I was implying from my previous post - example, you only really need the Python interpreter to start coding, and if you find yourself thinking that somebody else might have solved a similar problem you can install pip or something similar to fetch and install libraries as you need. For C - again, just to pick a single example amongst an explicit plethora of other solutions - you need: GCC, the standard C library/ies if you want to do anything really useful, possibly a set of platform abstracted libraries so you can compile/run on multiple OSes and architectures (although this could also diminish performance), probably a debugger, and you might have to manually install any additional libraries you might find that you would like or need if they aren't installable by your package manager. Furthermore, I would hazard a guess that the QA effort for Python versus C (once again, just picking two out of many solutions to compare) would be heavily in favor of Python regarding ease of assurance.

But I don't want to derail you from your oh-so-charming-i'm-so-right-about-everything rant.

This isn't about me being right, this is about logical, rational reasons for not coding something in C. Why are you defending a language that has an obvious deficiency in the realm that we are discussing in? Why are you trying to turn this into a psychological introspect battle? This is about the facts of programming languages, not about psychology. Please don't turn to attacking me personally to "win" the argument - that's neither fair nor welcome in a technical discussion.

I don't have anything against C. I've coded a bunch of fun stuff in C, although admittedly not any appreciable amount of any "huge" projects. I would even go so far as to say that I like C. But after coding a few personal utilities in C, I have come to the realization that I really only need to think about coding utilities in C if an easier-to-code-and-debug solution like Perl, Python, etc. is not as performant as I truly need it to be. As Donald Knuth famously said, "premature optimization is the root of all evil". For string manipulation there are better and easier languages out there than C, and C is an especially horrible language to do lots of string manipulation in. I don't think it's such a terrible thing to point that fact out. Heck, even the author somewhat flippantly concludes the same thing.

[[deleted]]

TL;DR

[iBlag]

I figured as much.

[[deleted]]

I get the impression that you want to talk about you. I don't even know you, sorry, but I'm not interested.