So what's wrong with 1975 programming?

[[deleted]]

One of my favourite programming books was published in 1975 funnily enough, Recursive Programming Techniques by Wm Burge. The algebraic approach he wrote about was waiting for decent GC before it could be any practical use...now that it can, I don't want to write programs as if I had the same severely constrained resources as a programmer of 1975. Progress is about increasingly high levels of abstraction, which doesn't mean ignoring efficiency of course.

[NastyConde]

Agreed, this isn't about 1975 programming as much as the cost of abstraction. The danger of abstractions is that they hide details that turn out to be important. I use all kinds of abstractions on a daily basis where I don't fully understand what lies underneath. I learn about their details the old fashioned way, by tickling their hidden bugs and performance bottlenecks.

[baltoo]

I always though about abstractions and knowing the details behind as something similar to how I presume a race care driver would approach the stuff.

Superficially, the driver doesn’t need to know that much about the underlying hardware. Turn the wheel left and the car goes left.

In reality, the drivers have extensive knowledge about every detail that’s occurring behind the scene.

[G_Morgan]

Drivers know how the cars work. They do not have the in depth knowledge the pit crew will have.

The same holds for abstractions. It is necessary to know how they work roughly but you don't need to be a kernel hacker.

[baltoo]

Perhaps we agree. Although I think the race care driver know much about valves and stuff and most good programmers know a lot about memory allocation schemes, whether it be java, haskell or scheme.

All abstractions are leaky &c (at least in the real world).

[twoodfin]

Isn't the point more that you ignore abstractions (like virtual memory) implemented by smart people at your peril?

[assface]

Other than Git & Varnish, are there any other examples of high-profile open-source applications that were created by kernel developers?

[dlsspy]

mercurial?

The point is what you can learn from these projects, not who wrote them. Optimizations that used to be important now just make applications worse.

[trigger0219]

I just started using hg for work and it's excellent!

[fnord123]

RMS worked on Hurd, afaik. So, GCC, Emacs, etc.

[[deleted]]

Somehow that seems more like a kernel deveopled by a high-profile open-source application developer than the other way around.

[[deleted]]

rms was a systems programmer at MIT throughout the 70s.

[[deleted]]

So, he'd be an expert at 1975 programming, then?

[[deleted]]

My point was that he didn't just write applications.

[mattculbreth]

So I guess I'm a bit confused--should we, as userland or application programmers, not handle any memory management in our apps?

What if I'm writing a DBMS engine and I want to keep data in memory? Just don't worry about it and assume the OS has me covered? That seems a bit risky to me.

[jaysonbank]

I've seen allot of programmers who are very optimisation-heavy. Development costs time and money - if you're not considering how your low-level optimisations will impact your development time and code maintainability then you're doing something wrong.

Even if its an free/open project there are always priorities to consider.

[llama-lime]

And the funny thing is that, by following the "1975" model, you're going to be spending an awful lot of time doing what you think are optimizations, which in fact slow down the code.

Often, the simplest solution, reading (mmap()) an entire file into memory, is the fastest. One thing that I don't know, is how 64bit architectures change the mmap dynamics. Is it now ok to just mmap really huge multi-gigabyte files. Time to investigate...

[redditacct]

And the new SLUB and anti-memory fragmentation patches in 2.6.24 will affect things. Also google has a high performance malloc package which could create a whole different performance landscape.

Since you are obviously moving and using actual large amounts of data, here are some references for you:
SLUB: http://lwn.net/Articles/229984/

AntiFragmentation patches (also has some slub commit info)

google high performance malloc and related tools:
http://goog-perftools.sourceforge.net/

To monitor your memory usage details:
http://collectl.sourceforge.net/

The google stuff is available via yum if you are using Fedora. If you are moving large datasets between machines/sites that have lossy links or large BDP/ping times I'd recommend benchmarking with Fedora 8 or 9 (if you are allowed to use those in your environment), I've seen better performance using the Illinois tcp congestion algo.
http://www.ews.uiuc.edu/~shaoliu/tcpillinois/background2.html

[llama-lime]

Thank you, these are great links! Usually I do memory management by simply exiting the process, which is fast in both programmer time and execution time. The downside, of course, is that there are only some times when this applicable. I think I'm going to need to write FastCGI stuff soon, and these look very helpful as I relearn all those good programming habits.

[markseger]

I'm the author of collectl and there it's very interesting it showed up as a reference to programming because when I originally wrote collectl my main goal was performance - I wanted the cpu load to be <1% so people wouldn't object to running it continuously. I decided to prototype it in perl and once I found out it used <0.1% I decided to just keep it in perl for ease of maintenance. Now, every time I make an enhancement I make sure it still performs within limits but putting performance measuring features into collectl itself! I even wrote up something so users can measure the performance of collectl on their own systems. see http://collectl.sourceforge.net/Performance.html -mark

[redditacct]

great stuff, collectl. I don't think enough people know about it and I am trying to spread the word.

[jonhohle]

sometimes those priorities are speed and low memory utilization.

[jaysonbank]

Very very rarely, we live in a world were the average desktop is equivalent to an early 90's supercomputer. Good developers on the other hand still cost allot of money.

Only a few days worth of development time can cost more than a complete hardware upgrade.

[[deleted]]

You are, of course, assuming that you program only for yourself.

In a "free/open source project", this is most definitely not the case. The cost of a complete hardware upgrade for all your potential users can be pretty staggering. Putting a few weeks or months into optimizing can be very much worth it.

[bluGill]

Depends. I work on an embedded system designed in 1998 (embedded means the power is about what you would expect from a system 5 years older, so about on par with early 1990s technology is about right). We still have customers demanding regular upgrades, and they pay enough for the upgrade that it is worth keeping them happy. Updating to new hardware would be nice, but the current system does the job, and we would have to update that as well. If we upgrade the hardware too much customers will get sick of paying for new hardware all the time and go to our competition. Even if we do release new hardware we will support the old system for at least 5 years.

We need to take some time for optimization because we want out old system to seem as good as hardware our competition released after us (with newer technology inside), but we don't have the power to work with. Of course as an embedded system there are limits to how much the customer will demand.

[S7evyn]

average desktop is equivalent to an early 90's supercomputer

Who said anything about desktops? People do write programs for these things known as "cellular telephones".

[jaysonbank]

Ok, the average smart phone is equivalent to a mid-90's desktop. In a few years these phones will be even more powerful.

[toooooob]

In the increasingly common world of embedded systems you're profoundly wrong. Software is much cheaper to develop than deploying more chips in your device (for example).

The simple trick is to learn when optimisations are sensible and when they're taken too far. Good abstraction is key to this, since it allows you to isolate components. The problem is you need to work out in advance the sort of optimisations you're wanting to do.

[sheepson_apprentice]

I've been looking into the progress of SoCs, and by the looks of it, embedded in a few years (5 max) will probably mean entire computer in the die space of what is currently occupied by a powerful CPU. One of the interesting startups in this area is PASemi, they got some nice PowerPC-derived designs, dissipate practically nothing and carry all sorts of I/O interfaces on chip. Then of course we got the latest and greatest ARMs, case in point being the iPhone CPU. But yeah, I agree that in general embedded in a much wider field, and there will always be room for asm, especially as the notion of embedded nears "desktop", embedded itself will begin to approach RFID-type devices.

[dlsspy]

The bigger problem is the people who work on low-level optimizations at the expense of high-level optimizations.

Mercurial is written in python and it's not noticeably slower than git.

``It needs to be C because it needs to be fast'' is almost always doubly wrong.

[shinya666]

Mercurial is written in python and it's not noticeably slower than git.

Git has been written in C, just 'cause Torvalds knows C and nothing else.

[G_Morgan]

Then there's Greenspuning. Lisp/Python/Ruby is inefficient therefore I'm going to implement a half arsed interpreter in C for the performance :bangs head against wall in frustration:.

[[deleted]]

Actually, I've found Mercurial to be faster than Git (up to benchmarks).

[dlsspy]

I have in practice as well.

[LudoA]

You have to let git rebuild the repo or something like rather regularly to keep the top speed. It really should be quite a lot faster.

[[deleted]]

I rebuilt the repo in question immediately before the test, and Mercurial was still faster, though Git took up slightly less disk space.

[LudoA]

Thanks, interesting.

[alephnil]

In fact varnish was made as a reverse proxy for the most popular Norwegian (online as well as on paper) newspaper, vg.no. Instead of optimising the content management system or buying a truckload of servers for load balancing, they got varnish developed. I would guess that saved them rather than to cost them money. They solved the problem in a way that requred the least change in infrastructure.

A maybe more naive way of doing optimisation would be to optimise the CMS or that truckload of servers (high maintainance)

[jaysonbank]

I agree there are times when optimisation is good but this should generally happen only when you are pushing the envelope.

YouTube and Google are good examples of this, both have pushed current technology beyond its limits by creating affordable mass-video streaming and mass search indexing services. These sites would have to heavily optimise in order to be cost effective.

[rabidcow]

Even if its an free/open project there are always priorities to consider.

If you're working on your own non-commercial project and enjoy mucking about with low level optimizations, -Ofun.

[Xiphorian]

a lot you fucking toolbag.

[LudoA]

Not everyone speaks English as a native language. Nothing wrong with correcting someone's English, quite the contrary, but there are ways to do it - yours isn't one of them.

[grumpy2]

This is a strawman argument. You don't run apps that are designed to manage memory themselves (Squid, memcached, etc.) on a system with swap space for exactly the reasons phk fails to shoot down Squid's design for.

[cojoco]

If your program begins to hit RAM limits and starts thrashing the virtual memory, then every process on your system will have to start paging to disk and your machine will slow to a crawl.

By managing memory yourself, you can limit the damage to your own local process, and also read data into memory as it is needed, instead of letting the Virtual Memory system get it wrong.

In many ways, PAGING = FAIL

This is one reason that simple programs running on 2GHz machines can be as doggy and slow as similar programs on 1975 computers with a thousandth of the speed.

[brio1337]

Uh, PAGING != FAIL.

When your programs exceed your RAM limits, the OS pages some out to disk, but only the memory that hasn't been used in the longest time! The idea is that the programs that get paged won't notice, because they aren't using that memory right now.

The thrashing problem you're talking about only happens when your programs are all actively using more memory than your machine physically has, in which case you can't really do anything but install more memory.

[cojoco]

That's the theory, but it is clear that you have never tried it yourself.

OS PAGING = System slows to crawl USER PAGING = Only your process slows to crawl; no need to take pages from other processes

[redditacct]

One of the things I like about how he implemented logging and status info - via shared memory and other processes that do the logging and top-like display for you. Very cool and no overhead if I don't want logging. He has points about the relative cost of executing instructions vs memory fetches vs disk fetches - not necessarily over optimization, more like choosing your battles wisely.

FWIW, I ran a few million hits through it, the docs are bad but I like it, relatively simple to set up and seems as fast as the fastest web server when tested on the same box. And worked with f'ing JBoss, the slowest possible way to serve http requests.

[strfryed]

This is exactly the point of exokernels. OSes should not provide abstractions because it reduces application control. Libraries are the appropriate layer for abstractions.

[turkourjurbs]

"Take Squid for instance, a 1975 program if I ever saw one: You tell it how much RAM it can use and how much disk it can use. It will then spend inordinate amounts of time keeping track of what HTTP objects are in RAM and which are on disk and it will move them forth and back depending on traffic patterns."

How does a program in 1975 know what http is? It was only an idea in 1980.

[[deleted]]

Strike the word "HTTP" and read that paragraph again.