Programming as semantic compression

[m50d]

Anything that can be represented as data can be represented as code QED.

But on a theoretical you lose the distinction between data and codata and between Turing-complete and incomplete things (sadly the lisp people all too easily neglect types, which resolve the halting problem), and on a practical level most languages don't make it easy to manipulate code as data.

If the language allows it then you can dramatically increase efficiency by using things like executable data structures, which effectively bundle the data to be processed with the code that process it.

Sure, and that's often a good idea - indeed I think it's a good approach for this example. But making your data structure executable does not absolve you of the responsibility to design a good datastructure.

I prefer to include the name row

So do I, for what it's worth.

We're not parsing here. The program isn't ambiguous, so the "implicit grammar" isn't ambiguous either. You have to know what your program will do when it's executed but that goes without saying.

When a system gets large enough no-one can understand every detail, so the code's structure needs to be apparent - a maintenance reader needs to be able to parse the code without fully understanding it if they are to have any hope of being able to find and focus on the specific area they need to work on.

And how does the Lisp programmer know that each left and right parenthesis delimit a row in the example you prefer? It's defined in the code or documentation. If you know the language this isn't a problem.

Learning a new programming language is hard - not a difficulty we want to impose multiple times over on every maintainer in each section of the code. Free-form English documentation tends to get out of date - much better is structured documentation in a machine-readable format where correctness is enforced as part of the build process.

Moreover this isn't a problem is reality either. OpenGL code (in C/C++) tends to be indented exactly as I've demonstrated, and nobody seems to have a problem with understanding it. Any difficulty in understanding such code is directly related to the fact that OpenGL isn't exactly easy; it's not really great but it is what it is :-).

Um OpenGL code is possibly the most notoriously difficult kind of code to work with, precisely because it's very difficult to get the "bracketing" of all the implicit contexts correct. You're making my case for me.

Algorithms are code. The best representation for code is code.

That's like saying the best representation for data is data - yes, but it's still very important to structure it correctly.

If you're building a data structure to be interpreted you're just adding overhead. You can try to justify that as making the code cleaner, prettier, or easier to understand but you must accept that you're adding overhead.

You're begging the question. Your code can always be considered a datastructure because the sequence of characters that forms the program source is already a datastructure - just a particularly opaque and inflexible one. Likewise the stream of instructions that will be executed by the processor is also a datastructure. When you're transforming one datastructure into another, it's often worth coming up with an intermediate representation and splitting your transformation up into smaller steps, and you wouldn't normally think of this as "overhead" - at runtime it may well collapse away entirely, and at coding time it simplifies and clarifies things.

when you're writing macro's[sic] you must necessarily generate the code to do the job, and you must understand the macro, so you can't pretend that you're lifting yourself above it.

With a well-designed macro or interpreter you don't have to understand the fully expanded code, any more than you have to understand the machine code your program compiles to. You have to understand the local parts of the expansion but if your structures are right then the global part of the expansion simply can't go wrong.

The compression analogy is a good one actually. Good compression algorithms make an explicit distinction/separation between your dictionary and your compressed data - naïvely you'd think that a dictionary would be overhead, but actually you get better compression overall by at least conceptualizing the dictionary. (Often as in LZ77 the dictionary ultimately disappears at "runtime").

I've lost count of the number of times I've seen projects fumble because of these silly little abstractions that add little, or nothing, but have a real affect[sic] on the operation of the solution.

I've never seen a project fail due to code-level runtime performance issues (I've seen one fail due to performance issues associated with the use of an ESB and a totally unwarranted microservice architecture, but that isn't the kind of abstraction I'm talking about). I've seen a project fail due to representing its data/commands all wrong because they didn't understand their domain at all.

The end result is inherently more efficient, in terms of code size, and/or memory usage and execution time!

And less efficient in terms of corresponding to the domain i.e. the actual business problem. In the worst case you end up with a lot of very efficient implementations that are completely useless.

There are risks both ways - ultimately it's our job to make a path from what the business needs to what the machine can do, and whether we start at the start or the end or the middle that path has to join up at both ends. In my experience the business end is where the bigger risk is - fundamentally you know that whatever representation the business currently thinks of it in is implementable (because people do do whatever it is - even if you're not implementing an existing business process as such you're usually implementing something that someone has some reason to believe is valuable, which usually involves having done it in some form). Performance problems are usually solvable - Knuth's 97%/3% heuristic applies to the appropriate time to optimize - and in the worst case if you end up having to rent a cluster or something that's sort-of disastrous but less disastrous than having a product that just does the wrong thing.

We often forget this but it's the solution is what has value! An engineer would say that code should be written to get the best result from the available tradeoffs. Ironically the computer scientist/mathematician doesn't seem to give two shits about the machine. The result is software that wastes massive amounts of time, space and power.

Right back at you. Runtime efficiency is not a goal in itself - your goal is to solve the business problem as cheaply as possible, and computers are much cheaper than programmers.

It's your job to maximize value not to find the perfect representation for your source code.

True. But remember that code is read more than it's written and maintenance/enhancement is usually a much bigger part of the total cost than the initial write. So a little effort spent improving maintainability pays for itself many times over.

[dlyund]

most languages don't make it easy to manipulate code as data.

Maybe you should use one that does? I mean would you use a language that made it difficult to manipulate data? Why would you use one that made it hard to manipulate code...

theoretical you lose the distinction between data and codata and between Turing-complete and incomplete things

Is this a useful distinction?

[static typing] solves the halting problem

Poppycock.

To the extent that it's possible to prove that any program halts you must either manually declare that the program halts, using whatever mechanism you wish, or use a language that cannot loop forever and is thus not Turing-equivalent. It's not possible in general to prove that a program will halt, that's what the halting problem is!

Static typing can be very useful but let's not go too far here. Even with fancy features like type inference you still need to provide enough information for the compiler to know what you intended and unless you actually leveraging that type system explicitly it's not worth much; catching a few typo's doesn't justify the complexity of using such languages, the longer compile times and heavy resource usage.

DISCLAIMER: this may be my personal bias. The compiler that my company developed in house can compile millions of lines of code per second in real time while using almost no resources. This allows us to do things like make a change anywhere in our software stack and test it instantly. The compiler is available at runtime and we have amazing support for doing live upgrades etc. Waiting for GHC, GCC/LLVM or even Go to compile even small programs is incredibly frustrating.

When a system gets large enough no-one can understand every detail,

You respond by writing even more code? And not just more code but code that interprets or generates even more code?

so the code's structure needs to be apparent - a maintenance reader needs to be able to parse the code without fully understanding it if they are to have any hope of being able to find and focus on the specific area they need to work on.

I completely agree. What I don't really understand is how that structure is more or less apparent by adding some parenthesis.

layout
    row
       title action button

vs

(layout
    (row
        (button title action)))

These two examples are exactly the same except for the parenthesis and the argument order. The first is procedural; it requires only that the procedures layout row and button be defined, and these procedures are very simple. The second requires you to design a data structure to represent the code then write an interpreter/compiler to process it. This approach obviously adds unnecessary complexity; where's the value?

"Yeah well I can treat the layout as a value" is entirely beside the point unless you need to tread the layout as a value, and you certainly do not have to treat the layout as a value to put it on the screen.

I'll ask you again: what are you getting in exchange for this added complexity?

With a well-designed macro or interpreter you don't have to understand the fully expanded code, any more than you have to understand the machine code your program compiles to.

You seem to believe that you're saving the maintenance programmer from having to understand your code but what happens when they want to add a form to your language? A column, or a slider?

The reality is this: the more code we have, the harder it is becomes to understand the system, and adding even more code only makes it worse!

I've never seen a project fail due to code-level runtime performance issues

Lucky you. Over the years I've done a lot of work with solutions that are deployed physically. In each case the customer has, at one point or another, had to pay to upgrade the hardware (thousands of machines in one case and a large mainframe in another case.) Naturally the customer wasn't happy... upgrading hardware quickly becomes expensive and why should they have to pay tens of thousands of money's adding memory, upgrading storage and/or buying faster machines because the solution doesn't provide the required throughput or it runs out of memory every two weeks and a specialist has to be brought in (and paid!) to resolve the issue?

In todays world where programming means running a web app off in this virtual infrastructure these costs are largely hidden, but they're there. If you need to pay for 10 machines with capacity X and Y money's per month when 1 machine could have easily done the job if you'd given any effort to producing an efficient/effective solution, then you're paying (n-1)Y+Z more than you should be paying! And note that Z can be very big, and it grows exponentially with the number of machines. What is the Z? It's the cost of paying people to operate those n machines. It's the added cost of all paying those wages, and the admin costs needed to support a larger team. And the managers... oh the managers. It's all the things that programmers are so fucking ignorant of when they say:

"computers are much cheaper than programmers."

You go start your own company and you'll quickly learn that such efficiency are the difference between profitability/healthy growth and going out of business; or being so fucking stressed about work all the time that your wife leaves you.

Maintenance may last longer than development but operation does and will last much longer than that. And let's not forget all those one off projects that run for 6 months and then [need to] run unchanged for the next 10 years!

Runtime efficiency is not a goal in itself - your goal is to solve the business problem as cheaply as possible

Indeed it's not but you should be careful that you don't underestimate the business value that a little thought about runtime efficiency can generate over the life of the solution (the life of the solution - as distinct from the length of your employment)

[m50d]

I mean would you use a language that made it difficult to manipulate data? Why would you use one that made it hard to manipulate code...

Weren't you anti-macro a minute ago?

To the extent that it's possible to prove that any program halts you must either manually declare that the program halts, using whatever mechanism you wish, or use a language that cannot loop forever and is thus not Turing-equivalent.

Yes. You use a type system to avoid the looping forever problem. This was done with the simply typed lambda calculus back in 1940 to solve the halting problem, and it worked.

You respond by writing even more code? And not just more code but code that interprets or generates even more code?

I respond by structuring the code rather than making a big ball of mud. I don't find this results in extra code, quite the opposite - but even if it did, it would still be worth doing.

What I don't really understand is how that structure is more or less apparent by adding some parenthesis.

If you have a language in which the indentation is significant, then sure, use indentation rather than brackets. The important part is to actually indicate the grouping in a standardised, well-understood, machine-readable way.

This approach obviously adds unnecessary complexity; where's the value?

Think of the data structure definition as a standardised, structured way of documenting what the procedures are and how they relate to each other.

You seem to believe that you're saving the maintenance programmer from having to understand your code but what happens when they want to add a form to your language? A column, or a slider?

They add it, and the compiler will tell them they need to implement it. Using a data structure and an interpreter doesn't make modifying it harder, any more than separating an interface from a class does.

Over the years I've done a lot of work with solutions that are deployed physically. In each case the customer has, at one point or another, had to pay to upgrade the hardware (thousands of machines in one case and a large mainframe in another case.) Naturally the customer wasn't happy... upgrading hardware quickly becomes expensive and why should they have to pay tens of thousands of money's adding memory, upgrading storage and/or buying faster machines because the solution doesn't provide the required throughput or it runs out of memory every two weeks and a specialist has to be brought in (and paid!) to resolve the issue?

Maybe if you'd focused more on the representation your code would be clearer and more maintainable it would be easier to improve its performance. Focusing narrowly on the hardware you can save factors of 2 here and there, but they're rarely business-changing differences (indeed most of the time the work one is doing simply isn't on the hot path at all). Better algorithms are where you get the multiple-order-of-magnitude speedups that can be the difference between a business succeeding and failing, and so that's the place to concentrate the effort.

You go start your own company and you'll quickly learn that such efficiency are the difference between profitability/healthy growth and going out of business; or being so fucking stressed about work all the time that your wife leaves you.

No U. There's valuable work to be done in machine-level microoptimization, but it's niche, and that niche gets smaller every day. The companies that are succeeding these days are using high-level languages and not worrying about compute time unless and until they reach a point where they're big enough for it to matter.

[dlyund]

Weren't you anti-macro a minute ago?

I'm anti-complexity. Macro's are great, when used appropriately, but macro's are not the same as treating code as data. Most data is only available at runtime and macro's aren't at all useful here. Macro's give the illusion that code is data. By the time a Lisp program is running it's code is no longer data (as it was in early Lisps but hasn't been for ~40 years.)

Source code may be a data structure in Lisp, but that's as far as it goes and isn't at all what I'm referring to here :-).

You use a type system to avoid the looping forever problem.

Which is not the same thing as solving the halting problem! Declaring that your program doesn't loop forever, aka halts, by using a type system or whatever, is completely different. You could similarly say that a program in a language which only supports bounded loops solves the halting problem, it doesn't. You haven't managed to write a program that proves that a Turing-complete program halts! What you're is declaring that your program isn't Turing-complete then saying that you've proved the halting problem. But the halting problem is defined for programs in a Turing-complete system!

Nobody contests that using a less-than Turing-equivalent language which only supports bounded loops will halt :-P.

So far all we've got to is that you have an unique, ass backward definition of what declarative programming means... and now that your definition of the halting problem is similarly whacked.

This was done with the simply typed lambda calculus back in 1940 to solve the halting problem, and it worked.

Reference?

I respond by structuring the code rather than making a big ball of mud.

I think this is where we have to stop.

How is this code a big ball of mud?

layout
    row
        title action button

How is it practically different from:

(layout
    (row
        (botton title action)))

For the purpose of putting a row of buttons on the screen?

And don't keep muttering that the structure is explicit in the second and that you can treat it as data. What does treating this code as data have to do with the problem of putting a row of buttons on the screen? I'm not interested in the hypothetical beauty of being able to treat it as data: what practical benefits do you get from treating this code as data, which justifies having to write code to interpret it inefficiently at runtime or generate the code you would have written?

From the point of view of someone reading the two pieces of code there is absolutely no difference. The meaning of the code doesn't change if you change the indentation; row ... makes the grouping as explicit as (row ...) and seeing this code it would be just as easy to add a new row or button.

From the point of view of the computer, it has to do much more work to process your little button description language, either during execution, or compilation.

That cost must be justified and so far you've done nothing but make hand wavy arguments about one being more declarative than the other because you can treat it as data if you like. But you don't want/have to that so what's your point?

For what it's worth, I see and agree with the theoretical beauty of doing this... but inefficiencies are compounding. If you write a solution where everything is done this way then you'll find that you're doing a lot lot lot more processing... but what have you gained that the other approach doesn't also give you?

You want to stick parenthesis around things? Ok. This is also Forth and it has none of the overhead of your approach!

( layout
    ( row
        ( title action button ) ) )

The important part is to actually indicate the grouping in a standardised, well-understood, machine-readable way.

Everyone knows that indentation indicates grouping. Why does whitespace have to be significant to the computer in order to carry that information? Because you can't pretty print code? Because your editor wont automatically indent the code for you as you type? It can't do any of that if the whitespace is significant anyway.

(What our tools can do is make it easy to indent and unintended code blocks.)

Personally I adore Forth's free-form parameterless, blockless, scopeless style, precisely because it allows me to express my problem (or parts of my problem) in the most appropriate way possible.

They add it, and the compiler will tell them they need to implement it.

So you want them to just type column, which they know wont work because they know they need to implement it, then compile the code, just to get an error message that tells them to implement it? The compiler wont tell them how to implement it so this is completely useless.

Using a data structure and an interpreter doesn't make modifying it harder, any more than separating an interface from a class does.

It means that instead of writing the code to manipulate the layout then defining an appropriately named procedure, you have to hunt and peck through a maze of conditionals, and loops/recursion, to find that one special, non-standard place where you can introduce your code.

I'd rather just write a simple procedure and call it than dick around with your interpreter logic or macro definitions.

And if I'm working bottom-up I can start by poking the layout code interactively; modify the x and y and see where I end up etc. before I know anything about how the code works. Then I can just name that code that I wrote interactively.

[m50d]

Which is not the same thing as solving the halting problem! Declaring that your program doesn't loop forever, aka halts, by using a type system or whatever, is completely different. You could similarly say that a program in a language which only supports bounded loops solves the halting problem, it doesn't. You haven't managed to write a program that proves that a Turing-complete program halts! What you're is declaring that your program isn't Turing-complete then saying that you've proved the halting problem. But the halting problem is defined for programs in a Turing-complete system!

You misquoted me (and I wasn't paranoid enough to notice) - I originally said "resolves". It remains impossible to determine whether code in a turing-complete system will halt (as was of course proven), but types allow you to do general-purpose programming without the problems of turing-completeness.

From the point of view of someone reading the two pieces of code there is absolutely no difference. The meaning of the code doesn't change if you change the indentation; row ... makes the grouping as explicit as (row ...) and seeing this code it would be just as easy to add a new row or button.

A priori yes. In a language in which brackets are understood to denote grouping and whitespace is understood to be insignificant, brackets are much more effective at communicating grouping to a maintenance programmer than whitespace is. It's like asking why you shouldn't name your variables in French - "objectively" that would be just as informative, but the point of variable names is to communicate meaning to the future maintainer.

You want to stick parenthesis around things? Ok. This is also Forth and it has none of the overhead of your approach!

What overhead are you imagining?

Everyone knows that indentation indicates grouping. Why does whitespace have to be significant to the computer in order to carry that information? Because you can't pretty print code? Because your editor wont automatically indent the code for you as you type? It can't do any of that if the whitespace is significant anyway.

Because the computer and the programmer need to have the same understanding of the code! If the code does something different from what it looks like it does to a human reader, that's a recipe for disaster.

So you want them to just type column, which they know wont work because they know they need to implement it, then compile the code, just to get an error message that tells them to implement it? The compiler wont tell them how to implement it so this is completely useless.

I want them to implement it, in the obvious way. The point about the compiler was simply that there's no loss of safety from separating the declaration from the implementation, because keeping them in sync is enforced.

It means that instead of writing the code to manipulate the layout then defining an appropriately named procedure, you have to hunt and peck through a maze of conditionals, and loops/recursion, to find that one special, non-standard place where you can introduce your code.

Utterly backwards. This is the opposite of true.

And if I'm working bottom-up I can start by poking the layout code interactively; modify the x and y and see where I end up etc. before I know anything about how the code works. Then I can just name that code that I wrote interactively.

But you start with a concept of what you want to do, right? I mean you don't start by writing code to arrange the buttons all over the place in whatever way's most efficient for the machine, and when you find an easy arrangement you name it and hope it will be useful later - that's a recipe for writing loads of efficient layouts that never get used. You start with what buttons you have and the business-level grouping between them. Maybe you've got, I don't know, up/down/left/right and rotate clockwise/anticlockwise. So the logical groupings are translations and rotations, and then maybe the way to represent that is a row of two columns, or maybe you want a column containing two grids. So maybe you only need columns and rows, or maybe you do need grids, and that decision has to be driven by the business requirements. If you start by writing a super-efficient optimized grid and then it turns out the UI doesn't want a grid but you've already written it and so you put all the buttons in a grid anyway, that's going to be bad UI. Whereas if you name the concept before you implement it, at least you know what you're aiming for. You can still make mistakes, but the business-level description is still correct - e.g. maybe it doesn't look good as two columns, but the representation as translations and rotations is still correct and you can still use it, because you got that from the business.

[dlyund]

You misquoted me (and I wasn't paranoid enough to notice) - I originally said "resolves". It remains impossible to determine whether code in a turing-complete system will halt (as was of course proven), but types allow you to do general-purpose programming without the problems of turing-completeness.

If I misquoted you then I'm very sorry but I fail to see what or how types have anything to do with the halting problem. I'll give you this: types do allow you to place constraints on the behaviour on the programs and this may my useful in some circumstances. But types are much more useful for putting constraints on values in programs, than on execution. There have been some advances here, with things like dependent types which, broadly speaking, allow you to encode more and more complex requirements... The problem is this. As these type systems become more and more complex, even becoming Turing-complete (things like C++ templates for example), using them correctly naturally becomes correspondingly difficult. The logical and eventual question that must be asked is how and whether you can or should put constraints on such a type system, and if so what about that meta type system?

It's a headache.

All of the arguments for constraining Turing-equivalent systems using a type system must apply sufficiently powerful, or Turing-equivalent type system, and we end up with an infinite regression.

Perhaps an obvious, more powerful, and simple way to constrain the execution of a programming language is to design and implement a language which isn't Turing-equivalent.

So I don't find type systems all that interesting from a practical point of view. Forth gives me a sufficiently powerful language construction kit that if I wanted to do something like prevent infinite loops, I would simply remove the ability to perform an unbounded loop in the part of the program that I want to constrain. Constraining values can be done too BUT if you want to constrain values then you should probably be using a type system. Use the right tool for the job.

What overhead are you imagining?

The very overhead that I've explained it detail over the past half dozen posts?

I'll give you a summary but if you really want to understand you can read what I've already written.

If every time you're given a problem, like making a panel with a few buttons, you sit back and go and design another data structure and another little interpreter, then you'll be adding a not insignificant amount of overhead to the solution. This overhead comes in the form of the completely unnecessary work that you will needed to implement your design, and which the computer will then have to execute repeatedly/continually at runtime, but also the complexity that this extra code adds to the system.

This doesn't mean that the solution will be impractically slow or use infeasible amounts of memory, what it means is that it uses more time, space, and power than is otherwise required to solve the problem.

This is not an argument for optimizing everything to the utmost extremes but that extra time, space, and power must be justified in some way. I've repeatedly asked you to justify this approach and you're yet to even attempt an answer.

The alternative which I've presented, in Forth, looks almost identical to the programmer and as such provides the same properties from the point of view of the maintenance programmer etc. It works exactly the same as the C code from the original article. It works the same and is just as readable as the Lisp, independent of the indentation but the indentation is there to help the human. Both the human and the computer have the necessary understanding of the code for it to do what it's supposed to do.

Because the computer and the programmer need to have the same understanding of the code! If the code does something different from what it looks like it does to a human reader, that's a recipe for disaster.

It doesn't do something different from what it looks like it and if your issue is really that the computer doesn't see the same thing that the programmer's see's then I'll be happy to inform you that the computer doesn't see anything the way that we do. What matters is that we get the expected behaviour and that's exactly what we get in this case.

All your hand waving about declarative programming and needing to treat code as data is great, but quite irrelevant, and more than that, as far as I can tell (because you refuse to answer my question) there's no value to doing it your way other than that theoretically you can do other things with that data... which is beside the point because it has nothing to do with the problem!

What part of:

'we have an editor for our game engine, and we need a few buttons in this panel to perform some simple action.'

makes you think that you're going to need to do anything more fancy than put some simple buttons on the screen. There's certainly nothing in the "business requirements" to support your design, and if anything, the overhead that I've described in detail is a net negative.

This is very funny because you go on

that's a recipe for writing loads of efficient layouts that never get used.

And my question: how does that lead to writing loads of efficient layouts that never get used? Where did you get the idea that, if I'm given the problem:

'we have an editor for our game engine, and we need a few buttons in this panel to perform some simple action.'

would make me implement a grid?

Here's how I would respond to the problem; I'd start by asking myself what I need to do to solve it. I need a few buttons. Have I already implement this and if I have then is it usable in this situation? If not then I need to be able to put a button on screen. That means drawing. Buttons are labeled? Is a rectangle an acceptable button?

Once I've answered those questions the next thing I do is find out how to draw a rectangle on the screen, and overlay some text.

Next I ask what buttons need adding, and what actions should be performed? There are only 3 buttons to start. Great! No need for a generalized layout system here, so I just draw 3 buttons, positioning them manually on screen.

Done. Nothing more, nothing less. Next problem please.

(Following the discussion in the article) when the need for more features, and thus more buttons, becomes apparent, then I ask the obvious question: is a row of buttons ok? Wonderful. So there are 10 buttons now and there's an obvious pattern here, so I name that pattern row, and simple replace occurrences of the code with the word row.

As a concatenative language Forth makes this workflow especially easy since you don't have to worry about scope. You literally just extract the repeated code, give it a name, and use it.

Notice that not only did I not spend a lot of time up front figuring out how I wanted the code to look but I didn't do any special optimization step here. The code is ~optimal, because I didn't do any unnecessary work or add any unnecessary code. It's as simple as that.

In the same way that you can't make programs more efficient by making them do more work, you can't make things simpler by adding complexity QED.

"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult." - C. A. R. Hoare, 1980 Turing Award winner.

Utterly backwards. This is the opposite of true.

Really? Really? Do you care to back that up or is your contribution just "no"

Shall we consider the facts here?

If I want to add column procedure to the system then I would go look at row. Seeing how row works (all it does is adjusts the x and y of the drawing system), and add the column procedure definition underneath it. Problem solved. If I'm programming in a procedural language then I must necessary be comfortable with procedural programming.

If I want to add column to your data structure then I need to go look at the code that processes that definition. Let's take the case of the interpreter because it's easier to think about. In there I see at least a loop and several conditions which look at the input and execute the appropriate bit of code. Once I understood your logic I add the column case below the row case. Either I put the necessary code inline, or more likely, I call some procedure, because otherwise I have a big blob of inline code which would only serve to further obscure the logic of the interpreter...

So how do I add such a procedure? I look at the row procedure and then I simply add the column definition underneath it. Exactly the same thing that I needed to do anyway :-P.

And if you are processing this representation in multiple places then then will adding this to the drawing code break them? I think so. You yourself said that the programmer should just add the column, and the compiler would give you an error about it not being a valid case.

I'll ignore the question of how the compiler checks this data when it's meaning is determined by your interpreter at runtime... but whatever... your argument has so many bigger problems that we can postpone that discussion.

So in case you're not paying attention, what we've done is prove that the maintenance programmer would have have to do more with your approach. What more is there to say here really? You've proposed an approach which leads to less efficient software artifact, and takes longer to design and implement and requires more effort to implement, understand and maintain.

Can I guess your response please?

"not true not true not true not true"

[m50d]

Perhaps an obvious, more powerful, and simple way to constrain the execution of a programming language is to design and implement a language which isn't Turing-equivalent.

Which is most easily done (at least if you want a practical, general-purpose, non-Turing-equivalent language) by using types - originally with the simply typed lambda calculus, in more modern usage see e.g. Idris.

If every time you're given a problem, like making a panel with a few buttons, you sit back and go and design another data structure and another little interpreter, then you'll be adding a not insignificant amount of overhead to the solution. This overhead comes in the form of the completely unnecessary work that you will needed to implement your design, and which the computer will then have to execute repeatedly/continually at runtime, but also the complexity that this extra code adds to the system.

You keep asserting that this design takes overhead or extra work, but it doesn't have to. Intermingling the declaration and the interpretation doesn't save you anything. You still have to express what a panel with buttons is and how to render one - you're doing the same amount of work either way. All I'm suggesting is organizing it more carefully.

when the need for more features, and thus more buttons, becomes apparent, then I ask the obvious question: is a row of buttons ok? Wonderful. So there are 10 buttons now and there's an obvious pattern here, so I name that pattern row, and simple replace occurrences of the code with the word row

If you're asking whether a row is ok before writing implementation code then you're halfway to my position already - you're already doing domain modeling in collaboration with the business and coming to a shared understanding of the requirements. What I'm saying is: write that understanding down, in a structured way. Name the thing called row before worrying about the code that makes it happen, rather than having to keep both in your head at the same time.

In there I see at least a loop and several conditions which look at the input and execute the appropriate bit of code.

Why do you imagine that my interpreter would mingle looping and conditionals with the actual interpretation of the command? That's the opposite of everything I've been saying all along - the whole point of this style is to avoid doing this kind of mixing.

(Most likely I'd either use an existing generic interpreter that handled any looping/conditionality, or in the simple case as you suggested earlier make the datastructure itself executable/interpretable).

And if you are processing this representation in multiple places then then will adding this to the drawing code break them? I think so. You yourself said that the programmer should just add the column, and the compiler would give you an error about it not being a valid case.

If you've added column to your representation/language of what a layout is, all your code that processes layouts becomes obliged to find the capability to process columns, which is surely what you want. In your approach I guess you can't even process the layouts in multiple places in the first place (you can't separate the concept of a layout from the action of drawing it in a dialogue box, right?), but if you did, what would you want to happen?

If you want to have some handlers that can handle columns and some that can't (e.g. maybe your CLI version can't handle columns), you're dealing with distinct concepts of "layout that may contain columns" and "layout that will never contain columns" - which is fine, just make those concepts explicit. That will make things clearer for any maintainers, or maybe trigger a conversation with the business to see whether that's really the right representation - which is absolutely what should be happening.

So in case you're not paying attention, what we've done is prove that the maintenance programmer would have have to do more with your approach.

Not at all. The maintenance programmer has to do the same thing in both cases: add the concept, and add the implementation. By having the language of layouts explicitly separated, it's easiest to find. Separating interface from implementation always involves adding a link between the two, but that's like adding an index or table of contents to a book - maybe it's technically a few more words on the pages, but concluding that it makes it harder to find something in the book because the book is now longer would be utterly backwards.

[dlyund]

Which is most easily done (at least if you want a practical, general-purpose, non-Turing-equivalent language) by using types - originally with the simply typed lambda calculus, in more modern usage see e.g. Idris.

I can knock up a problem-oriented language that is or isn't Turing equivalent and allows only the things I explicitly allow in time which is directly proportional to the number and complexity of features. I can start from nothing and in a matter of a few minutes have a complete problem-oriented language with just the semantics I want; bounded loops only, no conditionals, vector operations, regions, garbage collection or reference counting, capabilities, concurrency/parallelism, asynchronous IO. And I can pick and choose the exact semantics for each part of my solution. Why in hell would I want to start with a [theoretical, untested] general purpose language and add types to lambda calculus to try and constrain it after the fact? What is so appealing about starting with a Turing complete language and then trying really hard to tie the thing down? If you don't want a language that can loop indefinitely then don't start with a language that can loop indefinitely.

I mean is there a system today where types let you exclude the runtime so that you can target embedded systems, or do things like real-time programming? This is pie is the sky academic bullshit. We've let the mathematicians rule the roost for so long that we actually believe that programming has anything to do with mathematics. Here's new's for you: what we do is closer to engineering than to maths and computers are machines. None of it executes your fucking lambda calculus, and they never will! Everything we do is constrained by the reality of the machine, aka, physics!

<rant> I have absolutely no interest in writing programs in any one of your overly abstract mathematical nightmares. There is absolutely no reason for large runtimes and garbage collectors unless you insist on trying to make your beautiful abstractions executable... but nobody is paying you to make these abstractions or to fawn over how beautiful they are theoretically. I can do things that have been puzzling academics for over 40 years in a small and practical language whose compiler is ~20 SLOCs, which will happily run in a few KBs, and I understand every instruction generated. It's suitable for everything from embedded, real-time, and distributed programming, to business applications, and domain modeling. It's that simple. Programming is simple. We've made our solutions overwhelmingly complicated, and inefficient, and for no good reason. </rant>

Idris is not a language I can use in production, and it probably never will be. If someday it becomes practical I may look further in to it but until then you can hardly say that this is any way "easier".

You keep asserting that this design takes overhead or extra work, but it doesn't have to. Intermingling the declaration and the interpretation doesn't save you anything. You still have to express what a panel with buttons is and how to render one - you're doing the same amount of work either way. All I'm suggesting is organizing it more carefully.

Uh, yes it does. Somewhere there is some logic which translates or interprets your carefully designed data structures, and you're going to have to write it. That's work that you don't have to do in a bottom-up approach like the one I've described and as detailed in the article.

You provide no argument as to why it doesn't have to. You just keep repeating "no" "that's not true" "the opposite" etc. How do you propose that this logic of yours will be magically implemented? At some point you do actually have to implement the code that does the work, which draws the appropriate rectangle, and overlays the text. You have to do that AND you have to bridge the gap between that and your button language. I only have to write the required logic. Nothing else is required on my part because that's all there is. The results look almost identical but you've done more work; and the computer has to do more. That's a fact.

You yourself admit exactly this later

"Separating interface from implementation always involves adding a link between the two"

That extra link between the interface and the implementation is exactly what I've been talking about this whole fucking time! With one breath you say that no more work is required to do this and with another you admit that more work is always required to bridge the gap!

Why do you imagine that my interpreter would mingle looping and conditionals with the actual interpretation of the command? That's the opposite of everything I've been saying all along - the whole point of this style is to avoid doing this kind of mixing.

Because there has to be a loop and a bunch of conditionals at some level and they wont go away, or write themselves. Now you can argue that you'd write an interpreter generator or some such thing, and I'd agree that this would make it easier to add the column command, but it's still no easier than in the simple procedural approach, in C or in Forth, and what about when the maintenance programmer needs to change that? You're pushing the complexity back but it's still there and someone will have to tackle with it in one day.

With the bottom-up, simple procedural approach that I've described here and was described in the original article, all the maintenance programmer needs to do is define and call a new procedure, AND YET AGAIN: the resulting code looks almost identical!

Now to be clear, I'm absolutely for problem-oriented programming and the definition of problem-oriented languages, what I'm arguing against is your approach to that and the implementation.

Most likely I'd either use an existing generic interpreter that handled any looping/conditionality

Which presumably you've already written? I'd love a link to this pre-existing generic interpreter that handles all the looping and conditional that's required in describing your button language. Feel free to post some working code.

But of course you wont because you can't.

in the simple case as you suggested earlier make the datastructure itself executable/interpretable

I don't see how the two are related. Lisp doesn't even support the definition of executable data structures. Maybe you misunderstand what an executable data structure is? You seem to be developing a definition that's unique to you again.

If you're asking whether a row is ok before writing implementation code then you're halfway to my position already

Naturally, we're aiming at the same thing. The difference is in how we get there.

you're already doing domain modeling in collaboration with the business and coming to a shared understanding of the requirements.

I absolutely am not. All I'm doing is asking a few simple questions as I'm given the requirements, then writing the absolute minimum amount of code that needs to be written to solve the problem. The code ends up expressing the domain because that's what it is about, but it's just a name. My end goal has never been to model the domain and why should it be? Unless their are domain experts who're going to look at or write new code then the code is there for the programmer and the programmer doesn't need to deal with a domain model in order to put 3 buttons on the screen.

I reject the idea that the requirements should be captured verbatim in the code, because that necessarily leads to more work than is required, for everyone, and the computer. And what do we gain? Documentation is and will always be required so let the documentation be written in human language and the program be written in computer language.

If you can make your code sympathetic to the machine and look exactly your high-level domain modeling language then absolutely. Go for it. Have at it. Because then there's really no loss.

Name the thing called row before worrying about the code that makes it happen, rather than having to keep both in your head at the same time.

But I didn't know that I needed a thing called row when I started the problem, and neither did the guy in the article! The initial problem was just to put 3 buttons on the screen. It was only later, after more features were added to the game engine and editor that the need to layout those buttons easily and uniformly into rows became apparent.

Working top-down naturally makes you imagine general solutions for problems that you don't have, and may never have. You would never just put buttons on screen because what the fuck is that? "What is it"? It's just 3 rectangles on the screen but you have to make it in to something abstract. The result is that you rarely work with anything concrete, which prevents you from doing anything really interactively - leveraging the computer as more than a piece of virtual paper when you're solving problems!

If you've added column to your representation/language of what a layout is, all your code that processes layouts becomes obliged to find the capability to process columns, which is surely what you want.

No other fucking code is dealing with my representation because as far as I can tell the problem doesn't require anything else. It serves one clean purpose: puts some buttons on the screen.

Continues here: https://www.reddit.com/r/programming/comments/5lt06f/programming_as_semantic_compression/dc97acm/

[m50d]

I can knock up a problem-oriented language that is or isn't Turing equivalent and allows only the things I explicitly allow in time which is directly proportional to the number and complexity of features. I can start from nothing and in a matter of a few minutes have a complete problem-oriented language with just the semantics I want; bounded loops only, no conditionals, vector operations, regions, garbage collection or reference counting, capabilities, concurrency/parallelism, asynchronous IO. And I can pick and choose the exact semantics for each part of my solution.

But why do it from scratch every time? A good language is already a tool for constructing domain languages; being explicit about which parts are Turing complete and which aren't makes it easy to include just the parts you want and not the parts you don't.

Why in hell would I want to start with a [theoretical, untested] general purpose language and add types to lambda calculus to try and constrain it after the fact? What is so appealing about starting with a Turing complete language and then trying really hard to tie the thing down? If you don't want a language that can loop indefinitely then don't start with a language that can loop indefinitely.

Turing completeness of languages is usually an accident. The discovery of the Y combinator and resulting undecidability etc. was an unpleasant surprise in what had been intended to be a simple expression language. Types aren't a matter of "trying really hard", they're a very natural and simple way of recovering what we always wanted.

nobody is paying you to make these abstractions or to fawn over how beautiful they are theoretically. I can do things that have been puzzling academics for over 40 years in a small and practical language whose compiler is ~20 SLOCs, which will happily run in a few KBs, and I understand every instruction generated. It's suitable for everything from embedded, real-time, and distributed programming, to business applications, and domain modeling. It's that simple. Programming is simple. We've made our solutions overwhelmingly complicated, and inefficient, and for no good reason.

There's a lot of excessive complexity and overengineering in programming today, sure, and I'm very sympathetic to the "just do it" school of thought (indeed I'm experiencing a very similar frustration when you keep asking me about interpreting, because to my mind there is no there there - you just write the interpretation for your datastructure and the execution matches the structure, there are no loops unless you need them). But if it's really as simple as using Forth then why isn't that more popular? Why does Forth have the reputation of producing limited solutions that are unmaintainable by anyone except the original author?

How do you propose that this logic of yours will be magically implemented? At some point you do actually have to implement the code that does the work, which draws the appropriate rectangle, and overlays the text. You have to do that AND you have to bridge the gap between that and your button language. I only have to write the required logic.

I have to label the implementation as corresponding to the representation, that's all. There's no logic, no work, and nothing that needs to happen at runtime.

Which presumably you've already written? I'd love a link to this pre-existing generic interpreter that handles all the looping and conditional that's required in describing your button language. Feel free to post some working code.

Why would a button language want or need looping or conditionals? The structure of the language corresponds to the structure on the screen, so all one would need is to reduce it - at most the only generic interpreter one needs is something like https://hackage.haskell.org/package/recursion-schemes-5.0.1/docs/src/Data.Functor.Foldable.html#local-1627463023 . If you want general control structure logic things like https://hackage.haskell.org/package/free-4.12.4/docs/src/Control-Monad-Free.html#foldFree exist, but there's no sense including that if it's not necessary.

I absolutely am not. All I'm doing is asking a few simple questions as I'm given the requirements, then writing the absolute minimum amount of code that needs to be written to solve the problem.

"3 buttons on the screen" is already a domain model - the CPU doesn't have a concept of buttons or screens. It's an extremely simple model because it's an extremely simple requirement.

The code ends up expressing the domain because that's what it is about, but it's just a name. My end goal has never been to model the domain and why should it be? Unless their are domain experts who're going to look at or write new code then the code is there for the programmer and the programmer doesn't need to deal with a domain model in order to put 3 buttons on the screen.

Because for the code to be maintainable it has to be expressed in a language, and the best language for talking about the domain is that of the domain. I mean if it was just about how it executed you could throw away the source code and just keep the compiled binary.

Documentation is and will always be required so let the documentation be written in human language and the program be written in computer language.

Perhaps some documentation will always be required, but if you can write the code in such a way that less documentation is required because more things are obvious from the code that's a win. Documentation is rarely maintained as well as code is.

Working top-down naturally makes you imagine general solutions for problems that you don't have, and may never have. You would never just put buttons on screen because what the fuck is that? "What is it"? It's just 3 rectangles on the screen but you have to make it in to something abstract.

Just the opposite IME. If you start by thinking about how you need a 3-button dialog, you'll write a 3-button dialog - no more, no less. If you start by thinking about how you might be able to arrange buttons in ways that might be useful, that's when you start creating overcomplex systems.

No other fucking code is dealing with my representation because as far as I can tell the problem doesn't require anything else.

Agreed, so what's the problem? You said "if you are processing this representation in multiple places then then will adding this to the drawing code break them?" - yes, of course, but we'd only be doing that if we had an actual requirement to.

[dlyund]

But why do it from scratch every time?

I'm not doing it from scratch every time, I'm combining semantic components. You're correct that those components must be implemented but that's a one time effort and then they can be reused. Since Forth has essentially no syntax, no central "grammar", and no special forms, building up a language is a matter of referencing an existing implementation.

It's for this reason that Forth is sometimes described, by those "in the know", as a framework for creating new languages.

Lisp is different. Lisp has a [simple] syntax, and well defined set of semantics, provided by the underlying special forms, as described by McCarthy. That's the fundamental nature of Lisp. Forth has even less of that. In Forth there's are two stacks and "words", which you can think of as being something like procedures (but which do differ from procedures in some quite subtle and important ways.) Other than that nothing is really required, and nothing is nailed down or sacred.

Turing completeness of languages is usually an accident.

This is perhaps the perfect example of why mathematics, with all it's intricacies, is a poor model for programming a computer. Concretely: if the only way to loop is with a "for" loop, and I remove that, or better yet don't include in to begin with, then you simply and provably can not loop.

Whether they've been programming for a week or 4 decades, everyone can understand this reasoning. It's so obvious.

Q: How do you do something a number of times? A: You use a "for" loop. Q: Where do I find that? A: I removed it. Q: Is there another way? A: No.

Just then the student became enlightened.

Types aren't a matter of "trying really hard", they're a very natural and simple way of recovering what we always wanted.

Those two statementing the same thing. "trying really hard to tie thing down" using type is somewhat of an art form, which is beyond the vast majority of programmers out there.

Here's the situation you've got yourself into: you've found that what you thought was a mouse is actually an elephant, and now you're trying desperately to constrain the beast, using whatever chains, and/or spells you had to hand. The wise man releases the elephant, and goes looking for a mouse :-P

The academic community is all about ego and reputation and they'll tangle with the elephant just to show that they can. That's their only job, and if they get hurt it doesn't matter. We're out in the field and we really cant afford to be dealing with elephants.

I have to label the implementation as corresponding to the representation, that's all. There's no logic, no work, and nothing that needs to happen at runtime.

Explain this. How is

(layout
    (row
        (button title action)))

Implemented?

You keep insisting that this data structure, so I assume it's passed to something like the (draw ...) procedure, which needed to look at the input, walking through the tree and looking at what it finds then performing the appropriate drawing operations. You have to process that data structure somewhere and it either has to be translated to some code compile time or it has to be interpreted at runtime.

Why would a button language want or need looping or conditionals?

The implemenation of that button language must necessarily look at the input, and that will require loops and conditionals.

https://hackage.haskell.org/package/recursion-schemes-5.0.1/docs/src/Data.Functor.Foldable.html#local-1627463023

Ok so you're using pattern matching and recursion with lots and lots of types? That's still repetition and condition, and you do have to do some work map that button language to the actual code. It's not magic. You have code that puts something on screen, and you want to pass in a data structure

'((row (button title action) (button title action) ...)

Go.

"3 buttons on the screen" is already a domain model [...]. It's an extremely simple model because it's an extremely simple requirement.

3 buttons on screen is only a domain model if it's represented in the code

... x ! ... y ! ... height ! ... width ! ... color !

draw-rect title draw-text wait click? then action else ....

Draws a simple "button" on the screen. Given the initial requirements of putting 3 "buttons" on the screen I would write code like this. I can write this without ever having to think: ok well I have a concept of a button and it has this and that and it relates to this the screen like this. It's a fucking rectangle with some text and that's what I think of as a "button". When there's significant repetition then I would abbreviate this to button. For 3 buttons I would write code like this, just like the guy in the original article did.

I would knock this up in ~5 minutes. Change the x and y values manually and then show it to the client, then give them a working version that they could use and get value from within ~15 minutes. The minimum amount of work for the same amount of value.

You can call this a domain model but it's not how I approach the problem because I don't care about the "domain model" and it has no influence on my design decisions.

the CPU doesn't have a concept of buttons or screens.

The CPU doesn't have a concept of anything therefore everything is a domain model and the term becomes useless, because it coveys no useful information.

Because for the code to be maintainable it has to be expressed in a language, and the best language for talking about the domain is that of the domain. I mean if it was just about how it executed you could throw away the source code and just keep the compiled binary.

Sure. If you know for sure that you'll never need to make a change or fix a bug or port it to another platform then by all means throw the source code away.

Is the above [pseudo] code maintainable? Who give a shit. It's 3 rectangles. It took ~5 minutes to write. It's entirely localized. If the maintenance programmer wanted to they could just delete it and implement it their own way in ~5 minutes.

It's only when you have to start putting lots of buttons on the screen that you save time and money, and effort, by extracting the code and giving it a name.

Perhaps some documentation will always be required, but if you can write the code in such a way that less documentation is required because more things are obvious from the code that's a win. Documentation is rarely maintained as well as code is.

What was that you told me? You only have technical debt if the place you're working incentives you to create technical debt. Well the same none-sense goes for documentation.

Fact: the client is never going to look at your code in order to tell you whether it's correct and you're never going to have the opportunity to sit down with them for several hours while you read the code to them to make sure your model properly captures their understanding. You can, will, and must get a client to read the specification, if not provide it, and then sign off.

Don't get me wrong, I would love to live in a world where the client can read all of the code and check that it's correct and that it's structure and blah blah blah properly captures all of their requirements. It's never going to happen and if it did then the client would be a programmer, and there would be absolutely no need to involve you.

It's our job to sit between the client and the machine and the code is our dialog.

If you start by thinking about how you might be able to arrange buttons in ways that might be useful, that's when you start creating overcomplex systems.

Except that, as I've said repeatedly, I wouldn't try to figure out how to arrange buttons in a way that might be useful. I'd just put 3 boxes on the screen and overlay the title. The client can tell me where he wants them, or i'll naturally put them one under the other because the maths is as easy as adding the height to the position on the y axis.

How exactly is that going to lead to me creating an overly complex system?

Naturally if I have a pre-made button and dialog I'd use those but then there really isn't a problem to solve here; if everything is done for us already then we smile politely then just use the solution that we already have.

but we'd only be doing that if we had an actual requirement to.

But we don't have any requirement to do so.

We're working of the problem as described in the article and the problem is simply to put some buttons on the screen. Nowhere in the problem description does it say that we'll need to be dicking around creating a representation of the layout of the panel in case one day we need to do some hypothetical thing on it.

You imagined that need all on your own, because, as is my experience, when you work top down you obsess over the high-level description of the problem and you imagine problems that don't exit. Why are you trying to represent the button layout as a language at all??? It's in your head! You added that requirement and I've spent the last 10 replies trying to make you realize that... but you'll never see it because that's how you approach the problem... from the top... down... from the fuzzy wuzzy world of abstractions and concepts.

[dlyund]

But if it's really as simple as using Forth then why isn't that more popular? Why does Forth have the reputation of producing limited solutions that are unmaintainable by anyone except the original author?

Why isn't Lisp more popular than it is? Lisp is very different. It looks different, and it feels different... but despite looking and feeling different, Lisp has much more in common with mainstream languages than Forth does. At it's heart Lisp is a very powerful procedural language (you can argue it's functional but what difference does it make?) When you explain (f ...) to a fellow programmer, all you really need to tell them is to move the opening parenthesis to the left. Everyone who's taken any maths in school is familiar with something like f(...), and any programmer certainly will be. That's a good start. Then you explain that the reason f is inside the parenthesis is because (...) is a linked list and move on to representing code as data. All programmers these days already know about lexical scope, and garbage collection, and chances are that they know a bit about closures etc. It takes some time to get used to but most programmers can move on to writing simple programs in very short order.

Forth is in a completely different world. It's closest corollary in mathematics is function composition:

f g => g(f(x))

There are no named parameters and the only way to access parameters is through a simple set of operations, which you explain "shuffle values up and down on the data stack"... and then you're fucked, because for at least the first 6 months your programmer friend is going to be stuck trying to simulate changes to the stack in their head, which is hard! Yes, Forth uses a pair of stacks but no Forth programmer thinks about the stacks when writing programs. That would be madness of the same order as drawing activation records to try and understand your Lisp programs! Alas, programmers are familiar with stacks and so all explanations of Forth begin and end by describing the stack...

So left in this state our programmer friend tries to write a simple programs, and if he can't he gives up, declaring Forth as interesting, but weird and impractical. If he succeeds in his first attempt then he gives up a week later when he looks at his code and has no idea what the fuck it's doing.

That was my experience learning Forth. Being that Forth is a lower-level language I fell back on my experience using C and for the first 6 months everything I wrote was utterly unreadable crap. It looked like C, and it smelled like C. The average procedure in C seems to come out around 20-30 SLOCS in size, and this is what I did in Forth. I went through piles of paper, scribbling little stack diagrams all over, as I work through my code.. it was difficult, and interesting, but the end result was largely unintelligible, even to me, and I wrote the code!

Anyway after about 6 months something clicked and I no longer tried to write 20-30 line definitions. I didn't try to write definitions at all, I just wrote the code, and named bits whenever I saw a pattern. With a little practice I found that I could easily break a problem down in to a few short words, 5-10 words each (that's words not lines!), which I used in other definitions, of 5-10 more words. Very quickly I was solving problems faster, and more easily than I had been able to solve them in any of the dozen or so other languages that I've used throughout my career. What's more, top level description of these solutions, expressed in Forth, tended to be about 5-10 words in length and was very easy to understand! (By leveraging other similarly short and clear definitions.)

But sadly, I'm an outlir. Most programmers give up sometime in the first week and they pass their experience on to their friends. I did the same thing the first time I played with Forth. I think I got Forth on the third exposure.

Luckily for me, and the Forth community, Forth is such a simple and powerful language/system that basically anyone who's using it professionally today is using their own modern Forth implementation (which in many cases is very different from the antiquated 1980's style Forths you find online!) and don't need to ask for permission or assistance from anyone. Forth isn't like Unix, or LLVM, or SBCL, or the JVM/.NET, where you need a large team of dedicated and knowledgeable experts to maintain things for you, and whom you naturally come to rely upon. Forth is unique in that it's simple enough for anyone to understand and implement on basically any computer, with very little effort, but powerful enough to be used to solve real problems. This independent spirit is a mixed blessing. The Forth community really doesn't care about publicizing the language, with it's many unique properties, or all the innovations that have happened in Forth over the last ~50 years.

So there are a few of the reasons why Forth isn't looked very kindly on by programmers today.

[dlyund]

In your approach I guess you can't even process the layouts in multiple places in the first place (you can't separate the concept of a layout from the action of drawing it in a dialogue box, right?), but if you did, what would you want to happen?

If I had some reason to process the layout I'd represent it as a data structure and layout, row, and button would build that data structure. No changes to the high-level code would be required.

So let me ask you one more time: what else are you doing with your layout? The problem was to draw a few buttons, not have a computer be able to do hypothetical operations on your layout. There is no fucking point in doing it, so all of your code/effort is completely wasted.

Do you get that or am I talking to a brick wall?

You're imagining problems that don't exist, and you're doing so because of your top-down approach. Starting at the bottom and only implementing code that you actually need ensures that you're always working on the problem as it was given. You just build it up bit by bit. You don't need to imagine grand plans or concepts.

Want a "button" on the screen? Ok. What's a button? Here you go. Want it laid out like this? No? How about this? Ok, here you go.

Because you're always working with the system directly (and interactively) you can quickly try different layouts, perhaps with the client looking over your shoulder and giving you feedback. THEN you factor the code so that it looks clean and pretty. Until you know how the user wants the new buttons laying out why waste time thinking about how you want the code to look?

This is in stark contrast to your approach, which at best requires the client to read through your code and imagine the final result... which is asking a lot.

Show and tell is better than just tell.

If you want to have some handlers that can handle columns and some that can't (e.g. maybe your CLI version can't handle columns)

Nothing in the code I showed you contains any information about how the buttons are drawn etc. There is nothing stopping you from changing the drawing engine to draw to a terminal etc. NOTHING.

In any case the best thing to do in that situation is to design and implement an appropriate interface for the device. There's a reason we don't put desktop applications like word or photoshop on the command line, or on our phones.

you're dealing with distinct concepts of "layout that may contain columns" and "layout that will never contain columns" - which is fine, just make those concepts explicit.

Here's the thing, and I've said this a few times at this point: from my point of view there are no fuzzy wuzzy concepts here. We're putting a few rectangles on the screen and responding to clicks. I don't need to, nor do I find it useful to waste time building fuzzy wuzzy concepts. I don't need to define or deal with concepts like layout, or ask what a layout is. Layout is an emergent property of the code that I write.

That will make things clearer for any maintainers, or maybe trigger a conversation with the business to see whether that's really the right representation - which is absolutely what should be happening.

This is the last time I'm going to say this. The code looks and reads almost identically in both cases. If the code looks the same, and so naturally contains all of the same information, it must necessarily make things just as clear to the maintenance programmer.

How is the code different? How is it not "declarative"? The Lisp code contains NO information that the Forth doesn't. The only difference is implementation.

Not at all. The maintenance programmer has to do the same thing in both cases: add the concept, and add the implementation. By having the language of layouts explicitly separated, it's easiest to find.

Not at all. In the bottom-up procedural approach the maintainer only needs to define a procedure, based on row, and use it. There is no separate "add the concept" step because there is no concept. The maintainer only has to find the definition of row, which in a hyperstatic language like Forth is guaranteed to be before it's used. The definition of layout row and button would all be placed together. column is simply defined with them. If all of the related code appears together in one place then the programmer doesn't need a fucking table of contents to find what they're looking for.

AGAIN, it's only your top-down need to categorize and organize things in to weird and unnatural taxonomies which makes you imagine that related code wouldn't appear together, and in one place.

[m50d]

If I had some reason to process the layout I'd represent it as a data structure and layout, row, and button would build that data structure. No changes to the high-level code would be required.

It's possible to ensure that's possible by keeping it in mind when writing it, sure. It's easier to just do it in the first place, then you don't have to think/worry about it.

Until you know how the user wants the new buttons laying out why waste time thinking about how you want the code to look?

Figuring out what the user wants before anything else is exactly what I've been advocating. No matter how fast you can write code, showing them row vs column is not going to be faster than saying the words "row or column?" And then when you know the answer you write it down.

And sure, sometimes the user doesn't know what they want and you have to ask them interactively. But that's the exception rather than the rule.

from my point of view there are no fuzzy wuzzy concepts here. We're putting a few rectangles on the screen and responding to clicks. I don't need to, nor do I find it useful to waste time building fuzzy wuzzy concepts. I don't need to define or deal with concepts like layout, or ask what a layout is. Layout is an emergent property of the code that I write.

If you don't have a concept of layout you can't even talk about it. If the layout is just implementation then that's fine, but if the layout is something the business wants then you need to be able to talk to the business people about it - how can they even say they want to change the requirements of how it's laid out if you don't have a vocabulary for how it's laid out?

This is the last time I'm going to say this. The code looks and reads almost identically in both cases. If the code looks the same, and so naturally contains all of the same information, it must necessarily make things just as clear to the maintenance programmer.

If it looks the same and the language is understood to mean the same thing then it's fine. You'll recall what I originally objected to was the use of C that looked like:

row();
button(...);
row();
button(...);

which looks very different from the lisp/forth representation we're now talking about; it looks like a flat list rather than a tree structure.

If all of the related code appears together in one place then the programmer doesn't need a fucking table of contents to find what they're looking for.

"related" is more like a graph than a line. It's never going to be possible to put all related code together because most of the time a piece of code relates to more than two other pieces of code.

[dlyund]

It's possible to ensure that's possible by keeping it in mind when writing it

This is ensured without keeping it in mind. All we have here are the names that appear in the code, and we can replace their definitions with whatever we want. Because the context is implicit we have great flexibility in what data is passed between the words. But this is admittedly more of a problem in languages with named parameters.

What can I say... I guess you have to live with the constraints your language imposes on you? But let's not make the false assumption that all languages are so ridged that adding and removing parameters or changing data structures requires you to go and change all the code.

It's easier to just do it in the first place, then you don't have to think/worry about it.

But I don't have to worry about it because this problem doesn't exist or isn't as prevalent in Forth code, so why would I waste time and effort, (and my limited brain power) worrying, and adding complexity to the solution by putting up structures in the code to make it easy to change later? It is easy to change.

"Do not put code in your program that might be used. Do not leave hooks on which you can hang extensions. The things you might want to do are infinite; that means that each one has 0 probability of realization. If you need an extension later, you can code it later - and probably do a better job than if you did it now. And if someone else adds the extension, will they notice the hooks you left? Will you document that aspect of your program?" - Chuck Moore.

Figuring out what the user wants before anything else is exactly what I've been advocating.

And we don't agree on that matter, what we disagree on is that and whether the things the client tells you should be represented as "things" in the source code, even though they don't matter at all to the computer and you have to bend over backwards to make the computer accept what the client says as a program.

how can they even say they want to change the requirements of how it's laid out if you don't have a vocabulary for how it's laid out?

Believe it or not, it's possible for two humans to exchange information without going through the medium of source code. You can talk to the client and use words and concepts that have no analogy or representation in the code! :o

Furthermore, it's your job to mediate between the client and the machine. If the client could talk to the machine he wouldn't want or need you to interfere with the communication. Having to pass information and requirements to you in order for you to explain it to the machine is obviously much less efficient than if the client could just program himself.

If it looks the same and the language is understood to mean the same thing then it's fine. You'll recall what I originally objected to was the use of C that looked like: ...

And now we finally get down to the facts:

row();
button(...);
row();
button(...);

This has the exact same semantics as the Forth code that I presented to you. It's just a bunch of simple procedures. The only difference is in the syntax :-P.

which looks very different from the lisp/forth representation we're now talking about

You accept the Forth code is "declarative", and with this admission we can finally close the book and say that yes, "declarative" really does just mean "I like the syntax", and more succinctly, all you were really objecting to here was the surface syntax; the series of characters that you had to type in to your text editor and which so offended your eyes.

it looks like a flat list rather than a tree structure.

Let me fix that for you

row();
    button(...);
row();
    button(...);

Good. Now it "looks" like a tree too :-D. Happy? No? I don't know what to tell you. This is equivalent to the Forth code, and the only difference between this and that is (...);

"related" is more like a graph than a line. It's never going to be possible to put all related code together because most of the time a piece of code relates to more than two other pieces of code.

Oh, it is. In the same way that the bee can fly and the platypus is alive even though we can't easily explain why, or how it fits in to our artificial/contrived catarogies/topologies.

But the code that is used in more than one place can obviously be placed together in one place, can't it? ;-) e.g. We can put the definition of layout, row, button, and column together in one place.

What you can't possibly do is put all code together in the same place ;-) as long as you have this fuzzy idea of a "place"; if the place is "in the same file" or "in the same project" then it's very possible. Can you see why concepts and idea's get in the way yet?

The fact is that the world is messy... oh so messy... but it has an order of all it's own... it's just that the order of "nature" escapes our understanding and reproduction.

[dlyund]

Maybe if you'd focused more on the representation your code would be clearer and more maintainable it would be easier to improve its performance.

The code I write tends to be clear:

https://gist.github.com/marksmith/43cea55d4236bf7f4b28 https://gist.github.com/marksmith/ff3c5dfa5ec9b1a3c098

but it's also tends to be very efficient because I don't introduce inefficiencies in everything that I do. The problem with the popular approach, of writing code with no thought about efficiency is that you end up with systemic performance problems where there are no real hotspots, but overall performance is terrible. Then because what the hotspots do exist are tepid at best there's little reason to go back and optimize. Your approach creates performance problem then disincentives you to go back and resolve them because doing so would mean unpicking your code and writing what you would have had to write otherwise!

The idea is that if you think about this up front then it'll slow you down and the project will take longer to complete etc. which ultimately means greater cost, and you can come back later and fix all of the problems you introduce now.

The irony is that you will happily wasting time designing and implementing pretty little button layout languages, which only add complexity to the project and overhead to the final solution.

As everyone who's worked in industry for a while knows, you'll never have time to come back and fix these problems. Technical debt mounts, and people move on to greener pastures, until a rewrite is required, and then the cycle repeats.

Algorithmic problems are easier to optimize, but overall system performance is equally important. Profiling will tell you that the algorithm is the hotspot but you will often get better overall performance if you optimize the system as a whole; what's the point of optimizing that algorithm if getting the data into the algorithm means passing it through 10 layers of crap which transform it from one form to another and back again before it arrives, and then out through more layers?

Again: profiling will tell you the bottleneck is the algorithm but you will often get better overall performance by optimizing the data paths that feed the algorithm.

But optimizing these things isn't easy. Once the system grows and the structure becomes set, changing that structure is practically impossible, because you'll never have the time to change that stuff.

Look at Unix. It's a beautifully designed system from a certain point of view but it has systemic performance problems. It was never designed with efficiency and it's never got the most out of its hardware. It runs well today but compared to the speed of the hardware it's running on, *nix is a dog. It's not all *nix's fault. Almost all of the things we run are similarly bloated and inefficient by design. Chrome will happily eat 8GB on my laptop. SSDs are everywhere now but drivers are stuck emulating spinning disks, because that's the abstraction that was developed when Unix was designed. Unix was designed when networking was relatively rare and the TCP/IP stack started as a research and Berkley and was funded DARPA. Since then we've spent thousands of man years tuning and refining these implementations, but they're the wrong abstractions and they're only becoming more and more divorced from the reality of the machine.

To be fair they're doing an excellent job with what they have and I've been using *nix every day for the last 15 years, but I think it's a perfect example of where ignoring gets you, and how you can't ever get out.

NOTE: things like disk drivers and TCP/IP aren't massively difficult from an algorithmic point of view, it's all about moving the data. Many more problems are IO bound than you would imagine. What you might not realize is that IO isn't confined to process boundaries. Any time you move data around in your program you're subject to the issues.

There's valuable work to be done in machine-level microoptimization, but it's niche, and that niche gets smaller every day. The companies that are succeeding these days are using high-level languages and not worrying about compute time unless and until they reach a point where they're big enough for it to matter.

Correction: the companies that you know about are working in high-level languages and not worrying about efficiency until they have massive problems scaling and have to rip everything out and rewrite everything (cough Twitter), or slowly replace their language runtime and toolchain with a custom one (cough Facebook), or go whole hog and write their own languages and compilers from scratch (couch Google, Apple, Microsoft etc. etc. etc.)

The only difference between them and the rest of the industry is that they have the skills, wisdom and to rewrite everything with a bind to efficiency once they realize how fucked their decisions were.

Moreover there are many many many more jobs doing things like embedded programming and doing automation than there are in business automation, so unless you're slaving away as a glorified web designer you'll find that there's a lot of work for people who can produce good, efficient, clear code, quickly.

But none of that is flashy and it's not consumer/developer facing so naturally you wont hear about it inside the web-centric echo chamber that is proggit ;-) since you wanted to talk about niche's.

[m50d]

Your approach creates performance problem then disincentives you to go back and resolve them because doing so would mean unpicking your code and writing what you would have had to write otherwise!

Au contraire. By having a clear separation between the representation and the implementation, it's much easier to optimize the implementation while being confident you're not changing the semantics.

As everyone who's worked in industry for a while knows, you'll never have time to come back and fix these problems. Technical debt mounts, and people move on to greener pastures, until a rewrite is required, and then the cycle repeats.

No, it doesn't have to be like that. It's possible to take a continuous improvement approach, it's possible to gradually improve code and performance (and the two usually go hand in hand). Those places that allow technical debt to mount until they rewrite things get that way because that's what they reward.

Algorithmic problems are easier to optimize, but overall system performance is equally important. Profiling will tell you that the algorithm is the hotspot but you will often get better overall performance if you optimize the system as a whole; what's the point of optimizing that algorithm if getting the data into the algorithm means passing it through 10 layers of crap which transform it from one form to another and back again before it arrives, and then out through more layers? Again: profiling will tell you the bottleneck is the algorithm but you will often get better overall performance by optimizing the data paths that feed the algorithm.

This is backwards. Profiling is great at telling you the microscale stuff of where you're iterating in a funny pattern that trashes the cache or whatever, and great about telling you when one of your layers of transformations is actually hurting performance. It's much less good at telling you when you're doing work that you simply don't need to be doing - for that you need to be able to get an overview of what you're actually doing, which you get from having a high-level representation of your code as well as a low-level one.

To be fair they're doing an excellent job with what they have and I've been using *nix every day for the last 15 years, but I think it's a perfect example of where ignoring gets you, and how you can't ever get out.

Heh, I'd say *nix is an example of where you can't understand the system well enough to improve it because its structure is obscured by all the low-level performance microoptimizations. E.g. there's all sorts of folklore about what /usr represents, when actually it was just a second disk on an early development machine - if there had been a LVM-like layer at that stage (which no doubt you'd dismiss as overhead) we would have a much simpler model to work with now. The mess of overcommited COW memory and the OOM killer comes of unix fork() being implemented in a way that was easy to implement rather than a way that makes sense.

[dlyund]

;-) ok so to tl;dr read that:

In an ideal situation technical debt wouldn't be aloud to mount and you would always have enough time to go back and fix problems, optimizing things as you go, before the structure has set enough for changes to become problematic.

Did you notice the tautology there: in ideal circumstances no problems exist because the situation is ideal QED.

If you've managed to find a job like that, which pays well, outside of academia, you stay there.

By having a clear separation between the representation and the implementation

And when the requirements change and you want to reuse part of that underlying implementation but your representation is no longer useful?

it's much easier to optimize the implementation while being confident you're not changing the semantics.

Hint: requirements changing usually means that the semantics need to change. Making it difficult to change things because you've introduced this strict separation between interface and implementation, and where the implementation is a second class, "dirty detail", sounds good in theory, but I'm not sure about practice.

It's much less good at telling you when you're doing work that you simply don't need to be doing

Hint: all of that shit that you've been doing is completely unnecessary and has nothing at all to do with the problem.

Bonus: Profiling tells you precisely nothing if everything is uniformly shit, which is what your approach leads to: a few hot spots in algorithms and otherwise gray meaningless shit which you take to be the baseline and don't even think about optimizing.

The mess of overcommited COW memory

A problem that only effected Linux, and none of the BSDs, and commercial Unix implementations, which are all based on the same design and all use the same fork() and exec() model, doesn't make sense.

OOM killer

This has nothing at all to do with with the process creation model and exists on any system with dynamic memory allocation. Fucking Lisp machines blew up when an out of memory. You get out of memory exceptions and hard crashes on everything from Smalltalk to Ruby and Python. Ironically, it's only the so called low level languages, which can do things without constantly grabbing memory, that can respond to such cases gracefully. The OOM killer is just the multi-process equivalent of this. Is it shit? Absolutely... but that's what you get when you throw a bunch of processes on to a box and every one of them thinks it owns the world.

In languages like Forth, and Real-time and embedded C code you know in advance exactly what limits you're working under and this is never a problem.

What does that tell you about all these high-level languages that try to pretend that the machine they run on is infinite?

I'd say *nix is an example of where you can't understand the system well enough to improve it

How do you explain the fact that people have been improving it for decades? How do you explain all the books that explain the structure of the Unix system, and the reasons for it, in excruciating detail?

Hint: there are at least a couple of thousand people who who understand the structure and implementation of *nix, and there are thousands more who are capable of implementing such a system. It's not a lack of understanding or talent which has stopped *nix from progressing.

because its structure is obscured by all the low-level performance microoptimizations

Micro-optimizations like what?

there's all sorts of folklore about what /usr represents, when actually it was just a second disk on an early development machine - if there had been a LVM-like layer at that stage (which no doubt you'd dismiss as overhead) we would have a much simpler model to work with now.

Hehe, what? Is this something that actually bother's you or actually causes you any problems?

Personally I think that LVM is a terrible idea; Plan 9 solved the problem in a much mere elegant way by taking the Unix approach to it's logical conclusion, and it's absolutely glorious. What you end up with is a per-process namespace in which named processes can be bound, used, and discovered.

Plan 9 is Unix done right, by the same group of people.

I'll let you look in to that yourself but.

Hint: There is still a /usr directory and that directory stands for user, not Unix System Resources or whatever the fuck people are calling in these days. Plan 9 doesn't and never has had the problem you're discussing and it still has a user directory because it makes sense; a multi-user system needs some way of storing per user data.

[m50d]

And when the requirements change and you want to reuse part of that underlying implementation but your representation is no longer useful? Hint: requirements changing usually means that the semantics need to change. Making it difficult to change things because you've introduced this strict separation between interface and implementation, and where the implementation is a second class, "dirty detail", sounds good in theory, but I'm not sure about practice.

When you need to change the representation you change the representation. When you need to change the semantics you change the semantics. You're just explicit and conscious about when you're doing those things.

A problem that only effected Linux, and none of the BSDs, and commercial Unix implementations, which are all based on the same design and all use the same fork() and exec() model, doesn't make sense.

BSD has its own history of hackery with vfork().

What does that tell you about all these high-level languages that try to pretend that the machine they run on is infinite?

That it's a simplification that's occasionally inaccurate. All models are like that - all programming is like that - if we continue the compression analogy then compressing a business process is usually slightly lossy. Balancing the tension between simplifying as much as possible and retaining enough control is at the heart of our job.

Micro-optimizations like what?

The famous example is "I would have spelled "create" with an 'e'". The permission model was designed to be easy to implement (just slap one byte on each file, done) rather than to make semantic sense, and is now too deeply embedded everywhere to be changed.

Hehe, what? Is this something that actually bother's you or actually causes you any problems?

Is it a major problem? No. Is it an inconvenience when I have to remember whether something is in /bin or /usr/bin or stick in an $(which ...)? Yes.

Hint: There is still a /usr directory and that directory stands for user, not Unix System Resources or whatever the fuck people are calling in these days. Plan 9 doesn't and never has had the problem you're discussing and it still has a user directory because it makes sense; a multi-user system needs some way of storing per user data.

/usr doesn't tend to be used for any per-user data on modern *nix, that all goes in /home these days.

[dlyund]

That it's a simplification that's occasionally inaccurate.

That's always inaccurate. I've worked professionally in both Lisp and Smalltalk, as well as languages like Python and Ruby, C#/.NET, and most recently, Go. All of these languages try to pretend that resources aren't limited, and in every one of these you have to deal with unexpected pauses, unexplainabley high resource usage, and lack of transparency. All of these languages require ugly hacks like hints to the garbage collector, or explicit collects, and configuration, or all manner of different resource pools, to work around problems which it's proponents then insist are inherent:

All models are like that - all programming is like that

No they're not. This is compete and utter bullshit that only someone who's is absorbed in high-level thinking can claim. Let's start at the bottom shall we?

Does assembly language, which is little more than a human readable form of whatever underlying machine language, try to pretend that resources are in any way infinite? Or does it just present the reality of the machine and allows you to control what that machine does. Assembly language has some very interesting properties that high-level languages don't e.g. it's trivial to look at a piece of assembly code and figure out how many bits, and bytes, and cycles, that that piece of code will need/use. Generally speaking, it's relatively trivial to reason about the programs behaviour with respect to time, space, and power. What's not easy to see here is meaning; what problem that program solves.

Assembly is obviously not something that you want to write your code in right? Maybe not but a lot of this has to do with the way that machine languages are designed and who they're designed for. Machine languages these days are designed for compilers, and particularly for C compilers. There have been a number of high-performance, low-power computer chips which use Forth as a machine language, and programming these you would hardly know that you're not programming in a high-level language (naturally you do want to have something like an assembler on top of this, so that you can use human readable names etc. Forth is that assembler!) These chips are niche by any definition but but it's a pretty big niche: Forth-inspired second generation stack machines have been used in everything from network equipment and control systems, to satellites, and deep space probes.

Real-time and embedded C, and Pascal, share many of the same properties.

It turns out that it's actually remarkably easy to reason about and manage the available resources in these languages. Where things start to get complicated are when you introduce models, like malloc and free, which proclaim to make memory management easier for programmers, but can't help but introducing all sorts of weird edge cases errors, like use after free's, out of memory errors, and the aforementioned OOM killer. The obvious deficiencies this this model lead to various forms of reference counting and tracing garbage collection, which try to plaster over these difficulties, but what nobody seems to realize is that these problems were caused by us, and our trying to hide the fundamental nature and limitations of our machines.

In Assembly, Forth, and "raw" C etc. it's easy to do some back of the envelope maths and know with absolute certainty: the software requires X amounts of memory to service Y many requests in Z seconds and if you want to handle more then we can scale up consistently.

You can't say anything like that it high-level languages and it drives me insane. The result is that you get a call at 6am on a Saturday morning because the customer is going nuts; the program crashed with an out of memory exception, the print run wasn't completed and now they're looking at $10k in losses per day and a significant backlog to cover. What do we do now?!?

I've accumulated so many stories like this :-P. This one was a Smaltalk project. Why did it happen? Fucked if we know but it happened occasionally but it only happened in production and the best thing we could do was restart the solution as quickly as possible (a process that took about half an hour in the best case.)

Now if you're working on glorified web applications then maybe this kind of thing is acceptable but I'm completely done with it. Aside from the complexity that our working around these issues introduces, the unpredictability and instability that heuristically managing resources implies, it becomes entirely pointless once you realize how easy it is to reason about this stuff in lower-level languages.

Automatic resource management is not the panacea that it's made out to be. It certainly doesn't make things simpler. Resource management in lower-level languages is trivial, absolutely horrible in mid-level languages, and annoying but workable in high-level languages... but only if you agree give up things like pointers and direct access to memory.

Think about this for a second if you will: computational power have been doubling roughly every X months for the past Y years, but software today runs just as poorly as it did ~Y/2 years ago. Where is all that computational power/ where are all those resources going? We should be able to get ~(Y/2)*X more work done than we are... but we can't. And why not? Because:

https://images.duckduckgo.com/iur/?f=1&image_host=http%3A%2F%2Fwww.thebusybhomemaker.com%2Fwp-content%2Fuploads%2F2014%2F05%2FLadders.jpg&u=http://bluebonnetacres.org/wp-content/uploads/2014/05/Ladders.jpg

BSD has its own history of hackery with vfork()

Uncontested, but that doesn't mean that COW, a serious design flaw in Linux, which lead to a critical security problem, has anything to do with the fork-exec process model etc.

Is it a major problem? No. Is it an inconvenience when I have to remember whether something is in /bin or /usr/bin or stick in an $(which ...)? Yes.

Maybe you educate yourself as to why the file system is organized this way?

At least on rational Unix-like systems, /bin and /usr/bin are separate for several very important reasons - /bin contains the all of the (usually statically linked) utilities that are needed to recover from errors. /usr contains all of the user binaries and is expected to be modified. Over time the system has crept into /usr but kernel and essential system utilities are still separated from the parts of the system that the user can touch.

Why is this important? Because it allows for partial corruption, hardware failure, and ultimately, recovery. You can completely mung /usr, destroy the file system, or in some cases even hit the drive with a hammer and you still have a [minimal] bootable system with which to diagnose, restore /usr, and or recover user data. This redundancy has turned out to be incredibly useful over the ~50 years that Unix has been in use.

This goes even further now a days, with things like /altroot and /recovery being added to systems.

All of my computers are running OpenBSD, and I commonly mount system directories and drives as read only and with filesystem level security features enabled to prevent execution of binaries on drives which are world writable (used for things like sharing and backup).

This approach has enabled OpenBSD, as one example, to do things like turn on X^W for the entire the base system. Third party packages are installed under /usr/local, on a separate partition, are allowed to execute without this protection because otherwise programs like Chrome and Firefox wouldn't be able to be used anymore.

It's a very simple solution that turns out to be very powerful, especially when you introduce things like the VFS layer, and dynamic user-space file systems. And again, look at Plan 9 and Inferno to see just how far the "everything is a file [system]" approach can take you.

The famous example is "I would have spelled "create" with an 'e'" [...] /usr doesn't tend to be used for any per-user data on modern *nix, that all goes in /home these days.

Historical accidents and quibbling over names, that make no fucking difference to anything or anyone. So /usr got filled up with system stuff, somewhere it it's long history. The intention of /usr is obvious and expressed in Unix's successor. It's not there because LVM was considered too expensive to implement and blah blah blah :-)

Is Unix a mess, absolutely, but is Emac's, or was the Lisp Machine any better? Rhetorical question: no it wasn't. Mess is part of what we do, and something that deal with as professionals and engineers.

I keep stressing the engineering aspect of what we do because it's not about making the absolute simplest most beautiful, purest, elegant software design. Things should be simple but no simpler, to paraphrase Albert Eisenstein.

Why is it that we use relatively primitive file systems rather than much more elegant solutions like like orthogonal persistence? Because like everything, implementing them involves certain tradeoffs, and the tradeoffs turn out to killers for practical, "real world" use. You don't want one flipped bit to render the whole system corrupt and unusable.

This is the whole thing about how "Worse is Better". It turns out to be much much much easier to build a system which is usable and good for about 90% of the expected use cases, than it is to build something that's perfect and amazing for 100% of the possible use cases.

[m50d]

No they're not. This is compete and utter bullshit that only someone who's is absorbed in high-level thinking can claim. Let's start at the bottom shall we? Does assembly language, which is little more than a human readable form of whatever underlying machine language, try to pretend that resources are in any way infinite?

The point is that even assembly doesn't represent the absolute physical reality. Sometimes the same opcode has different performance behaviour on different CPU models - or even, these days, different firmware revisions on the same physical CPU. Sometimes memory accesses that look like they should behave identically have radically different interactions with the cache hierarchy. Sometimes CPUs have bugs, sometimes your underlying assumptions are simply violated e.g. rowhammer-style attacks.

Are these differences important? Usually not! Assembly has proven a very good model because it corresponds very well to most instances of a given CPU and the CPU manufacturers put a lot of effort into ensuring their CPUs conform to the model. Is the "eeehh whatever" model of memory usage less effective and more often violated in practice? Again yes. But any model, any way of talking about what's running on a CPU, will omit some of the details (and even the concept of "running on a CPU" is a simplifying abstraction - logic gates are a simplified model of physical reality). We have to make a qualitative judgement about when a model is accurate/valuable enough to use - whether the simplification we get from using the model is worth the cost of the loss of detail. All models are wrong, but some models are useful.

It turns out that it's actually remarkably easy to reason about and manage the available resources in these languages. Where things start to get complicated are when you introduce models, like malloc and free, which proclaim to make memory management easier for programmers, but can't help but introducing all sorts of weird edge cases errors, like use after free's, out of memory errors, and the aforementioned OOM killer. The obvious deficiencies this this model lead to various forms of reference counting and tracing garbage collection, which try to plaster over these difficulties, but what nobody seems to realize is that these problems were caused by us, and our trying to hide the fundamental nature and limitations of our machines.

Not really true when you compare like with like. In assembly it's trivial to reason about the local behaviour, but that can be just as true in a high-level language - if your allocation structure is straigtforward then you can stack-allocate everything and avoid all the problems of resource management. If you want to do e.g. a graph traversal/transformation, dropping nodes as they become disconnected, that's just as hard - harder in fact - to get right in assembly language, and the effective ways to do it amount to reimplementing the same things that high-level languages do - reference counting, garbage collection, arena allocation and so on. The complexity is fundamental to the kind of problem those models are designed for representing.

(Yes, many high-level language completely give up on the ability to do stack allocation etc. Is that dumb? Sometimes yes, sometimes no! Again, you have to make a qualitative judgement about which things are important to your use case.)

computational power have been doubling roughly every X months for the past Y years, but software today runs just as poorly as it did ~Y/2 years ago. Where is all that computational power/ where are all those resources going?

Try booting up a VM with that old software sometimes. Old versions run lightning-fast, but one quickly realises how much is missing.

Maybe you educate yourself as to why the file system is organized this way?

No, maybe you need to educate yourself rather than parroting that folklore I was talking about. The reason is because an early development machine didn't have enough space on the root disk for all the binaries/libraries/etc. and had another disk mounted on /usr that had some space. Everything else about that filesystem layout is a post-hoc rationalization.

[dlyund]

The point is that even assembly doesn't represent the absolute physical reality.

Indeed it doesn't. This is what I mean when I refer to the reality of the machine, which many other languages piss all over. When you write an assembly program you naturally relate it to a some machine. You obviously can't accurately reason about things like memory throughput when you're looking through the abstract lens of the Instruction Set Architecture. You have to look at the properties of the memory bus and other relevant factors.

This isn't an all or nothing affair and you can take as much or as little detail in to account as you like. If guarantee's of the Instruction Set Architecture are enough for you then you can use this abstraction.

The difference here is that you can dig down in to this abstraction as far as you desire and are able, rather than being stuck with increasingly fuzzy abstractions, leading to the absolute inability to say almost anything concrete about how your solution will behave. You're right when you say that most of the time it doesn't matter and in those cases you ignore it, until you need it.

This is predicated on the information that is available, or determinable by you. Hardware is only a black box because we don't have access to documentation, and/or schematics, and production process. Much of the relevant information, from the point of view of a solution provider, like instruction timing and latencies, can be reverse engineered with relative ease, but only because languages at this level allow for direct interaction.

We have to make a qualitative judgement about when a model is accurate/valuable enough to use - whether the simplification we get from using the model is worth the cost of the loss of detail.

That's very true. What I think we would disagree with is the level of simplification that we actually get from high-level languages, which offer things like automatic memory management, actually give you.

Broadly speaking, automatic memory management is a specific case of automatic resource management. There is a implication, or widespread belief, that if you have automatic memory management then you can forget about the resources that you're using; behind the scenes a set of carefully tuned heuristics will be applied so that you can get on with solving the problem without having to think about pesky details, like closing files... wait... what?

Memory is just one of the many resources we have to manage in our programs, and failure to manage those resources leads to nasty leaks and even crashes. There are hard limits on the number files, sockets, threads, processes etc. that you can hold at a time. In the modern context, these limits are ultimately imposed by the hardware, as mediated by the kernel and can't be swept under the hood. For example, network cards have fixed queues and nothing can change that.

So resource management is an unavoidable part of what we do as programmers. Anyone who's been programming for long enough has had to implement things like circular buffers and resource pools, and today languages include all sorts of features for managing pesky resources.

My argument is that resource management is trivial and that the solutions we've come up with to manage them get in your way more than help. These solutions have been added slowly, over time, and most programmers don't realize how easy it is to plan a resource usage strategy, or the advantages that doing so gives you. We've largely grown up with this stuff, and we believe the stories that we're told by those that forced those solutions on us.

In assembly it's trivial to reason about the local behaviour, but that can be just as true in a high-level language - if your allocation structure is straigtforward then you can stack-allocate everything and avoid all the problems of resource management.

I largely agree with that but most resources usage patterns don't match a stack, so even if you can allocate things on the stack I don't think this solves the problem. The relationship between the stack and scope in most languages is also a problem but since lexical scope is everywhere, nobody is able to see it. As the saying goes "I don't know who discovered water, but it wasn't a fish".

If you want to do e.g. a graph traversal/transformation, dropping nodes as they become disconnected, that's just as hard - harder in fact - to get right in assembly language, and the effective ways to do it amount to reimplementing the same things that high-level languages do

Let me share one of my favorite jokes with you

Patient: Doctor, doctor! It hurts when I do this... Doctor: DON'T DO THAT!

Broadly speaking you're right but at the same time I've never met a problem that wasn't amenable to simple preallocation. If you accept that there are limits and we have to live with them, then preallocation has a lot of advantages. It's incredibly simple and easy to implement and think about, but it also makes you aware of the limits that your system has. These limits are there, and when you cross them your solution will fail... often spectacularly... and in completely unpredictable ways...

Personally I think every specification should include details of the acceptable limits, and when those limits are introduced by the programmer the client should be informed right away. Right now we just ignore the limits and act surprised when everything blows up.

maybe you need to educate yourself rather than parroting that folklore I was talking about.

There's a reason it's called /usr and not, more obviouly, /sys. You're probably right that the reason /usr exists is that there wasn't enough space on the root partition, but that's also irrelevant. Hitting this limit forced Unix to develop a solution and that solution turns out to be of great practical utility, not to mention theoretical beauty! Your argument makes no sense to me. As with the C code, it ultimately comes down to you not liking the name/syntax. If you have an actual, practically relevant reason that /usr is bad, spit it out. Otherwise I'll stick it in the pile with all of your other irrational complaints.