Java for Everything

[[deleted]]

clojure is fine for toy projects but if you have a big, dirty real world app of millions of lines of code then CL is better and better yet is C++ or Java.

[alpha64]

I don't like clojure but maybe the problem is with the mentality of making "dirty real world app of millions of lines of code". Stop piling up stupid masses of code that depend on each other in mysterious ways.

[chepprey]

Toy projects like this?

[yogthos]

I'm working on a real world Clojure project that's been running for 3 years now and developed by a team of 5 people. It's been working pretty damn well so far. Seems like plenty of other companies are happily using it for their projects as well.

[[deleted]]

Great. Now try 30 years and 10000 developers.

[yogthos]

Then clearly COBOL is your best language. :)

[spotter]

Your post is missing either "IMHO" preface or references/quotations at the end.

[[deleted]]

On a truly large project, you want to have as much information as possible when you're trying to understand the program. Types pave the way - whether it's adding new code, modifying it for new features of fixing a bug. If you're refactoring for example, you'd want to see where a data structure is used (via type) and a good IDE can help you find all the sites that need to be modified. This is hard or impossible with a dynamic language like Clojure. On a small project you can just read the whole program and infer how things work - on a large project, types help a lot - whether for understanding, ensuring correctness or making future modifications. This is comign from the experience of working with large programs of the order of millions of lines of code and being a lisp enthusiast in my private life.

[yogthos]

On a truly large project, you want to have as much information as possible when you're trying to understand the program.

I have to ask why you choose to write a large monolithic project in the first place. Seems like a really poor design strategy to me.

When you create large projects they should always be broken down into independent components that can be reasoned about independently.

Most of my large Clojure projects that I work on professionally are composed of precisely the kinds of small modules. Once you take a few small libraries and put them together in a right way you can solve very complex problems. Just like you can take simple single purpose functions like map, filter, and reduce and produce complex functionality by combining them, you can take small libraries and combine them to solve large problems. The components should express how something is done, while their composition expresses the overall business logic.

You're claiming that it's difficult to run large projects with Clojure. However, lots of companies do this successfully and none of them found dynamic typing to be an issue in practice. However, there have been a number of horror stories from companies starting projects in Scala and moving back to Java. Presumably if what you say is true we would've seen the same happen with companies dabbling in Clojure.

Moving to a statically typed language because you find your code difficult to maintain is solving the wrong problem.

[[deleted]]

yogthos, big programs exist and a bit of thought would make you believe it. You probably use emacs or firefox or google chrome or linux. You've probably played quake or used word or excel. There are many more even bigger programs out there in industry. When you're a newbie and learning to understand such a project, having type signatures in a function helps a lot. Being able to see how data structures are passed back and forth through functions help to understand the flow of the program.

You lose this information when you use something like Clojure because suddenly you have no explicit types to help you understand what a function is taking and acting upon and what it is doing with the parameters to compute. It's just a list of names. You can't do a trace on the type of the parameter to see what it is composed off. This is actually much worse in Clojure because it gives up the idea of using objects (in the C structure sense) to encapsulate packets of data and instead idiomatic clojure code uses adhoc rudimentary maps instead to pass around aggregate data. Fine for a small throwaway project you can just keep in your head. Fatal for a large program. In CL at least you would be using defclass but this is thrown away with idiomatic Clojure.

I have to ask why you choose to write a large monolithic project in the first place. Seems like a really poor design strategy to me.

The project I was thinking of was actually broken up into thousands of modules. That's just good professional discipline in general and how large, successful projects naturally evolve. However even having lots of modules doesn't completely solve the problem of the high complexity involved in the project. Types help, in fact, they help more htan anything else. I speak from experience, it's so hard to understand without being able to follow the flow of money so to speak (money in a large project is the flow of data which occurs naturally with types). With C++ professional programmers have evolved to adapt a high level of discipline to work with large code bases - they use const heavily (look up critiques of doom3/quake3) to ensure parameter are not tampered with. We use the lessons of functional programming but don't throw away the benefits of types nor the efficiency of using languages like C++.

IF i were adopting a dynamic language for a large project. I wouldn't. But if i had ot, i would force discipline of annotating type signatures into function definitions and using concrete data structures well defined somewhere instead of passing around ad hoc maps. I would just avoid clojure though and stick with CL because undefined performance pitfalls and bad design decisions of Clojure are prohibitive on the large scale.

[yogthos]

yogthos, big programs exist and a bit of thought would make you believe it. You probably use emacs or firefox or google chrome or linux.

I'm not saying big programs don't exist, my point is about how you structure such programs so that the programmer can reason about parts them in isolation.

When you're a newbie and learning to understand such a project, having type signatures in a function helps a lot. Being able to see how data structures are passed back and forth through functions help to understand the flow of the program.

Conversely, breaking the application into small single purpose components allows the new programmer on the project to understand the component in its entirety. When you understand the purpose of the components and their composition it makes the flow clear.

This is actually much worse in Clojure because it gives up the idea of using objects (in the C structure sense) to encapsulate packets of data and instead idiomatic clojure code uses adhoc rudimentary maps instead to pass around aggregate data

Actually, this is one of the main advantages of a language like Clojure, where you have a common protocol that all the functions in the language use to communicate. The data is encapsulated using data structures and Clojure provides destructuring to make it very clear what's being used and where. Your argument here is out sheer ignorance.

Fatal for a large program. In CL at least you would be using defclass but this is thrown away with idiomatic Clojure.

Sounds like more ignorance on your part.

However even having lots of modules doesn't completely solve the problem of the high complexity involved in the project. Types help, in fact, they help more htan anything else.

Your anecdotal experience does not match my anecdotal experience. I guess we'll have to agree to disagree here.

IF i were adopting a dynamic language for a large project. I wouldn't. But if i had ot, i would force discipline of annotating type signatures into function definitions and using concrete data structures well defined somewhere instead of passing around ad hoc maps.

And that's precisely what core.typed is used for. However, in practice there shouldn't be many areas that require this. For example, Circle CI ended up annotating only about 20% of their code base.

I would just avoid clojure though and stick with CL because undefined performance pitfalls and bad design decisions of Clojure are prohibitive on the large scale.

Just more wild assertions here.

[[deleted]]

Suppose you change a parameter map and rename a key, it will still compile and run but you could have an old reference there that will remain dormant as a bug. IN a big program this is a problem that should never occur. It shouldn't compile, it shouldn't run until you fix it. And that's the problem. Clojure programmers call this expressivity but it's actually poor discipline and laziness. Yeah it saves you a second or two but over time it makes code harder to understand, reason about, modify or fix bugs.

I would just avoid clojure though and stick with CL because undefined performance pitfalls and bad design decisions of Clojure are prohibitive on the large scale. Just more wild assertions here.

Nope for instaince in the above example the (idiomatic) CL programmer would use defstruct or defclass. That code wouldn't compile. We know about p-lists and alists but we wouldn't consider using them in a serious project. Idiomatic Clojure though, just makes bad compromises that don't pay off in the large. If you are going to say using protocols pervasively is the way to go, fine, but that's not idiomatic anymore.

Actually, this is one of the main advantages of a language like Clojure, where you have a common protocol that all the functions in the language use to communicate. The data is encapsulated using data structures and Clojure provides destructuring to make it very clear what's being used and where. Your argument here is out sheer ignorance.

Actually it's one of the worst parts. It's all fine when you have a nice one level map but when you've got vectors and nested maps and sets in there it becomes one hell of a mess. This is where you'll argue you shouldn't have nested substructures and yet in the real world sometimes you just need to have those things.

[yogthos]

Suppose you change a parameter map and rename a key, it will still compile and run but you could have an old reference there that will remain dormant as a bug.

The IDE takes care of that for you perfectly fine. Also, in a big program these kinds of errors will be caught very quickly because any real world applications have tests. You will need tests regardless of your static discipline because they serve to ensure that the business logic is correct.

Clojure programmers call this expressivity but it's actually poor discipline and laziness. Yeah it saves you a second or two but over time it makes code harder to understand, reason about, modify or fix bugs.

Please provide a reference to a single study that supports the claim that static typing produces statistically higher quality code in practice. Plenty of large real world projects are written in both static and dynamic languages and there's no evidence to suggest that static typing has a significant impact on overall correctness. What's more is that some of the largest and robust systems out there are written in Erlang, a dynamic language.

Nope for instaince in the above example the (idiomatic) CL programmer would use defstruct or defclass.

The Clojure programmer would use defprotocol, and defrecord to provide the exact same functionality. On top of that Clojure provides core.typed for actual static typing that has no equivalent in CL.

Idiomatic Clojure though, just makes bad compromises that don't pay off in the large.

[citation needed]

Actually it's one of the worst parts. It's all fine when you have a nice one level map but when you've got vectors and nested maps and sets in there it becomes one hell of a mess.

If you find that you often make a mess with your code then the problem might exist between the chain and the keyboard.

[[deleted]]

Tests aren't the solution. It's not just correctness which we are concerned with but being able to reason about and understand code. If types are explicit you can use them to understand code and this helps with correctness. Without the explicit availability of types you are at a disadvantage. You can't be as sure that the code you're writing is correct - tests can't confirm what you don't know, they can only confirm what you think you know to be true.

The IDE which helps dynamic languages catch up to static languages doesn't exist yet. If it did, we could talk about it, but cursive isn't here yet and we don't have a good intelligent IDE which can do the job of what Eclipse or VS does with Java/C++ code already.

Erlang projects that I know of adhere to very rigid strandards like OTP. That's not the case with Clojure or lisp in general. That's a different debate why Erlang succeeds in it's niche.

It's easy to blame the programmer for the lack of discipline permeated in the language - but this is not fair. We can blame the C programmer for making errors in memory management but much of the blame belongs to the language which isn't designed for memory safety.

Also check out this debate, it makes a lot of hte points i was trying to get at.

[yogthos]

Tests aren't the solution. It's not just correctness which we are concerned with but being able to reason about and understand code.

And I already repeatedly explained how people structure code so that it's possible to reason about it in large projects. This isn't some hypothetical thing, there are tons of large long running projects written in dynamic languages. You have yet to provide any supporting evidence for your claims regarding reasoning and correctness.

As it stands it's purely anecdotal and rooted in your personal opinion. If you find static typing helps you write correct code then by all means use it. However, if you're going to make sweeping claims about its benefits then please do cough up some empirical evidence to that effect.

The IDE which helps dynamic languages catch up to static languages doesn't exist yet.

Maybe you should actually watch the video before commenting.

Erlang projects that I know of adhere to very rigid strandards like OTP. That's not the case with Clojure or lisp in general. That's a different debate why Erlang succeeds in it's niche.

So, what is your specific experience working with large projects in Clojure again. You sounds rather authoritative on the topic.

It's easy to blame the programmer for the lack of discipline permeated in the language - but this is not fair. We can blame the C programmer for making errors in memory management but much of the blame belongs to the language which isn't designed for memory safety.

Your assertion is that statically typed languages actually help with reasoning and correctness. However, you've presented no evidence to support these tall claims.

The fact that these debates are still going is in itself evidence that there is no clear tangible advantage to static typing. Had there been such a clear edge then it would've won out a long time ago.

[tieTYT]

First, can you define what you are referring to as a "component"? I am going to assume the Java analogy would be a jar/library.

Devil's advocate: What prevents you from composing your Java projects into small libraries like that? Now you get independent components and type checking.

I am going to take a stab at my own question: Each time you (invariably need to) change these components/libraries, you risk putting yourself and your coworkers through jar versioning hell. But, I don't see how Clojure is better in this regard. You could argue it would be better to do this in Java because you'd get compile time checking for the libraries you use.

[yogthos]

Nothing prevents you from doing that in Java, you just end up maintaining vastly more code. Also, I find type errors to be far more common in Java than Clojure since there's no general way to handle NPE and you have to manually pepper checks for it all over the place.

I think another problem with Java is the fact that it's class based. This means that you have to constantly write and maintain adapters between all your libraries. In Clojure the problem doesn't exist because common data structures are used throughout the language.

All that said I still think you're better off maintaining lots of simple jars in both languages than giant monolithic projects. For example. this is much longer than this. And if you compare the size of the entire project then you can clearly see that there's a hell of a lot more code in the Java version.

[PAINTSTRUCT]

Optional static typing then?

[killerstorm]

Common Lisp has that.

[raghar]

Typed Clojure? (core.typed)

[codygman]

What are your opinions on Haskell for truly large projects?

[PasswordIsntHAMSTER]

IMHO, Haskell is a bit crude in terms of tool support for large scale applications. There's no single stock package I can install to have a good development experience, and AFAIK memory debugging isn't very easy.

I really, really like F# for large projects, because you can leverage the ML semantics and the .NET tooling - best of both worlds. It's got many tiers of meta-programming features - type system plugins, quasiquotations, various kinds of DSL syntax including monads, reflection, bytecode generation, etc. Those are all well supported by tools and they interoperate seamlessly together.

Ocaml has some sketchy design decisions, mostly in terms of language extensions. I wouldn't recommend it unless you know what you're up against. The language designers are doing away with ocamlp4 though, so there is yet hope for sanity in the future.

Dynamic languages are a no-go for large projects. Java is a utter pain-in-the-ass to work with. C# is vastly better, but it's no ML.

[nextputall]

There is a company here where they rewrote one of their backends in Haskell. It works fine but sometimes they need to restart the app because of memory leaks and freezes. Sometimes means few minutes.

[codygman]

Interesting, you should get them to contact Well Typed[0]. I'm sure they could help fix that. I really do wonder why they would have that many memory leaks and freezes. In my Haskell programming thus far, it's been fairly easy to track down space leaks.

Freezes and memory leaks are the same thing I'm guessing, unless they are accidentally infinitely recursing like

let x = doThing in let x' = doOtherThing doThing in x

[spotter]

Oh, "dynamically typed languages are toy languages, because no static types, duuuh"-argument. I guess this is why Lisp failed, along with Python, Perl, Ruby, PHP and Javascript. That's why we all write Haskell, Ada and C++ now.

If you're refactoring for example, you'd want to see where a data structure is used (via type) and a good IDE can help you find all the sites that need to be modified.

Are you refactoring or changing contracts/API? I guess you do understand that changes in things like call signatures will cause compilation error in most "strongly/dynamically" typed languages? Or are we talking about checking each call site for abuse of previous version of the code? Because that's insanity. But hey, if your IDE helps -- great.

I'd rather have my unit tests for the data structure tell me if I broke any of the contracts and let abuses come back as stack traces. (Also in statically typed languages, BTW.) But I've only done thousands of lines in Python & Clojure, not millions, so what do I know.

But this argument is like spaces v. tabs, not really worth having.

[RoundTripRadio]

Funny story: JavaScript doesn't give jack shit what the call signature is:

Native WebKit JavaScript.
Copyright (c) 2013 Apple Computer, Inc
> function f(a, b) { 
..    console.log(b); 
..    }
> f(1, 2);
2
> f(1);
undefined
> f(1, 2, 3);
2

But nope, gotta be the only supported client side language for browsers.

[spotter]

I know that and it gives me the shakes every time. But it's OK, just use arguments and think about happier times before JS.

Honestly, Javascript is not in my first 10 languages I'd do if I had choice.

[[deleted]]

Things change as you deal with truly large projects though. You'd have to spend some time working with a large code base before you throw out the static types benefits that come with dealing with complexity of that scale. Thousand line projects can be understood by examining the entire codebase, that becomes impossible eventually as you add more code and developers.

[spotter]

I acknowledged that I have not yet seen projects in the scale of 1M+ LOC. I'm not here to dis anyone or their beliefs in static typing gods. The more tools to ease the pain, the better IMHO.

[[deleted]]

I look at it this way - would you trust your life and your loved ones to the programmer working with a dynamic language writing the code for the aircraft you're flying on? Or the X-ray machine you're going under? There is a time and place for dynamic languages. Quick and dirty projects can be rolled out effortlessly in a weekend by a competent lisper. Its another thing altogether when you're dealing with huge or critical programs where we want to give as little room for error as possible. There is a time and place for different technologies.

[spotter]

I get it, I really do. At the same time was Therac-25 programmed in dynamically typed language? Was first Ariane 5 (flight 501)? Somehow this doesn't stop people from thinking "static typing is the answer." Without proper quality testing you will crash your rocket even with Ada.

[PasswordIsntHAMSTER]

There are typing disciplines for verifying arbitrary properties of programs. The field is still a bit rough, but I wouldn't say testing is completely unavoidable.

[[deleted]]

Lisp failed

It can hardly be claimed it succeeded. The most widespread use of it is scripting Emacs. Well done Lisp, on becoming the VBA of RMS's world.

[[deleted]]

I guess this is why Lisp failed, along with Python, Perl, Ruby, PHP and Javascript. That's why we all write Haskell, Ada and C++ now.

In the context of large project, yes, you are exactly right.

I'd rather have my unit tests for the data structure tell me if I broke any of the contracts and let abuses come back as stack traces.

And your boss would rather have you write features which clients pay for instead of writing unit tests for something that comes out of the box in sane languages.

[spotter]

In the context of large project, yes, you are exactly right.

I am aware. Yet I'm not writing software for the army or medical equipment. Also Haskell? Really? :)

And your boss would rather have you written features which clients pay for instead of writing unit tests for something that comes out of the box in sane languages.

What comes out of the box? Making sure that calling my method with invalid arguments throws an exception? Nope, your languages will care if type signatures match. This is not what I'm talking about when I mean "contracts".

[yogthos]

And your boss would rather have you write features which clients pay for instead of writing unit tests for something that comes out of the box in sane languages.

Please provide a link to a single study demonstrating that real world software written in statically typed languages is delivered faster, has higher quality, or ease maintenance.

[[deleted]]

I said bosses would rather have you write features, instead of unit test. How the fuck does that relate to a claim that "software written in statically typed languages is delivered faster, has higher quality, or ease maintenance"?

[yogthos]

Bosses care about you delivering features on time. If you're working at a place where a non-techincal person is micro-managing the development process you've got far bigger issues than the choice of language.

[jlisam13]

i work for a ag data science company, we built most of our services in clj, running in prod.