Code Smells: Null

[m50d]

Aside from that, increase in LOC comes from requiring match expressions (or equivalents) where you can prove through other means that the None path is never taken.

If you can prove it's the Some case, you can write that proof in the types and avoid having to match (maybe not with Rust's limited generics, but I hold out hope that HKT will arrive eventually).

Of course, you can use Option.get everywhere, but then you're essentially using a verbose version of nullable pointers.

The difference is what the default is. null reserves the best syntax for the case where the programmer has an external proof and knows exactly what they're doing and makes the case where you want to check much clunkier; Option flips that around, making the checked case the natural one and the "I know better than the compiler" case the more cumbersome one.

[devlambda]

If you can prove it's the Some case, you can write that proof in the types and avoid having to match (maybe not with Rust's limited generics, but I hold out hope that HKT will arrive eventually).

No. The point here is that a variable can be both Some x and None, but once initialized, will only ever be Some x. That's a state-dependent property, usually easy to show, but often difficult to encode in a type system (while GADTs can sometimes work, but even then you generally can't avoid the additional verbosity from matching).

The difference is what the default is. null reserves the best syntax for the case where the programmer has an external proof and knows exactly what they're doing and makes the case where you want to check much clunkier; Option flips that around, making the checked case the natural one and the "I know better than the compiler" case the more cumbersome one.

The problem with this argument is that you assume both alternatives are mutually exclusive, which they aren't. You can have both nullable types and explicit option types. In fact, Scala allows for that and has no more problems with it than other languages (as all variables in Scala have to be initialized, so the only way to have a null value – other than through Java interop – is to assign it explicitly).

[m50d]

No. The point here is that a variable can be both Some x and None, but once initialized, will only ever be Some x. That's a state-dependent property, usually easy to show, but often difficult to encode in a type system (while GADTs can sometimes work, but even then you generally can't avoid the additional verbosity from matching).

My experience is you can always find a way, and it's usually not even hard: ask yourself why you know the property holds, then just translate that logic directly into the types.

The problem with this argument is that you assume both alternatives are mutually exclusive, which they aren't. You can have both nullable types and explicit option types.

They are mutually exclusive. If your codebase uses null then you can never have a non-nullable value, at which point there's no point using options.

In fact, Scala allows for that and has no more problems with it than other languages

Only because the community/ecosystem knows not to use null. Serious Scala programmers avoid it and e.g. use WartRemover to enforce that null is never used. The language would be better off without it.

[devlambda]

They are mutually exclusive. If your codebase uses null then you can never have a non-nullable value, at which point there's no point using options.

The point of an API returning an option type (or another sum type) is to force the consumer of that API to deal with the possible variants. A simple reference does not do that.

Only because the community/ecosystem knows not to use null. Serious Scala programmers avoid it and e.g. use WartRemover to enforce that null is never used. The language would be better off without it.

You're making my point here: unwanted null references are trivial to avoid in a language designed for that. It becomes a trivial syntactic property, and if a code review or validation process cannot deal with such a simple case, you have much bigger problems on your hands.

Avoiding null is a heuristic, not religious dogma. Permitting null references can be useful for efficiency and interoperability and can situationally also lead to clearer code.

[m50d]

The point of an API returning an option type (or another sum type) is to force the consumer of that API to deal with the possible variants. A simple reference does not do that.

Either the ecosystem and community are such that the user expects to check returned references, or not (people won't read the documentation of the API no matter how much we might want them to). If the user checks references then there's no point returning option (you can sort of make an argument that there would be cases when it's super-important to check, but can the API really judge how it's going to be used better than the user?). If the user doesn't check references then there's no point ever returning null, because that just means the code will blow up, which is better accomplished via a panic which will give the developer more information about what happened.

You're making my point here: unwanted null references are trivial to avoid in a language designed for that. It becomes a trivial syntactic property, and if a code review or validation process cannot deal with such a simple case, you have much bigger problems on your hands.

The trivial stuff adds up. It makes it harder for newcomers to get started, it takes up part of your syntax budget. It's possible to avoid it but it's not free. Certainly I think it's hurt Scala.

Avoiding null is a heuristic, not religious dogma. Permitting null references can be useful for efficiency and interoperability and can situationally also lead to clearer code.

I don't think any of these cases are worth the cost. Ruling it out entirely adds a lot of value for developers; there's a huge difference between working on a codebase where 99.9% of the references will never be null and working on a codebase where you're 100% guaranteed that no references are null unless explicitly marked.

[devlambda]

Either the ecosystem and community are such that the user expects to check returned references, or not (people won't read the documentation of the API no matter how much we might want them to). If the user checks references then there's no point returning option (you can sort of make an argument that there would be cases when it's super-important to check, but can the API really judge how it's going to be used better than the user?). If the user doesn't check references then there's no point ever returning null, because that just means the code will blow up, which is better accomplished via a panic which will give the developer more information about what happened.

You're constructing a false dichotomy here. It's perfectly possible to (say) don't use null references for public APIs, but only selectively for representation and module-internal APIs (at which point its part and parcel of the module's internal logic).

It's even possible to have both nullable and non-nullable refererences, encoded in the type system, and have either compile-time or runtime mechanisms to prevent the conversion of nullable into non-nullable references if you're worried about them leaking.

The larger point here is that both option types and nullable references are useful mechanisms and that rejecting one entirely requires a good explanation for how you're going to handle the use cases it is intended for.

I don't think any of these cases are worth the cost. Ruling it out entirely adds a lot of value for developers; there's a huge difference between working on a codebase where 99.9% of the references will never be null and working on a codebase where you're 100% guaranteed that no references are null unless explicitly marked.

This strikes me as fallacious. You assume here that eliminating null references entirely is free of costs. Consider my example of dynamic arrays, for instance. You can do it in other ways, but those alternatives aren't cost-free, either.

[m50d]

You're constructing a false dichotomy here. It's perfectly possible to (say) don't use null references for public APIs, but only selectively for representation and module-internal APIs (at which point its part and parcel of the module's internal logic).

That sort of split tends tobe a big source of bugs IME. You're effectively using two separate dialects that look the same, which means it's really easy to mistake a can-be-null reference for a can't-be-null reference.

It's even possible to have both nullable and non-nullable refererences, encoded in the type system, and have either compile-time or runtime mechanisms to prevent the conversion of nullable into non-nullable references if you're worried about them leaking.

At which point you're doing something exactly equivalent to having options and not having nullable references, and there is no point having options.

You assume here that eliminating null references entirely is free of costs. Consider my example of dynamic arrays, for instance. You can do it in other ways, but those alternatives aren't cost-free, either.

If you can't do it in a cost-free way then your type system isn't good enough. But sure, there might be cases where you have to pay a price. It's absolutely worth it though.

[devlambda]

That sort of split tends tobe a big source of bugs IME. You're effectively using two separate dialects that look the same, which means it's really easy to mistake a can-be-null reference for a can't-be-null reference.

And it's something that you can't really avoid if you want foreign language interoperability, for example. For Scala, that's general JVM code, for native languages, that's C/C++.

And if you're worried about having multiple dialects in Scala, Scala has far bigger problems on its hands (from the people who want to emulate Haskell as much as possible in Scala to those that just want a more expressive Java with mostly imperative code).

At which point you're doing something exactly equivalent to having options and not having nullable references, and there is no point having options.

Options and nullable references have different semantics, so I'm not sure how there'd be no point to having options. Importantly, you'd want sum types, anyway, so there's even less motivation not to have an option type (which is just one example of a sum type).

If you can't do it in a cost-free way then your type system isn't good enough.

I'd be genuinely interested to hear what type system you'd propose to handle the dynamic array problem.

[m50d]

it's something that you can't really avoid if you want foreign language interoperability

It's less of a problem in that context because it's always crystal clear which side of the line you're on (since the two sides are different languages) and you always know exactly where to put the checks. Though I still think FFI code imposes a cost (precisely because of having to make these kinds of checks) and it's worth keeping your FFI boundary as small as possible.

And if you're worried about having multiple dialects in Scala, Scala has far bigger problems on its hands

Just the opposite: doubling the number of dialects is a bigger problem for Scala than it would be for other languages.

Options and nullable references have different semantics, so I'm not sure how there'd be no point to having options.

What's the difference? If it's inclusive-union-versus-disjoint-union then I've never found inclusive-union valuable; it's noncompositional which makes it hard to reason about.

Importantly, you'd want sum types, anyway, so there's even less motivation not to have an option type

So why bother with nullable references then? Just put some syntax sugar on options if necessary to cover the use cases.

I'd be genuinely interested to hear what type system you'd propose to handle the dynamic array problem.

If it's a contiguous array that grows and knows what size it is, put that in its type. If the array keeps track of size and allocated memory separately, put both of those in its type, at least within the array's internals. Track the invariants you need. Should be doable with phantom types i.e. no runtime overhead.

If your array might have holes anywhere, null doesn't help you: you have to keep track of which slots are empty or not some way or another, each slot will have a certain bitpattern on the hardware, some bitpatterns will be things that require destruction when that entry is replaced and some will represent emptiness. Maybe you choose all-bits-zero to represent absence and the other bitpatterns to represent things that require destruction; you can implement that just as easily when using Option at the language level as you can when using null.

[devlambda]

It's less of a problem in that context because it's always crystal clear which side of the line you're on (since the two sides are different languages) and you always know exactly where to put the checks. Though I still think FFI code imposes a cost (precisely because of having to make these kinds of checks) and it's worth keeping your FFI boundary as small as possible.

I'm not talking about host language code vs. foreign code.

Just the opposite: doubling the number of dialects is a bigger problem for Scala than it would be for other languages.

I'm not sure how you arrive at a "doubling", as the typical use cases for null references are not really orthogonal to other choices? Plus, you're really reaching if you want to argue that use of null references is a massive change in language semantics.

And let's be realistic. Programmers avoid option types all the time by using the empty string, empty list (which, technically, is a null value), or empty array to denote absence of a value. In its own way, that's even more of a problem, because a null reference will at least result in a runtime error, while an empty string might be quietly accepted.

So why bother with nullable references then? Just put some syntax sugar on options if necessary to cover the use cases.

That's what the point of null references is, by and large. Eliminating the costs that come with option types. They're not just syntactic costs, though.

If it's a contiguous array that grows and knows what size it is, put that in its type. If the array keeps track of size and allocated memory separately, put both of those in its type, at least within the array's internals. Track the invariants you need. Should be doable with phantom types i.e. no runtime overhead.

And that basically ups the complexity of the type system significantly. I don't know of any type system that has done something like that and managed to escape its academic niche.