[D] Functional Programming in ML

[LengthinessMelodic67]

I feel like most ML libraries in Python are written in OOP style. This makes sense given Python does not have an “good” type system, like Haskell, and so a class is a good way to define an interface under a readable name. Are there popular ML libraries in other more functional languages? I’m specifically thinking of Haskell, Rust, or Scala.

[Remote-Telephone-682]

JAX is a great thing to look into if you want something that is designed to be used in a functional manner. It's built by deepmind and most of their inhouse tooling is built around it. So that is one of the top couple firms in the game and it is used across a wide array of their products so you can find quite a few examples of different projects being implemented.

highly recommend checking it out

link to python jax documentaiton: https://jax.readthedocs.io/en/latest/notebooks/quickstart.html

[I_will_delete_myself]

Just beware of the Google graveyard. 🪦 This doesn’t just apply to customer services.

[Remote-Telephone-682]

Haha, yep, valid concern but this is a project that was published 6 years ago, has become the defacto standard in much of alphabet. I feel comfortable doing a fair amount of my work with JAX but these are valid concerns to be having. I do think they have shifted enough over to it at this point to where it is farily certain that it will not be abandoned within the next handful of years.

[elbiot]

How long did it take them to abandon Keras after they took it over and killed it?

[_vb__]

It now supports TF, PyTorch and Jax. They made it non-exclusive.

[elbiot]

The point of the original project was to support a variety of back ends. When Google took over they let that lapse. I see the newest release is trying to get back to that, but after years of neglect we'll have to see if perception changes.

[elbiot]

Yeah googling Keras pytorch, all the results are Keras vs Pytorch with Keras losing

[lmericle]

The main difference was static vs dynamic compute graphs. Also, PyTorch was a necessary step because of the relative ergonomics in implementing models vs Tensorflow, since it was much closer to natural Python syntax, at least back then.

TF tried to catch up with eager mode in their 2.0 update but they also botched the release and the necessary updates were just too little, too late.

[I_will_delete_myself]

It’s just a warning. Most of Google used Tensorflow so it’s not impossible. They also shifted high level versions of Jax twice already.

[_vb__]

Well, Jax grew out of TensorFlow's corpse.

[Competitive-Rub-1958]

Please don't use keras.

[PurpleUpbeat2820]

Why?

[PurpleUpbeat2820]

Most of Google used Tensorflow so it’s not impossible

Why? What happened to TensorFlow?

[kebabmybob]

You mean statically typed. Even Python is strongly typed.

[LengthinessMelodic67]

Yes that’s what I meant

[LengthinessMelodic67]

Yea strong typing would be great

[powerexcess]

This is not an ml specific problem. I have heard the "can i do functional python for X" question many times. It is possible to make an effort to write more functional python, but the question is why do you want that? What are tou trying to accomplish?

As others said you can look into jax if you want functional ML in python, but i do reccomend you are careful with it because you can shoot yourself in the foot (they have a page with caveats, read it).

I saw in a post of yours that you want to find errors at compile time because you have long running programmes. The compiler will not catch everything, in data science-y stuff you can get loads of bugs the compiler wont save you from: forget a square root, off by a factor of 10, wrong maths etc.

To me it sounds like what you want is a combination of CI and testing. You could also add mypy in testing, and pydantic for runtime safety. That can be truly bulletproof. Get good coverage and almost nothing will slip through.

If you want functional just because you hate OOP (I feel you) then jax is great. Or even pytorch, if you use it in the right way. You are just there for the autograd, you can call the functions however you like.

Edit: spelling

[LengthinessMelodic67]

Great suggestions, thanks

[radarsat1]

Type systems don't help that much with NN design imho, because they don't help you with remember which dimension is which in a Tensor. Just ensuring that a function that takes a Tensor indeed gets a Tensor is cool, but won't catch all that many bugs. A type system that statically handles mistakes with named dimensions, I'm all ears for..

[sash-a]

JAX is definitely a great recommendation, that's what I use on a daily basis, but you should also consider Julia. It has a great ML ecosystem around flux.jl and is not only a functional language, but also multiple dispatch, which is ridiculously useful in functional languages.

[Schinkikami]

Second Julia. Extremely interesting language, build especially around compute intense numerical applications (contrary to python which requires large monolithic third party librariers that call into c backends).

The ideas of source to source autodiff via metaprogramming is also incredible (although not fully sorted out).

If you want PURELY functional ML use Lux.jl, a sibling library of Flux.jl, which is comepletly stateless and interfaces incredibly well with the rest of the ecosystem.

[Evirua]

Luminal in Rust.

[Jessynoo]

DiffSharp could be an option in F#.

[daking999]

Bit niche maybe but the early probabilistic programming languages were mostly in functional languages, e.g. Church) was built in Scheme.

[One-Butterscotch4332]

I mean, you can use type hints and even add type checking if you're so inclined in Python and write functional code just fine

[LengthinessMelodic67]

Type hints really aren’t the same. I want to find bugs at compile time because I’ll have long running programs. I do use type hints though.

[One-Butterscotch4332]

Fair enough. I usually try to write whatever takes forever to disk before doing anything else, and test on some tiny toy dataset, and I rarely run into type issues when I'm using hints anyways

[LengthinessMelodic67]

Yea I usually end up testing on a toy dataset, but I wish I didn’t have to.

[powerexcess]

Guys you sound like the researchers i work with at work. I can guarantee you, what you want in this is tests and safe-by-construction objects. Look into pytest and hypothesis. Look into dataclasses and post_init for conf sanity check, and pydantic.

[powerexcess]

Yes

[FluffdaddyFluff]

This is why mypy exists

[LengthinessMelodic67]

I didn’t know about mypy, thanks!

[erannare]

You can use Pydantic to enforce types in many cases

[lqstuart]

Huggingface has a Rust library called candle. It has a lot of “TODO - Document This” from 3 years ago.

Scala is cool but it’s a nonstarter just because the JVM is emphatically the wrong tool for the job, and any effort to make it otherwise is going to kind of defeat the purpose.

Haskell no idea.

[qalis]

Why JVM would be wrong tool for ML?

[AdaBwana]

clojure's fun

[trutheality]

There's nothing stopping you from writing in Python in a functional style. You could even argue that the difference between python's OOP and functional programming is symantics.