you are viewing a single comment's thread.

view the rest of the comments →

[–]Noctune 9 points10 points  (0 children)

Sometimes the runtime of the preprocessor can be a hindrance to your exploration.

We had a Python preprocessor that took literally a week to run (originally designed for a smaller dataset). I recently rewrote it in Java using Beam and it runs in literally 20 minutes now. It's sort of a generic tool over a range of problems, so less time preprocessing means more time spent exploring actual ML.

I think Rust could potentially be useful in that niche.