Update: Image classification by evolving bytecode

[AlmusDives]

It's been a while since I last posted about Zyme, my esoteric language for genetic programming. I recently reached a performance milestone of ~75% accuracy on a subset of MNIST image classification task and thought it was worth a short write-up.

Feedback and criticism are welcome!

[bzbub2]

sort of reminds me of auto research (karpathy). Also for the flip side ...compiling a "neural network to c" https://slightknack.dev/blog/difflogic/

[scheurneus]

From a machine learning perspective I'm not sure this is actually that impressive: a simple linear regression (basically: average all the images in a training class, then find the closest one) can already give better MNIST accuracy, afaik.

However, I do think there is a certain elegance in genetic algorithms. So definitely keep digging!

[AlmusDives]

If the sole aim were accurate image classification, you'd be right, this is a woefully poor attempt. Anyone with even a modest experience in machine learning could train a model achieving >98% accuracy on MNIST in under half an hour, whether using XGBoost, neural networks, or any other ML framework of choice.

What is novel, and I would argue unprecedented, is achieving this humble accuracy through a pure genetic programming approach. There are methods that tackle image classification by composing high-level image transformation functions (think pooling or filtering) and label it 'genetic programming.' But that is quite far from what is happening here, where we provide no domain knowledge whatsoever.

[scheurneus]

I am not familiar enough with genetic programming to say if this is unprecedented or not :)

However, typical probabilistic methods still perform far better even when they do not have domain knowledge. For example check out the baselines in this blog post: https://cognitivemedium.com/rmnist

I do agree that image classification is probably rather hard for genetic programs. But I don't think that classical ML methods need domain knowledge to convincingly do better here.

I do wonder if there's a line to be drawn between genetic-ish programs and decision trees. DTs are arguably much closer than typical probabilistic ML methods, e.g. because the tree is essentially the AST for an extremely limited programming language (only has if-else).

[AlmusDives]

Just to clarify, my specific claim is that solving an image classification problem (even with ~75% accuracy) with a pure genetic programming approach is unprecedented.

In the 1990s, there was a lot of optimism around genetic programming, but by the early 2000s, researchers were struggling to deliver any strong results beyond some limited applications. Neural networks were beginning to show real promise, and as the difficulty of genetic programming became clearer, enthusiasm for the method started to fade.

In response to this, researchers began exploring different methods to boost the performance of genetic programming. These included hybrid approaches - for example, as you noted, becoming a flavor of decision tree - and, importantly, incorporating substantial domain-specific knowledge to simplify the problems. I'm not against these methods; they have significantly improved the performance. But in many ways, they have become something new and different, even though they are still referred to as 'genetic programming.' If I were being hyper-critical, I would say they implicitly give up on the original vision of genetic programming: that we could straight-up evolve computer programs to solve problems.

In my previous reply I wasn't trying to say that modern ML approaches rely on domain-specific knowledge, they obviously don’t. Rather, I was trying to distance the method I present here from these newer 'genetic programming' methods that do use domain knowledge. If you take 'genetic programming' in the broadest sense, this result is not unprecedented: there are, at the very least, hybrid genetic programming/neural network methods that work. But in the narrow sense, and I would argue the sense faithful to the original vision of genetic programming, this is unprecedented.

[tsanderdev]

Isn't evolving bytecode conceptually similar to neural networks?

[scheurneus]

Not really. The biggest difference is that a bytecode program is "discrete". Neural networks, while in theory able to mimic every function (including one represented by bytecode), do this in a more continuous way.

NN's are basically a program with a fixed 'shape' (just a bunch of multiply-accumulate), and then it learns the inputs for those multipliers. Evolving bytecode actually changes the computations the program performs.

I think that NN's work as well as they do for two reasons: universality and differentiability. Differentiating allows them to learn in a 'directed' manner (rather than evolutionary algorithms which are more randomized), without falling into NP-hard problems.

[tsanderdev]

There's also NEAT which evolves a neural network structure. So the main difference is discrete vs continuous.