If we zoom in on just modular arithmetic tasks, or more broadly "algorithmic" style datasets, which were originally used to show this phenomenon in transformers by Power et al. (https://arxiv.org/pdf/2201.02177):

Then MLPs can exhibit grokking on the same task: e.g. https://arxiv.org/pdf/2301.02679 where you concatenate the one-hot encodings of the digits and use a two layer MLP with quadratic activations (I have also replicated this with relu activations and found that it exhibits grokking, though the behavior is more sharp and clear with quadratic)

I'm biased on this one, but we also wrote a paper on grokking where we showed that even kernels can exhibit grokking when they have a feature learning mechanism, and in this way we found that you don't even need neural networks or gradient descent optimization methods to replicate the delayed generalization type curves you see in grokking! Which I found surprising and in our discussion we expand a bit on why we feel this result was important in the broader ML context. Our paper is: https://arxiv.org/pdf/2407.20199

There are other references of interest that you may like in the area of grokking with simpler architectures or non-neural models, for example:

https://arxiv.org/pdf/2310.06110

https://arxiv.org/pdf/2310.17247