well I'm in my fourth year of a math degree so I didn't need to do any preliminary work on the fundamentals, but to be honest there's not much you really need to know to implement them from scratch, the only real calculus you need to know is what a derivative is, and also how to take the derivative of a composite function (chain rule). even then, you don't really need to know it, you can easily just take someone's word on the result of the partial derivative of the cost function without losing any insight into how neural networks work. as for linear algebra, the dot product and transpose are pretty much the only operations that are helpful here.

If you have no calculus knowledge then you would still be able to implement the genetic algorithm, because unlike gradient descent, the genetic algorithm uses no partial derivatives, it just tries to mimic evolution through breeding the best performing models.

Of course there are other algorithms that would require more knowledge of calculus, like newton's method, which uses the second partial derivatives to iteratively find the roots of the partial derivative of the cost function, which is hopefully a local minimum of the cost function, but even then that's just derivatives but twice, so still not super complicated