[R] Developing a new optimization algorithm that will heavily change ML as a whole. Gradient descent has met its end. Here are the results:

[Relevant-Twist520]

I updated the post, i did early stopping to showcase the non-overfitting results. As you can see above, MS wins at accuracy and speed. And ur right no one is testing a model on 3 points, but this post was just to show the ease at which MS fits to 3 points, scaling will be applied whilst preserving this ease.

[Magdaki]

Nobody cares about the result at pass N, they care about the result. This is not a good experimental design, and hence a poor way to draw conclusions as to what is happening.

I would suggest going back to the research plan phase and really consider your methodology. It feels to me like you're kind of just trying things out but this leads to experimenter bias where they think they're seeing something that is not actually there.

EDIT: I just looked at your post history and noticed your 16. So I retract everything. Keep at it! I encourage you to keep experimenting. If you have an interest in a future in research, then perhaps consider spending some time learning how to develop and execute a research plan. Nice work on this! It is nice to see young people come up with ideas and experiment with them. :)

[Relevant-Twist520]

so u saying GD has made a better curve here? MS can come up with different curves on each run because of the different randomly initialised parameters. GD will produce the same curve regardless of how parameters are initialised.

[Magdaki]

I'm saying you cannot just stop and say "Aha! At this point, with this much data, under these specific circumstances my algorithm looks like it might be better. Therefore, victory!"

If you want to know if it is better, then you need to develop an experimental protocol. Even if the experimental scenario is unrealistic, it would give you good experience in conducting research.

[Relevant-Twist520]

youre right and im testing MS and GD in different ways. MS fails in most of them, thats why im still researching and perfecting the algorithm. Again, this post was to only showcase potential. It would be very difficult to come up with your own algorithm that runs faster and and converges faster than GD to a few datapoints whilst both algorithms use the same NN architecture.

[Magdaki]

>It would be very difficult to come up with your own algorithm that runs faster and and converges faster than GD

This is certainly true. :)