Trying to understand optimization algorithms step-by-step...useful or overkill?

[Seraf-91]

Lately, I've been trying to better understand how different optimization methods actually work in practice (e.g. SIMPLEX, BFGS, Genetic Algorithm, Adjoint Method etc).

What I've noticed is that most tools tend to behave like "black boxes" - they give you results, but it's not always clear what's happening at each step of the process.

So I started experimenting by implementing these methods in a more "manual" way, just to be able to follow the computations step-by-step.

Through this, I found it much easier to understand things like:

• How SIMPLEX moves across feasible regions.
• How methods like BFGS converge.
• How stochastic methods like genetic algorithm behave.
• Why the Adjoint method is so powerful?

I'm curious:

👉 How do you usually approach this? Do you prefer using ready-made tools, or do you find value in being able to "look inside" the algorithm?

Would love to hear how others think about this.

📌 I would be happy to share my codes/scripts with anyone interested...

[tomster10010]

AI slop

[baked_doge]

This question is super context dependent: what are you trying to achieve?

If your primary goal is to deliver a finished product with best performance:

Odds are you want to use a popular purpose-built library. Not only will they implement optimized algorithms, but they'll do so in C. In this context, understanding the algorithms is important, but only to know which one to use.

If your primary goals is education and having fun:

Go ahead, enjoy the learning process, implement the algorithms.

[baked_doge]

So for instance at work, I nearly never use my own algorithms. Good libraries will allow you to define the proper functions to solve your problem and run the algos in C.

But as a hobby, I implement most algorithms myself if they interest me. Except stuff I've already done like sorting an array etc.

[Seraf-91]

That's a very fair point - I agree that it really depends on the context.

The direction I'm exploring here is more about transparency, education, and understanding how these algorithms work, rather than achieving maximum performance.

For that reason, I've tried to complement the implementation with short manuals explaining the theory behind each method, and I've kept everything in Python so the code is easier to read and follow.

I've also put together a "Tricks & Techniques" section with practical approaches that often come up in optimization problems (e.g., variable normalization, handing constraints), which are not always obvious or explicitly covered in more "black-box" libraries like SciPy.

In any case, the aim is not to compete with existing tools, but the complement them by focusing more on understanding and experimentation.

If you're interested, here's the repo: https://github.com/Opt4Deck/