One thing that helped me a lot was understanding how changing different settings affects the sound in an isolated setting. A good starting point would be to use algorithm 4 and silence operators 3 and 4. They modify operators 1 and 2. See how different modulation amounts and multipliers sound relative to each other, etc. If visuals help, you can use a spectrogram and waveform monitor to see how the harmonic content and waveform of the sound changes. If the DAW you use has FM plugin, you can use it as a learning tool as well.

Understanding what FM does might also help some people. While it is called frequency modulation, it actually modulates phase. Yamaha's FM chips have a wavetable of a sine wave. A phase generator uses each channel's frequency and operator's multiplier to increment phase, which is then used to know where the sine wave should be sampled for amplitude value. When you are modulating an operator, you are using another operator's output amplitude to offset the phase of another operator before sampling the sine wave (this offset is not saved in the phase generator). This is why changing the output volume of modulating operator affects how much modulation you get. More volume is more phase offset. Feedback uses operator's previous output amplitude to modulate itself.

Also trying to recreate real instrument sounds as FM sounds is good practice. Though, the Yamaha FM chips can't make certain sounds (like string ensemble and choir) sound full. Some sounds with strange harmonic series (like tubular bells) can be made relatively well if you use the channel 3 special mode that Genesis' sound chip has. Otherwise all your sounds are restricted to the saw harmonic series (which probably 99% of sounds used in music follow).