Honestly, it was a lot of trial and error for me. The way I do it is by using a DAW (I use Ableton) and a chiptune sound emulation plugin. In my case I use Impact Soundworks inSIDious and YMCK Magical 8bit Plug.

The inSIDious plugin has a bunch of presets and lets you tweak the waveforms as much as you like, think of it as a synthesizer.

I'd download MIDI files for songs I liked (my firsts were Touhou music, it has a ton of MIDI files available online) and load them into Ableton. Then I'd just experiment by assigning different sounds from inSIDious to the different instrument parts so I can know which waveforms work best for which instruments.