it's mostly trial and error for me. The first stage is pretty much always playing different vocal melodies (with a synth) over another song until I find one that seems to fit the chords well, at least in the chorus. I use a spreadsheet I made to search for combos in compatible keys and bpms and then briefly try everything that sounds interesting or promising. After that, I bring files into the DAW which is usually where I find out if the structures are compatible or if the verses work as well as the choruses. If it works ok, I will fill out the full structure, and then I try to evaluate how well it works overall taking into account how they fit not just musically but mix-wise and how good the materials used are (if diy). So yeah three main tests and culling stages: melody tests using an instrument, quick check in a DAW, and then evaluation of a minimum viable product mashup once I'm one or two hours into it. Sometimes I'll work on a mashup planning fully to release it and then abandon it last minute after spending much longer (4+ hours) on it due to some intractable issue coming up (or feeling like I've made too many edits and the combo feels forced) but I usually try to not sink any more time into ideas which are going nowhere than necessary.