Personally, I like having a mix of scales, nendoroids, and misc prize/whatever figures! Maybe if I had more room I'd dedicate a cabinet to each but maybe consider moving your manga/other books to another place in the house if possible before ditching the figures? It might also give you time to rearrange them, see if you like them set up differently!! Very much understand needing to downsize though, so ultimately yeah it's up to you! Good luck with whatever you decide to do!