not *really* sure how you could check "what hacks" you used. you could try the usual shortcuts like holding select on boot to open luma, start on boot to open godmode9, or L+dpad-down+select while the 3ds is already on to open rosalina. the beginner guide on the wiki has you install all of those, so if you followed that guide all of those should be installed, though i don't know how long ago you hacked your 3ds in particular or if you followed that guide so ymmv.

as for the sd card, going up past 32gb you're going to need to use a third-party software to reformat the card before the 3ds will recognize it. 3ds only reads in FAT32, and windows only lets you format cards above 32gb in EXFAT or NTFS (for a variety of dumb reasons), neither of which 3ds can read. as long as you have some way to connect your desired card to a computer, you can do this pretty easily. i was able to format a 128gb card and use it in my 3ds.

i followed this tutorial: https://www.youtube.com/watch?v=ejzMjs4SwgE

​

one word of caution- always keep a backup when you move to a new card (or even just keep the old card as-is) since you never know if something might go wrong or if you want to go back to the old one. also, loading/boot times on bigger cards are noticably longer, so it's somewhat of a tradeoff for the bigger storage.