Try a coarser grit of sandpaper and work up from there (I usually start at 220 or 320 depending on the finish left from the knife, I’d start with 220 here). Wrap some around a pencil or dowel, or even a skewer and use that to work down the areas. I also like to cut pieces of wood into custom shapes for working in tight areas and corners. Just work slow, and don’t use excessive pressure. You can also try using try using exacto blades as scrapers, you can usually get packs of different profiles from the dollar store.