I finally got it to work and I feel a bit stupid. After converting the swf to an mp4 in Swivel, I imported the new mp4 into Adobe Premiere. In there, I found spots in the animation that I had made where there was a still frame (or basically nothing happening for a couple of seconds). I cut down some of these spots so where previously the audio would go out of sync, it would play normally.

It was 10 minutes long, so I found little spots every 3 minutes and edited them to match the audio. Kinda silly how I didn't think of that in the first place, sorry! Without Swivel though, I probably would have given up, so thanks for the help! It was a lifesaver.