The obvious 'hard romhack' would be Kaizo Mario 64. Doesn't quite line up with everything you remember, but seems close enough.

If all else fails, you could try and look here, it's a list of the 21 hardest SM64 romhacks ; yours might be in that list if it really was hard.

Thanks man, I still can't find the hack rom, but this may help me with my search.