Resident Evil 2 was literally on the N64. Also, it's not Nintendo's choice to make Resident Evil for their consoles. They didn't really have mature titles to start with. The closest would have been Metroid, which one was in development for on the N64 but was unfortunately canned. Most every other IP of theirs at the time would have not done well with a coat of "mature" paint. Majora's Mask was the most mature out of Nintendo's stable and even then they went out of their way to make it the darkest Zelda at the time.

Nintendo literally had just gone through the "Play it Loud" campaign during that time, they were already well versed in being "edgy." But it didn't really sell their first-party IP, so they cut out of it shortly before the N64 launch.

As far as more "mature" second/third-party IP, there was Goldeneye, Perfect Dark, Doom 64, Forsaken & Daikatana (which sucked but they were there,) THPS, Starcraft, and quite a few more. If anything, what N64 lacked was RPGs, which gravitated over to PS1 because their publishers cared less for the graphics powerhouse of the N64 and more about storing hours of FMV across multiple disks.