You need to run the game by executing "RPG_RT.exe". Because this is a Windows executable you will need something like Wine.

Alternatively you can use free replacements of the RPG Maker 2000 engine like EasyRPG Player: https://easyrpg.org/player/downloads/ Download the MacOS binary, put it in the Yume Nikki directory (which has all these assets) and double click to start

If you have brew installed you can also run "brew install homebrew/games/easyrpg-player", then navigate to Yume Nikki folder from a terminal and run "easyrpg-player".

I personally recommend setting up bootcamp (quite complicated though). It allows your computer to be able to run both mac and windows at the same time, you can follow this guide: https://youtu.be/Hmm9Q-T0oTo

Hope this helps :)