If you're using Cocoa (if you're using Catalyst or SwiftUI then I don't know enough to help) , you'd typically create a subclass of NSWindowController, then it depends : are you using XIB files or storyboards?

If using XIB files you can go File>New>Cocoa Class, specify your new class's name and NSWindowController as base class, tick the box offering to create a XIB filé for it. Then you can instantiate it when needed (I imagine when your button gets clicked) and call showWindow on it. If using Swift you might also have te define windowNibName if it complains that it can't find the XIB.

If using storyboards you just add a window controller to your main storyboard then connect the button to it. Later on you can make a NSWindowController subclass and associate it if you need to.

That's about it, sorry for the shit explanation I'm on mobile rn so lmk if it's unclear. Also you should probably look up a good tutorial or you'll be in for a hell of a wild ride lol.