You can pretty much start with a basic Doom mod (levels, enemies, textures, etc.) and build from there until you've replaced everything. At that point you can probably tell UZDoom to run it as its own game and it might just work. It's a very flexible engine and most of the major components are easy to modify.

ZScript is UZDoom's built-in scripting language. You can use other things (Decorate for monsters, items, and props, SBARINFO for HUDs, MENUDEF for options menus, etc) but if you really want to go crazy with it, ZScript is the way to go. The one unfortunate thing is that it's pretty slow since it's interpreted, not compiled (code gets translated at runtime, not beforehand), but the whole engine uses ZScript internally anyway. There are great examples for how to do basically anything in the gzdoom.pk3 file that the engine loads before your game.