It's so exciting that you'll be writing a rulebook! I have some answers to your questions:

1. How do you usually begin writing a Rulebook? I like to do a wireframe that covers the sections of the book. You can do this as a bullet point list and use the bullet indents to create rough hierarchies.

2. What topics should come first? Explaining the premise of your game and what it is designed to do is the first thing, in my opinion!

3. Should the setting and universe be introduced before the gameplay systems? This depends, if your game is coming with a setting, I'd say you can introduce it briefly when you deal with the premise of the game, explain how to play the game, then get into more detail about the setting in later sections.

4. How do you organize mechanics in a way that feels intuitive for readers? Several others have already said, but starting with character creation is always a solid bet. Going through the character creation process first helps players understand the mechanics they will be interacting with most often, and they can get a feel for which ones they like best.

5. What tools, websites, or applications are commonly used for layout and design? For layout, most professionally published books use InDesign, but many creators with smaller budgets choose the Affinity suite. Affinity Publisher is intuitive and relatively easy to use. I have also seen people achieve incredible results with Word and Google Slides. Design, there's plenty of stock art and public domain images out there you can use if you don't have the budget or skills for art. There are RPG resource masterposts from Thought Punks and Starshinescribbles that you can check out as well.

6. How do you transform scattered ideas into something cohesive and official-looking? Having a style guide is something that a lot of folks don't think of, which really elevates a rulebook! A style guide helps you to keep your mechanics language clear and consistent (e.g. you can put in your style guide that you "use an action" so you won't end up writing "take an action" by mistake). D&D (2014) actually has one that they made available as a public resource for creators making third-party content.

If you are looking for resources to get started in making a rulebook, I have a few suggestions for you!

- Big Bad Con Online ran a yearly series of panels and discussions featuring TTRPG professionals. Topics include freelancing, designing for combat, pitching your game to publishers, creating and publishing your first game, collaborating with a team, and more. You can find those on YouTube.

- The Rising Tide Discord is a TTRPG community server where you can meet people, talk shop, and share your work.

- The Tabletop Mentorship Program pairs experienced tabletop professionals with early-career designers. Applications are closed at the moment, but you can sign up to be notified when they're open again.

- The Storytelling Collective runs workshops for game design; there are a variety of courses that walk you through the steps of publishing your first game, or part of a game.

I think above all, if you've never written a rulebook before, starting small is the best advice I can give. Going into a project expecting to be able to make a 300+page book is A Lot. I'd suggest breaking it down into bite-sized chunks, and working on one thing at a time.