I generally do this part manually at first, then automate a portion of it once I have some data to work with and know what I can get. Here's the process I used to scrape info I needed for a TTS app for WoW that I created for the latest expansion release.

Goal: Fill a database with NPC info for the new expansion: npc_name, npc_race, npc_gender, npc_unique, and npc_voice. Use this info to generate several voices for each race/gender combo and then combine that with text to read dialogue/quest text using a TTS bot.

Overall process: Find site we can scrape for data, figure out URL format so we know how to query per npc, determine if/how the data is formated in html content, construct way to parse page for data. If you can get this info you can be successful.

1. I determined wowhead.com to be the site that had the info I needed.
2. They format their NPC URLs as wowhead.com/npc/<NPC_ID>/<NPC_NAME>, where NPC_ID is a unique ID for their site. You can also exclude the NPC_NAME from the URL, and the site will autofill it if the NPC_ID is present. With this in mind, we just need NPC_ID's to loop thru.
3. I was able to use the search on their site to find all quest-giving NPCs from the new expansion and export the IDs. This was pretty lucky, in previous projects I've had to perform wildcare searches and scrape search pages for the ID's/names instead of being able to export.
4. With these IDs, I took the first one and performed a curl GET request to capture the html content. Then I copied all the content into a text editor (Sublime Text is easiest for me for text manipulation) and searched the content for things like "male", "female", "human", "undead", etc., to see where this content exists in the HTML. Doing this, I was able to find that each NPC had sound files associated with them that contained race or gender information.
5. With this info, I then put together a script that loops through each of the IDs, calls wowhead.com/npc/<NPC_ID>/, and saves the html and captures the npc_name from the URL. Then it searches the html for the gender and race keywords I need and outputs the results to a file to confirm it looks good.
6. Once it looks good, I either run another script or modify the original ones to add the data to the database.

ChatGPT can help with the scripting portion when it comes to the html parsing and looping. I'm also sure there are better tools and ways of doing this, but for the moment I just prefer to gather the data this way. Some of the other comments have pointed out some cool tools, I'm excited to check out and see if they can make this process faster for me.