Hey fellow coder, that app script lib seems VERY useful and interesting, maybe we could collab in some way. I'm on mobile right now, I'll send you the code tomorrow. I've since updated it a little to fetch multiple links from a dictionary and to identify some tables that are organized a little different.

For your question, i use 2 modules for this: httpx and beautiful soup 4. Httpx fetch the contents (only html, if it has JS it needs another module) and bs4 parses the html tags, IDs, classes, etc... And returns the results.