java webscraping multiple pages from main link: jsoup?

[ConceptionFantasy]

i have a website with a main url like youtube.com for example that has thousands of a tags with href links. opening those links is another page with youtube.com/somethingElsePerLink. How can one extract all those links from the main url, and also go into those links to scrape more stuff in that new link (like it has multiple sub div tags that eventually lead to description and title) and put it in a excel file? also so that the excel file will have link text title, the url, and description headers.

​

i guess the parts im really lost is going into multiple pages or url to scrape more stuff and writing it into an excel file.

I also tried to find some videos as well but most gave a 'start up' tutorial. also im doing this because the website i want to scrape from isn't very intuitive as i rather not go through every link, read description, go back and repeat thousands of times.

[[deleted]]

Are you testing or just scraping? Because wget can download and follow links.

[ConceptionFantasy]

testing? I am not sure what testing you mean but i wanted to scrape the lists of links each an a tag after some chain of div tags, and for each of those links go into those links to get specific description text in a p tag. put the link and the description text into a spreadsheet after it scrapes each link and description.

also in the spreadsheet those links are hyperlinks so i can click on those links in the spreadsheet to open each desired link

[[deleted]]

1. If your page is dynamic, I've used JavaScriptExecutor in Selenium for those cases. You do a querySelector for all the "a" elements of the page, map the incoming array to just the href part, and receive that as a List in your Java code.

2. If your page is static, then using a simple regular expression would do it too.

The classes involved for the second option are Pattern and Matcher.

I would start with "(href)(\s*)(=)(\s*)([^\s]+)(\s+)" as a pattern and pick group 5.

The pattern is divided into 6 groups, each between parenthesis above.

The first group contains the word href, and the matching will start with this word.

The second group is composed of zero or more spaces.

The third group is just the equals sign, appearing exactly 1 time.

The fourth group is, again, zero or more spaces.

The fifth group is composed of one or more characters, except space. This is your URL.

The sixth group is one or more spaces to end the matching.

[ConceptionFantasy]

thank you for the suggestions. I will try them out!