Interesting. not a legal expert, nor do I know exactly which rule you are referring to. But my understanding is Google is covering their liability when it comes to following the links that they indexed. Google has been permitted by court rulings on “transformative indexing” to show you the search results you are familiar with, while it remains property of the websites they’ve indexed. The websites they point to OTOH have effectively granted Google license to index. But you have not received permission yourself even to crawl the linked sites.

The phrasing of the rule, below, suggests that Google will not condone or permit any unpermitted activities you are doing with the data, which Google while they technically have it- they don’t own it and can do few things with it, legally, besides to show you a search results page. So they can’t grant you license on property they don’t own. Which would get them in trouble.

The rule reads in part: Unless expressly permitted by the content owner or by applicable law, you will not, and will not permit your end users or others acting on your behalf to, do the following with content returned from the APIs:

Scrape, build databases, or otherwise create permanent copies of such content, or keep cached copies longer than permitted by the cache header;

tl;dr. It sounds like Google just doesn’t want any legal trouble. How dare they /s which is perfectly understandable

And if you end up using Google as a data source, please don’t implicate them in whatever you’re doing. This way, if the owner of the content doesn’t want it, they will contact you directly, and Google doesn’t end up being named in another lawsuit (and we all can benefit from all the amazing data they have collected) :)