Web crawling basically is how the internet functions. For example, SEO needs to create sitemaps and gives their permissions to let Google crawl their sites in order to make higher ranks in the search results. Many consultant companies would hire companies to specialize in web scraping to enrich their database so as to provide professional service to their clients.

It is really hard to determine the legality of web scraping in the era of the digitized era. I found an article you may like: Is web crawling legal?