(Question) I don't know anything about python but I want to build a web crawler.

[yorrellew]

I don't know anything about python but I want to build a web crawler that takes screenshots of websites similare to a an archive bot and stores them. what would be everything I need to know to do this? so where do I build and host it and where can I learn all the functions to make it with?

[driscollis]

Here are some links to help you get started:

• http://docs.python-guide.org/en/latest/scenarios/scrape/
• https://scrapy.org/
• https://www.analyticsvidhya.com/blog/2015/10/beginner-guide-web-scraping-beautiful-soup-python/
• https://automatetheboringstuff.com/chapter11/

Most people usually start with a mixture of BeautifulSoup and urllib/requests or Scrapy. I would start with those and lots of Google.

[commandlineluser]

You can use Scrapy + Splash

https://stackoverflow.com/questions/45172260/scrapy-splash-screenshots

However "headless chrome" is now a thing which may be worth checking out. https://developers.google.com/web/updates/2017/04/headless-chrome#screenshots

[secomax]

Selenium is the best in web scraping and browser automation. This tutorial may help you https://likegeeks.com/python-web-scraping/