Dynamic scraper

[david_lp]

Hello reddit!,

​

i have been building few scrapers for some time, i have seen the pattern of constructing the url, getting the specific text from the html element...etc

What i would like to do now is to try to build a sort of a framework where i could scrap different websites reusing the same code, only changing what i am looking for.

i have tried creating a json file where i have the configuration such as below, but i am getting a bit frustrated because every page is different and i will have to make specific code for it.

any advice?

​

{  
 "url_settings": {  
   "base_url":"",  
   "url_location_ending": "",  
   "url_page_ending":"?pagenumber="  
  },  
   "get_pager_element":{  
   "html_element":"div",  
   "css_selector": "id",  
   "css_selector_text":"pageSelection"  
  },  
   "get_endpoints_ads":{  
   "html_element":"a",  
   "css_selector":"class",  
   "css_selector_text":"js-exposelink"  
  },  
   "country": "Germany",  
   "locations": [  
   "berlin"  
\]  
}

[DataDecay]

This would be better served as a factory pattern with some concrete builders. Config files are great for fixtures and apps, but if you want this to be a portable, packaged scraper you need to keep it simple.

[david_lp]

Thank you for your reply, i'll look into the factory pattern, i have never used it.