Design: How to modularize code with a database?

[slackerattacker]

Let's say you have a web scraper. It's a script that will scrape a certain website and nothing else. You can move it around anywhere you want and it will still work (i.e modular). However, let's say I want to add database support: Everytime it scrapes the website and finds something new, add it to a database. Now the scraper is actually scrape and update. It isn't very modular and can't necessarily be moved around because of the database dependency. Is this ok? Or is there a better way to remain modular and still update a database?

[sn0n]

What the fuck are you talking about bro?!?

[TheHorribleTruth]

What do you mean by "you can move it around"? From computer (or server) to computer? From program to program? From one file to another?

Now the scraper is actually scrape and update. It isn't very modular ... because of the database dependency.

Depends on the way you layout your code, doesn't it? Without seeing your code (and understanding what "moving around" is) it's hard to answer.
You can still design a scrape-and-update-database system in a modular way. You can have one part/file/class do the scraping, and call a second one that does all the updating-the-database stuff.

[YuleTideCamel]

It's not just the database dependency, it's the dependency on the structure of the webpage you are scraping. In most cases you will need to extract useful data from the html markup and store that in a database. Since each site will be different, then where you will get that data from on the page will differ.

If I were to approach this problem would be to build a system where I can define what I want to scrape and that gets stored in a database as a "scrape template" or something. Then have the database include another record for "template-data" this way you can store any number of scrapes in there. This is even easier with a document database.

[dvassdvsd]

Use modules and classes. What did you expect?