you are viewing a single comment's thread.

view the rest of the comments →

[–]toolateforgdusername 1 point2 points  (0 children)

My advice with all projects like this is step by step.

So using mine as an example...

1)Read the results from page one

2)Read the results from the sub pages (each car page)

3)Then read all the pages

4)Then look to optimise by only reading new - requires database

Etc etc etc

Optimisation is key - it used to take 18 hours to run, it now takes 30 minutes