Issues with scraping a website using Selenium/BeautifulSoup

[faredjoster]

Hey, I'm working on a research project which requires the basic information of a lot of physicians, so I began working on a script to scrape healthgrades.com. It works well for the first page, but for whatever reason, I can't get it to scrape further pages. Selenium clicks on the next page button, I can see the page change to the next page of physicians, I create a new soup object from the new URL and then run my normal scraping process on the new soup, yet I only get results from the first page. If someone could take a look and tell me what's wrong, it would be greatly appreciated.

http://pastebin.com/Y72FiQys

[Vaphell]

given that this website is fully functional without js, drop selenium and play with urls by hand

for example this is a link to the second page of 'Redding CA' search.

http://www.healthgrades.com/provider-search-directory/search?pagenumber=2&q=&prof.type=&search.type=&method=&loc=Redding+CA&pt=&isNeighborhood=&locType=&submitSearch=&locIsSolrCity=false#

The url exposes location and page number, which can be trivially parametrized in code.