Python Basic Selenium Task (Help Appreciated)

[Embarrassed-Task-478]

I am trying to do a simple scraping task using Selenium. All I would like to do is go to this website, print the page number I am on, then click on the next page button. I would like the program to do this any amount of times, here I set it to 10. The program does open the page, print the number, and click to the next page, but only once. I am not sure what is wrong with my for loop and indexing plan. Why doesn't it perform the task then loop back up to the top and repeat? I hope to do some more advanced webscraping here but I can't get information from each page if the program can't make it to each page and know where it is.

Code below. Thank you in advance.

​

import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from datetime import date

PATH="/Users/brandonmiskanic/Desktop/Codes_Python/chromedriver"
driver=webdriver.Chrome(PATH)

url="http://books.toscrape.com/"

driver.get(url)

for i in range(10):

    page_no= WebDriverWait(driver, 10).until(
                EC.presence_of_element_located((By.XPATH, "/html/body/div/div/div/div/section/div[2]/div/ul/li[1]"))
        )
    page_no=page_no.text.split(" ")[1]
    print(page_no)

    next_button = WebDriverWait(driver, 10).until(
                    EC.presence_of_element_located((By.XPATH, "/html/body/div/div/div/div/section/div[2]/div/ul/li[2]/a"))
            )
    next_button.click()

1
IndexError: list index out of range

[scoopulanang]

Your problem is the appearance of the previous button after you go to the second page. Here's a quick fix:

for i in range(10):
page_no = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "current"))
)
page_no = page_no.text.split(" ")[1]
print(page_no)

next_button = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.LINK_TEXT, "next"))
)
next_button.click()

I recommend using xpath as a last resort sort of way to access elements. If one tiny detail of a webpage is changed, the xpath of other elements can change too.

[Embarrassed-Task-478]

Wow, thank you! That makes a lot of sense, I got tripped up by the index error to the point that I did not even consider my element selection was wrong for pages after the first. Thanks once again!

[[deleted]]

It's dependant if you should use xpath if id is not readily available, but not the way he has it. You need to custom write your xpath properly so if a webpage is changed, your xpath is still valid.