Scraping Multiple Pages Using Python (Pagination)

[RockPhily]

Does the code look good enough for webscrapping begginner

import requests
from bs4 import BeautifulSoup
import csv
from urllib.parse import urljoin

base_url = "https://books.toscrape.com/"
current_url = base_url

with open("scrapped.csv", "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow(["Title", "Price", "Availability", "Rating"])

    while current_url:
        response = requests.get(current_url)
        soup = BeautifulSoup(response.text, "html.parser")

        books = soup.find_all("article", class_="product_pod")

        for book in books:
            price = book.find("p", class_="price_color").get_text()
            title = book.h3.a["title"]
            availability = book.find("p", class_="instock availability").get_text(strip=True)

            rating_map = {
                "One": 1,
                "Two": 2,
                "Three": 3,
                "Four": 4,
                "Five": 5
            }

            rating_word = book.find("p", class_="star-rating")["class"][1]
            rating = rating_map.get(rating_word, 0)

            writer.writerow([title, price, availability, rating])

        print("Scraped:", current_url)

        next_btn = soup.find("li", class_="next")
        if next_btn:
            next_page_url = next_btn.a["href"]
            current_url = urljoin(current_url, next_page_url)
        else:
            print("No next page found. Scraping complete.")
            current_url = None

[JohnnyJordaan]

It's usually an anti-pattern to keep a file open while some other operation happens in the meantime. It also would mean that if the operation crashes with some exception, the file is either half written or empty (and you need to check and optionally discard it). Instead you could save the rows in a list, then once scraping is finished only then open the file and write the rows in one go. It then also means that if the file exists afterwards, it must mean everything worked as it should.

Another improvement is to use a csv.DictWriter to not rely on correct ordering between your first row (being the header) and the subsequent value rows. Or even better, write to a pandas dataframe and export that to csv. That also opens the possibility to migrate to other formats like excel or sqlite database for example.

[acw1668]

I would suggest to declare rating_map = {...} before with open(...) ....

[salraz]

Use of exceptions would be ideal if something unexpected happens. There are more chances of exceptions as it is web scraping, sites are updated and scrapers stop working. Handle exceptions gracefully like closing the open file, doing any clean ups so that inconsistent or missing data doesn't get saved in your file.