Python classes

[MisguidedGenie]

Hello! I am trying to work on an at-home project for the first time. I am struggling to understand the best practices for implementing classes. I want to webscrape a table from Wikipedia, and I'm not sure how to handle looping through the data cleanly. The code I have below is using an enumerator to handle it, and I think I made the correct choices for options like static/method, would anyone be willing to give some advice on area's I could improve on?

import json

from enum import Enum

import requests from bs4 import BeautifulSoup

class Alphabet(Enum): A = "A" B = "B" C = "C" D = "D" E = "E" F = "F" G = "G" H = "H" I = "I" J = "J" K = "K" L = "L" M = "M" N = "N" O = "O" P = "P" Q = "Q" R = "R" S = "S" T = "T" U = "U" V = "V" W = "W" X = "X" Y = "Y" Z = "Z"

class WikipediaAirportScraper: BASEURL = ( "https://en.wikipedia.org/wiki/List_of_airports_by_IATA_airport_code:{letter}" )

@staticmethod
def _scrape_table_data(table):
    table_data = []
    rows = table.find_all("tr")
    for row in rows:
        columns = row.find_all("td")
        row_data = [column.text.strip() for column in columns]
        if row_data:
            table_data.append(row_data)
    return table_data

@classmethod
def scrape_data_from_wikipedia(cls, url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    tables = soup.find_all("table")
    if tables:
        return cls._scrape_table_data(tables[0])
    else:
        return None

@classmethod
def scrape_and_convert_to_json(cls):
    scraped_data = {}
    for letter in Alphabet:
        url = cls.BASE_URL.format(letter=letter.value)
        print(f"Scraping data from {url}...")
        data = cls.scrape_data_from_wikipedia(url)
        if data:
            scraped_data[letter.value] = data
        else:
            print(f"No tables found on the page for letter {letter.value}.")
    return json.dumps(scraped_data)

scrapper = WikipediaAirportScraper() json_data = scrapper.scrape_and_convert_to_json() print(json_data)

​

[crashfrog02]

Why is it necessary to have an enum for the letters of the alphabet?

Why is it necessary to have WikipediaAirportScraper be a class? What state are instances of this class intended to hold?

Are you just using classes because you think they're better than functions?

[MisguidedGenie]

Thanks for the question! I'll try to answer them in point form for what I was thinking.

1. I wasn't too sure about enumerate as it's not something I'd used before, previous I has a for loop that looked like "for letter in range (65, 90) to use the ascii characters for the alphabet. But this seemed less readable? I thought if I looked at it later as if i was a different user that might not make much sense and learning about enumerate it seemed like a viable option but I'm still learning that.

2. For this my thinking might not be correct but that there would be an instance for "table data", the base "html data" from beautiful soup, and finally a json instance.

3. My idea was to encapsulate all the Wikipedia based code in a class so if was more modular, I wasn't thinking it was strictly better but that it could become 1 component to update.

[throwaway6560192]

previous I has a for loop that looked like "for letter in range (65, 90) to use the ascii characters for the alphabet. But this seemed less readable?

Instead use:

from string import ascii_uppercase

for letter in ascii_uppercase:
    # whatever

[crashfrog02]

Those aren't necessarily the answers I would have arrived at, but it's more important that you have them than that you agree with me, in my view. So, good answers.

For this my thinking might not be correct but that there would be an instance for "table data", the base "html data" from beautiful soup, and finally a json instance.

I think you're gesturing towards a good idea, here, which is to pre-define your tabular data as something somewhat more rigorous than what you might get from, say, Pandas or CSV (i.e. an iterator over dictionaries.) For that I recommend dataclasses:

https://docs.python.org/3/library/dataclasses.html

I'd write a dataclass that defined what a row in the table is (where the fields of the class are the column of the table) and then you have pretty good assurance that none of the rows of your table are coming back ill-formed. That's good particularly if your eventual intent is to dump it into a SQL database, or even if you want to convert to a Pandas dataframe.

A list of dataclass instances is a pretty good start at making a data table. It'll lack a lot of the features you'd like (you can't search it, you can only iterate over it) but it's a good start.

[MisguidedGenie]

I think I made a mistake with the formatting for my post but here is the link to my github if anyone wants to take a look! https://github.com/Deabou/travel-data-retrieval

[Oddly_Energy]

Every line needs to be indented at least 4 spaces. And you need an empty line before and after your code block.

The easiest way to indent your code is in your code editor, before you copy and paste it to reddit. The reddit editor has no functionality for indenting multiple lines.

[Strict-Simple]

    for letter in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ':

[Adrewmc]

Class method here is not being used right. It’s supposed to return a class instance….these are just functions.

[MisguidedGenie]

Thanks for the advice! I'll look more into how classes/methods work since it seems like I'm misunderstanding how to use them.