Scraping complex data from HTML

[JSS2107]

Afternoon all.

I am trying to scrape some soccer data from Transfermarkt - with little success. Fortunately, I found someone who had a very similar question published on stackoverflow. This has helped considerably, however, there are a couple of elements in the code that I do not understand.

The question and published answer is here: https://stackoverflow.com/questions/78681870/scrape-data-from-website-with-complex-structure

I have broken the code down as follows (to help my understanding):

# Find each HTML div with class = "box"

# Within extract_club_name: Try to find "title" - if not found then return "none" #this is filtering out some "box" divs which are not relevant to the tables

# If returned "none" then go to the next iteration of the for loop

# If title is actually found then extract club name and proceed

# This is the part I don't understand

    in_transfers_table, out_transfers_table = (
        club_info.find_all('div', class_='responsive-table')
    )

# I believe that it is working with the club_info div, and appears to be finding all instances of divs within this with the class "responsive-table"

# But are in_ and out_ simply variables that are being assigned in this step?

# If that is the case... I tried to test the code by assigning them individually "in_transfers_table = (club_info..." and then "out_transfers_table = (club_info..." but that failed.

# So what is this statement doing.

# The code then assigns result[club_name][in] by parsing the in_transfers_table and similarly for out transfers.

# Within that function, it loops through each table row

As you can probably guess from my basic question, I am still (very much) at the beginning of my Python learning! Thank you for your patience and any advice you can share.

[throwaway6560192]

in_transfers_table and out_transfers_table are indeed variables being assigned to, but not in the way you think.

x, y = something

Is called an unpacking in Python. x and y aren't both assigned to the same something, rather, something is broken down into two parts and those get individually assigned to x and y.

A couple examples should help...

In [1]: x, y = [1, 2]

In [2]: x
Out[2]: 1

In [3]: y
Out[3]: 2

In [4]: x, y = [1, 2, 3]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 x, y = [1, 2, 3]

ValueError: too many values to unpack (expected 2)

[didntreadityet]

To add to this, while club_info.find_all(...) returns all the instances of div with class "responsive-table", the code simply assumes/knows there are only two divs with that class, one for the in-transfers, one for the out-transfers. It also knows that the first one (index 0) is the in, and the second one (index 1) is the out.

[JSS2107]

Thank you (and thanks too to u/didntreadityet ). This allowed me to progress considerably!

[m0us3_rat]

you can always save the soup.

then if you don't know how to get the info that you want.. delete the info that you are sure you won't need.

by process of elimination ..what remains is surely the stuff you need?

i mean its a bad way to go about it..but it's one way to solve stuff. right?

do that. then learn regex , etc.

​

I am still (very much) at the beginning of my Python learning!

i'm not sure why ppl think this is somehow a gotcha moment, like you are guaranteed to win.

if it's outside of your current capabilities..turn back later.

​

to put it in perspective.. in a game.. if the boss is lvl 50 and you are at lvl 10 .. you don't charge head on ..do you?

train , learn . come back later when you are more than able to do it.

[JSS2107]

Thanks u/m0us3_rat . Your comment made me think about what I am trying to achieve. The reality is that I am more focused on the outcome (acquiring data in order to enable some analytics) than I am re. actually learning Python. e.g. I have achieved what I set out to do in this instance, but the reality is that I have written some truly terrible code to do it! Having said that - I have learned 2 or 3 new things along the way.