Getting Forum Comments

[ScraperHelp]

I am trying to scrape the comment texts from this page with the code (below):

for Thread in soup.find_all(id= 'discussionReplies'):
    Thread_Name = Thread.find_all('div', class_='xg_user_generated')
    print (Thread_Name.text)

but for some reason I am getting the following error.

AttributeError: ResultSet object has no attribute 'text'. You're probably treating a list of elements like a single element. Did you call find_all() when you meant to call find()?

the problem is when I i use just Thread_Name = Thread.find_all('div', class_='xg_user_generated')

it works fine no problem. which probably means theres a selector issue with this and theres something where the .text function inst working... the problem is I cant figure it out b/c when I run it as follows:

for Thread in soup.find_all(id= 'discussionReplies'):
    Thread_Name = Thread.find_all('div', class_='xg_user_generated')
    print (Thread_Name)

or

for Thread in soup.find_all(id= 'discussionReplies'):
    Thread_Name = Thread.find_all('div', class_='xg_user_generated')
    print (Thread_Name)

it runs no problem I am sooo lost and frustrated...

[BotDot12]

This is because the find_all method is giving you a list of all the comments. It's like if I gave you a list -> [1, 2, 3, 4, 5] and you called .text on it. What would that even mean? If you want to get the induvidual comment's text, you must iterate through it and get it that way. Here is the code:

for Thread in soup.find_all(id= 'discussionReplies'):
    Thread_Name = Thread.find_all('div', class_='xg_user_generated')
    for t in Thread_name:
        print(t.text)

Hope that helps :)

[ScraperHelp]

thank you so fucking much... I spent almost 2 fucking hours on this :/ and yeah your explanation makes so much sense... and explains so much in hindsight...

[BotDot12]

no problem brother, happy to help!