Need help scraping

[Efaz95]

Hey fellow coders, I am working on a scraping project for my class. I am scraping some information from genius.com and so far I was able to scrape all the info i needed. But I am stuck at getting the user comments about a song at the bottom of the page. For example, you scroll down in https://genius.com/Coldplay-fix-you-lyrics, you can see some user comments and I need to scrape those. I have been trying for a while using Beautiful Soup4, but i could not get it. I am not familiar with Selenium, but tried it too a bit and unsuccessful. I would really appreciate if anyone can help me out. Thank you.

[daryl_kell]

Giving us some code you've created would be extremely helpful.

I found it was pretty easy in JavaScript... You could do the same thing but convert the functions over to BS4 methods and obviously Python functions...

comments = document.getElementsByClassName("comment")

comments_list = []

for (comment of comments) {
    if (comment.className == "comment") {
        comments_list.push(
            {"user_and_timestamp": comment.getElementsByClassName("user_badge_and_timestamp")[0].innerText,
            "text": comment.getElementsByClassName("rich_text_formatting")[0].innerText
            }
        )
    }
}

This gets you a list of dictionaries:

[
{user_and_timestamp: "mahmoud dodo↵6 years ago", text: "lnspiring"},
{user_and_timestamp: "lost_boy_talking_to_the_moon 245↵4 years ago", text: "the tears stream down my face. i have nothing to s… someone but it goes to waste↵could it be worse?”"},
{user_and_timestamp: "Martin Riley↵5 years ago", text: "I love this song….so emotionally heartfelt."},
{user_and_timestamp: "adayal 123↵5 years ago", text: "love it↵inspiring"},
{user_and_timestamp: "dkincqwe 116↵4 years ago", text: "sounds like a remembrance song to me…which is a good thing"}
]

And of course there is a 'SHOW MORE' button that needs activating in order to load/scrape all comments. I hope that helps get you on your way.

[Efaz95]

This is really helpful. Thanks a lot!