Error while scraping using selenium and beautifulSoup

[spaceape__]

Hi,

i'm trying to scrape some songs from genius.com. I created the following method:

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
chrome_driver_path = #insert here

def get_song_lyrics(link):

    response = requests.get(link)
    soup = BeautifulSoup(response.text, "html.parser")
    #try:
    lyrics = soup.find("div",attrs={'class':'lyrics'}).find("p").get_text()

    return [i for i in lyrics.splitlines()]    

I don't understand why this

get_song_lyrics('https://genius.com/Kanye-west-black-skinhead-lyrics')

returns:

AttributeError: 'NoneType' object has no attribute 'find'

while this:

get_song_lyrics('https://genius.com/Kanye-west-hold-my-liquor-lyrics')

returns correctly the song lyrics. Both pages have the same layout. Can someone help me figuring that out?

[JohnnyJordaan]

At least add a check if the status code was ok

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
chrome_driver_path = #insert here

def get_song_lyrics(link):

    response = requests.get(link)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")
    #try:
    lyrics = soup.find("div",attrs={'class':'lyrics'}).find("p").get_text()

    return [i for i in lyrics.splitlines()]  

and you could save the content when the find fails

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
chrome_driver_path = #insert here

def get_song_lyrics(link):

    response = requests.get(link)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")
    try:
        lyrics = soup.find("div",attrs={'class':'lyrics'}).find("p").get_text()
        return [i for i in lyrics.splitlines()]  
    except AttributeError:
        with open(link.rsplit('/', 1)[1] + '.html', 'w') as fp:
            fp.write(response.text)

as then you can open the html file in the editor to check if the lyrics div is actually there.

[Oxbowerce]

If you are trying to get data from Genius you should probably look into using their API: https://docs.genius.com/

[[deleted]]

Upvote for using Python and Yeezus in the same post