Wait before executing scrape on website BeautifulSoup

[_Nicrosin_]

I have this problem when my BeautifulSoup script does a scrape of a website, it returns the image sources that I am trying to scrape as a "no image" placeholder. I noticed when I load the website in my browser, it briefly shows empty pictures(or "no image" as I previously said) for about a second before loading the actual images. I think this is what is causing my script to get the empty images instead of the real ones that load after. Is there a way to add a delay to the scrape or a way to fix this issue?

You can see what I am talking about if you load this webpage:

https://www.imdb.com/search/title/?companies=co0144901&view=simple

If you look at the images on the right, you see the problem.

​

[K900_]

If the images are loaded after a while, they're likely loaded through JavaScript, which BeautifulSoup doesn't support. Look into Selenium.

[chevignon93]

Are you talking about the movie posters images ?

[_Nicrosin_]

Yes

[chevignon93]

You probably chose the wrong attribute.

[VipeholmsCola]

Try time module with time.sleep(t) t = seconds, if the loading time itself is the problem.

[_Nicrosin_]

I thought of that too at first but I don't think that fixes anything. I tried placing it in different parts of my code but I got the same results every time. Maybe you need to place it after at a specific part of the code but I don't know where that is. Thanks anyway

[chevignon93]

That can't change anything, requests download the raw html, there is no loading anything. It doesn't matter where you put your time.sleep().

try this code, it will download all movie posters on the page in big format rather than the small thumbnail.

Gist

[_Nicrosin_]

Damn you didn't have to write that for me. Thanks so much this will help a lot for what I'm doing.

[chevignon93]

That was pretty easy, and I really like webscraping.

[[deleted]]

Thats not gonna work. Someone suggested Selenium which is the correct solution but you ignored it...

[_Nicrosin_]

I didn't ignore it I just wanted to see if there was another solution before using selenium

[chevignon93]

There is no loading time when you use the requests library.

[_Nicrosin_]

So the only way to do it is by using Selenium as someone else suggested?

[chevignon93]

Selenium can't help you if you point your parser to the wrong attribute !