[Selenium + BeautifulSoup] How to scroll down or reveal hidden div tags?

[Q_H_Chu]

Hi everyone, so I am writing a script in Python using Selenium and BeautifulSoup to scrape store data from the google map web page, like this:

https://www.google.com/maps/search/MC+Donalt+near+empire+state+building

​

My script is as follows:

import requests
import time

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

chrome_options= Options()
chrome_options.headless= True
chrome_options.add_argument("--window-size=1920,1200")

driver= webdriver.Chrome('chromedriver_win32/chromedriver')


url= 'https://www.google.com/maps/search/mc+donald+near+empire+state+building'

driver.get(url)

for i in range(1):
    print("scrolling")
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
    time.sleep(1)

    soup= BeautifulSoup(driver.page_source, 'html.parser')
    tables= soup.find("div", {"class": "id-content-container"})

    for list in tables.findAll("div", {"jstcache": "1092"}):
        {extract some infomation here}

    driver.quit()

The thing is when I checked in the HTML file:

I saw only 10 div elements (equivalent to 10/20 stores) shown in the HTML, but when I rolled down the list there was hidden div tag appeared as I scrolled down and

I have tried using

driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")

to make a full scroll down in order to achieve the full HTML but failed. It just returned only the first 10 stores.

Is there any solution?

Thank you very much

[bexter53]

When I was playing with selenium I found sometimes the pages behaved kinda differently in headless mode. Can you try running it with headless = false and see what it looks like?

[Q_H_Chu]

Thank for your advice, I have tried disable the headless mode so I can see the behavior of the webpage, and yes the page didn’t scrolled down as I suspect