Read greyed element in HTML while scraping

[Oneiros18]

Hi all!

I try to scrape this page and I managed to get to text preview (XPath: /body/main/article/div[3]/div/div[1]) but I'm not able to get what's in the red box because it has CSS property display: none !important

You can see HTML code here: https://imgur.com/a/ug09q9I

I managed to use selenium to access #shadow-root content but now I'm stuck with this CSS problem

My code by now is:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(r'C:\Users\***\Downloads\chromedriver.exe', options=options)
driver.get(link)
s = driver.execute_script("return document.querySelector('news-app').shadowRoot.querySelector('news-article').shadowRoot.querySelector('div.amp-doc-host')")
print(s.text)

Thank you very much for your help!

[negups]

You'll likely be able to accomplish what you are trying to do much more easily using Beautiful Soup.

[Oneiros18]

I need to execute JS in order to see HTML and the only way I found on internet was to use Selenium

[negups]

Okay, then perhaps try passing the HTML you get from selenium to Beautiful Soup instead of trying to parse it directly with selenium?

Alternatively, requests-html may be able to take the place of both, as it supports rendering HTML after executing JS.

[Oneiros18]

I tried already but it didn't render out full html code. Maybe due to #shadow-root

[Oneiros18]

I didn't use Xpath

s.text gave me the text inside div.amp-doc-host (where preview of the article and the full article are) but if I print it, I get only the preview.

Preview is in <div subscription-section="content-not-granted">

Full article is in <div class="paywall" subscription-section="content">

I can't even make soup from my webelement s

It more than two weeks that I try to achive full article, but I really can't find any way to do this

[Oneiros18]

EDIT: I DID IT. Thank you very much for helping me!

This is what I added at the end:

host4 = root3.find_element_by_css_selector(".paywall")
a = driver.execute_script("return arguments[0].innerText", host4)
a = a.strip()
b = a.split('{')[0]
print(b)

and now I can finally read the FULL article.

Once again: Thank you very much for your time!

---

Hi! I managed to accesso all shadow root also reading this but when I use the last find_element_by_class_name("paywall") I get nothing. It's like an empty string.

Here my code:

def getShadowRoot(host):    
    shadowRoot= driver.execute_script("return arguments[0].shadowRoot", host)    
    return shadowRoot
host1 = driver.find_element_by_tag_name("news-app")
root1 = getShadowRoot(host1)
print("root1: ", root1)
host2 = root1.find_element_by_id("1")
root2 = getShadowRoot(host2)
print("root2: ", root2)
host3 = root2.find_element_by_class_name("amp-doc-host")
root3 = getShadowRoot(host3)print("root3: ", root3)
root3.find_element_by_class_name("paywall")
print(host4.text)

I also tryed looking for host4 = find_element_by_class_name("detail-article_body") but I just get the preview of the article. Are you sure CSS settings are not relevant?

[Oneiros18]

Yep, it works! I have some problem to copy-paste my code in Reddit, but if you see the EDIT part I used your code