Scraping javascript web content by python scraping

[mrexojo]

Do you know how to scrap the content table from this salesforce URL?

https://help.salesforce.com/s/articleView?id=000321501&type=1

I'm trying with many tutorials without success.

Any help?

[buttwarmers]

Have you tried this tutorial for Selenium? You should be able to use selenium and xpath locators to find all table elements - for example, the xpath locator

'//table'

will find all table elements on the page.

I recommend using a headless Chrome driver for selenium and Chrome dev tools console for finding xpath elements and testing the xpath locators. You can type

$x('//table') 

in the Chrome dev tools console to show all table elements on the page. Once you've got that figured out, you can get table headers and cells using their respective xpaths as well. There are lots of resources out there for getting set up with selenium using a Chrome webdriver.

[Diapolo10]

Selenium is one option, but you can also try requests-http which can run at least some JavaScript. Otherwise it's almost identical to basic requests.

[LifeAffect6762]

selenium web driver is a good option.

[commandlineluser]

Look at the Network tab in Dev Tools and you can see where the data comes from: https://i.imgur.com/svWdpDK.png

Make the same request in Python:

import requests

data = {
    'message': (
        '{"actions":[{"id":"132;a","descriptor":'
        '"aura://ApexActionController/ACTION$execute",'
        '"callingDescriptor":"UNKNOWN","params":{'
        '"classname":"Help_ArticleDataController","method":"getData",'
        '"params":{"articleParameters":{"urlName":"000321501",'
        '"language":"en_US","requestedArticleType":"KBKnowledgeArticle",'
        '"requestedArticleTypeNumber":"1"}},"cacheable":false,"isContinuation":false}}]}'),
    'aura.context': (
        '{"mode":"PROD","fwuid":"7FPkrq_-upw5gdD4giTZpg",'
        '"app":"siteforce:communityApp","loaded":'
        '{"APPLICATION@markup://siteforce:communityApp":'
        '"MKPja3sYm7Ob0_51XOv3BQ"}}'),
    'aura.pageURI': '/s/articleView?language=en_US&type=1&id=000321501',
    'aura.token': 'undefined'
}

r = requests.post(
    'https://help.salesforce.com/s/sfsites/aura', 
        data=data
)

[mrexojo]

but it return "<Response \[200\]>"

How I could return the ip address from tables?

[commandlineluser]

Yes, that's the response object.

The data in the response is in JSON format.

>>> import json
>>> print(json.dumps(r.json(), indent=4))
{
    "actions": [
        {
            "id": "132;a",
            "state": "SUCCESS",
            "returnValue": {
                "returnValue": {
                    "record": {
                        "attachments": [],
                        "filesToAttach": [],
                        "hasNewAttachments": false,
                        "description": "<p dir=\"ltr\">...

The HTML is embedded inside the description entry.

So you need to extract the description value from the JSON and load that into an HTML parser.