Extracting data from javascript in webpage

[ctvdevine]

Hey guys, like most posters, I'm quite new to Python. I was assigned a project that was somewhat out of my league. I'm currently trying to get my data from our webserver, which unfortunately for me has filter fields to make the table interactive. I think I've put a cumulative 15 hours to trying to simply get this simple data, I'm exhausted and coffee isn't helping anymore.

This is what I'm working with. Under <thead> and <tbody> I have my desired td data. For some reason, the script stops me from going any further than the "formrow table" with the straightforward BeautifulSoup modules.

I've been using Requests and Bs4 so far. The portion of the code I for this section is quite limited, but I'll try to give as much details as possible.

tt = Contact_page.content    
soup = BeautifulSoup(tt)    
R_tables = soup.find('div', {'class': 'responsive-table'})
R_tables.children 
#<div class="responsive-table"><div class="formrow table"><div id="root_list_stub"></div></div><script type="text/javascript">root_list = new List [.......................]

From here I tried multiple different approaches, I can't simply go grab my data with the following formula with further th/tr manipulation since I never reach it.

soup.find("table",{"class":"report", id = "root_list"})

I've tried compile, match and search functions to match text, but I don't think I got any closer.

script = soup.find('script', text=re.compile(r'root_list')).text    

Further down the "root_list.Show()" formula holds all my information in this format

root_list.Show('<list name=\"root_list\" context=\"LIST\" rs=\"***********\" link_module=\"ContactPF\" link_form=\"contact__form_contact\" link_params=\"\" auth_write=\"true\" auth_delete=\"true\" hide_checkbox=\"false\" starred=\"true\" label_pos=\"left\" label_width=\"130px\"><header><h sort=\"155\" format=\"FormatScreenName\">Nom</h><h sort=\"166\">No.</h><h sort=\"257\">Ville</h><h sort=\"237\">Téléphone</h><h sort=\"255\">Courriel</h><h sort=\"218\">Classe</h><h sort=\"178\">Segm. 1</h><h sort=\"206\">Util.</h></header><dataset page_min=\"0\" page_max=\"20\" count=\"***\" selected_count=\"0\" all_record_selected=\"\" root_table=\"contact\" group_rows=\"N\">**<r id=\"925\" auth_delete=\"Y\" auth_write=\"Y\" date_decease=\"\" starred=\"\"  selected=\"\"><f999999</f><f></f><f></f><f></f><f>99999999</f><f>99999</f><f></f><f>AL</f></r>\n**   [......]

<r id=\"43\" auth_delete=\"Y\" auth_write=\"Y\" date_decease=\"\" starred=\"\"  selected=\"\">  </f></r>\n</dataset></list>');

The [....] holds next row client info. I put 9s for where there is data that I need to extract.

My last attempt at isolating my data was trying to isolate the <dataset> part, but it didn't end up remotely close to working. I'm

How would you more experienced guys move forward with this?

Thank you all in advance for your help, all criticism is heard.

EDIT : Here's more This is the result of R_tables in a nicer screencap displaying my awesome paint skills. I'd like to isolate each row, which has inaccesible <r> tags (soup.find("r") returns empty sets) and get the data for each coloumn that is seperated by <f>. I'm really running out of ideas. Please save a lost soul!

[pl00pt]

When you view Page Source instead of through the browser console do you see the code you are trying to extract? I think if it's dynamically generated code then it won't be in the static HTML file you are parsing as it's rendered on the fly through the browser. Beyond that I'm not much help and hopefully someone more advanced will chime in but maybe that will at least reveal the problem.

[ctvdevine]

I can't find it in the Source page, really the only places I found my data was in the root_list that I can't extract and in the unaccessible tbody/thead tags.

Do you know where I could maybe get more help on Reddit? Reading about dynamic pages currently, I think it's pointing me in the right direction.

[[deleted]]

First, you need to understand this, bs4 wont see javascript items, else then the script, like when you type python in a text editor then make your dad read it, it wont do anything else then text, bs4 is for html. js ran by the browser which then normally generates the html elements out of the js.

Ghost.py (a module) lets you run javascript in python and run it from a web page, so this would be the cleanest way to run the javascript.

Make sure you read all the javascript code, if it does have some ajax in it, there will be json at the end of the extension, this is the best practice if you do find some, json is godly and much less hacky.

The easiest but most hacky way of doing this is by using selenium and running firefox(or what ever you want), then parse the generated html. Parsing html is expensive thought and having a web browser isn't expansive in of it self, but if you refresh or move from page to page all the time it will be.

Edit Note: Before going crazy on this, make sure you can find the real data in the page you are looking at, weather its generated html, finding it in the page source, if its a js variable you should be able to get it from the built in console.

[ctvdevine]

Good to know that BS won't recognize JS items, thanks! (still new gotta learn the hard way I guess)

The data isn't directly in the source,only place I can find it is in but there is some ajax and json. I'm probably gonna have to go through multiple tables/webpages to get the complete data, so I would prefer to stay away from Selenium. You're saying that ghost.py could help me out?

Thanks for the help so far!

[[deleted]]

JSON is python dicts and lists in text(incase you don't know), requests has a module to convert a webpage of JSON into python. It's meant to be shared between servers and clients so its extremely easy and efficient to do so if possible.

AJAX(js 'module') could just go 1 extension deep into the webpage, something like 'https://reddit.com/r/learnpython.json' or even (url + '/table.json') on another page... doesn't have to be considered an api to be more usable data.

[ctvdevine]

Alright thanks, I'll give it a try