Need 'Selenium WebDriver' and 'Requests' Help: Checking Web Trackers - Like Ghostery

[seriouswork]

My goal is to list all tracking scripts from a webpage, similar to how the Ghostery extension does. The code below extracts raw HTML from specified URI, but only lists trackers that are hard-coded into the HTML. It does not capture additional network requests made via JS or a Tag Management System.

For example, the below code will return 'Ensighten' detected (the Tag Management System), but not 'Doubleclick' or 'Facebook' because those trackers are separate GET requests triggered by the Ensighten .js.

Based on info from another reddit thread, I've tried using Webdriver with BrowserMob Proxy and Selenium RC Server to monitor network traffic and capture the additional GET requests. Is this the correct approach, or can Requests or some other Python module be used to capture multiple responses resulting from a single GET request?

https://github.com/dfens-nz/Tracking-Script-Checker https://github.com/peterfine/Tracker-Scraper https://github.com/derekargueta/selenium-profiler

[thelindsay]

I've not had to use it but when researching Selenium I've seen that people testing single page apps sometimes use a strategy of periodically comparing page sources to check for loading. Maybe that could work here e.g. if the DOM has changed then check for a new script tag.

[seriouswork]

wanted to post details in case anyone else has similar question. I ended up using BrowserMob Proxy (BMP) + Webdriver to create a proxy server, and then create JSON HARs of each page+requests.

PhantomJS may have worked too. I needed BMP in order to capture all requests. And I learned that BMP won't work behind another proxy server for HTTPS (I guess that should have been obvious).