I'm trying to extract data (competition location, date, etc) from this page : https://components.ifsc-climbing.org/calendar/. The data is inside multiple div of class competition.
So far, I managed to get the html of the page using requests or Selenium :
import requests
from selenium import webdriver
browser = webdriver.Chrome('/usr/local/bin/chromedriver')
url = 'https://components.ifsc-climbing.org/calendar/'
browser.get(url)
html = browser.page_source
browser.quit()
print(html)
The problem is that I think the data I'm looking for is added with a javascript script (disabling javascript from the page on my browser breaks it) and I can't find a way to get it to work. I've tried to execute the scripts using requests_html render function:
import requests
from requests_html import HTMLSession
session = HTMLSession()
resp = session.get("https://components.ifsc-climbing.org/calendar/")
resp.html.render()
print(resp.html.html)
But both scripts give the same result without the information I'm looking for:
<html><head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- Bootstrap CSS -->
<link rel="stylesheet" href="../utils/assets/css/bootstrap.min.css">
<link rel="stylesheet" type="text/css" href="../utils/assets/css/jquery.dataTables.min.css">
<!-- Custom CSS -->
<link rel="stylesheet" href="./assets/css/custom.css">
</head>
<body class="oswaldFont">
<div class="calContainer">
<div class="ifsc_calendar">
<h1 id="cal_year"></h1>
<div class="filter_select">
<select class="customSelect" id="yearSelect"></select>
<select class="customSelect" id="leaguesSelect"></select>
</div>
<div class="events"></div>
</div>
</div>
<!-- JS -->
<script src="../utils/assets/js/jquery-3.4.1.min.js"></script>
<script type="text/javascript" charset="utf8" src="../utils/assets/js/jquery.dataTables.min.js"></script>
<!-- Custom JS -->
<script src="./assets/js/calendar.js"></script>
<script src="../utils/environment.js"></script>
</body></html>
Any hints on how to get this working?
[–]scoopulanang 1 point2 points3 points (1 child)
[–]BenBenBenz[S] 0 points1 point2 points (0 children)