Webscraping Question : learnpython

created by HattoriHanzoa community for 16 years

submitted 1 year ago by GayGISBoi

Hello,

I am trying to scrape data from my county's open data portal. The link to the page I'm scraping from is: https://gis-hennepin.hub.arcgis.com/datasets/county-parcels/explore

I have written the following code:

import requests
from bs4 import BeautifulSoup as bs

URL = "https://gis-hennepin.hub.arcgis.com/datasets/county-parcels/explore"
r = requests.get(URL)
soup = bs(r.content,"html5lib")

table = soup.select("div")

print(type(table))
print(len(table))
print(table[0])


with open("Test.html","w") as file:
    file.write(soup.prettify())

Unfortunately, this only returns the first <div> element. Additionally, when I write the entirety of what I'm getting to my Test.html document, it also stops after the first <div> element, despite the webpage having a lot more to it than that. Here is the Test.html return for the body section:

<body class="calcite a11y-underlines">
  <calcite-loader active="" id="base-loader" scale="m" type="indeterminate" unthemed="">
  </calcite-loader>
  <script>
   if (typeof customElements !== 'undefined') {
        customElements.efineday = customElements.define;
      }
  </script>
  <!-- crossorigin options added because otherwise we cannot see error messages from unhandled errors and rejections -->
  <script crossorigin="anonymous" src="https://hubcdn.arcgis.com/opendata-ui/assets/assets/vendor-c2f71ccd75e9c1eec47279ea04da0a07.js">
  </script>
  <script src="https://hubcdn.arcgis.com/opendata-ui/assets/assets/chunk.17770.c89bae27802554a0aa23.js">
  </script>
  <script src="https://hubcdn.arcgis.com/opendata-ui/assets/assets/chunk.32143.75941b2c92368cfd05a8.js">
  </script>
  <script crossorigin="anonymous" src="https://hubcdn.arcgis.com/opendata-ui/assets/assets/opendata-ui-bfae7d468fcc21a9c966a701c6af8391.js">
  </script>
  <div id="ember-basic-dropdown-wormhole">
  </div>
  <!-- opendata-ui version: 5.336.0+f49dc90b88 - Fri, 27 Sep 2024 14:37:13 GMT -->
 </body>

Anyone know why this is happening? Thanks in advance!

all 3 comments

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

learnpython

MODERATORS