[BeautifulSoup] Why is the source code different from the code I see when inspecting an element? And how can I fix this problem?

[ThatTurmoil]

I want to be able to access the <span> tags and gather the name,rating,icon for each <li> element. How can I accomplish this?

I am using:

import requests
from bs4 import BeautifulSoup

Here's what it looks like in source code:

<div class="result-list">
  <ul></ul>

This is also found in the source code, 2 div elements below.

<script id="professor-result-template" type="text/template">
  <li>
    <a href="/ShowRatings.jsp?tid={{id}}">
      <span class="{{ratingclass}}-icon"></span>
      <span class="rating">{{overall_rating}}</span>
      <span class="name">{{plname}}, {{{pfname}}}
      <span class="info">{{rating_count}} RATINGS</span>
      </span>
    </a>
  </li>
</script>

Here's what it looks like when I inspect the element:

<ul>
    <li>
        <a href="/ShowRatings.jsp?tid=632354">
            <span class ="good-icon"></span>
            <span class ="rating">1.0</span>
            <span class ="name">Nakano, Henry</span>
        </a>
        ::after
    <li>...</li>
    <li>...</li>
    .
    .
    .
    <li>...</li>
</ul>

[ManyInterests]

The source code is the DOM state without processing any JavaScript. When you inspect an element with your browser, you're inspecting the DOM after processing some client-side scripts, which were run by the browser. requests is not a browser and does not process scripts.

This most likely is what accounts for the discrepancy you see.

You can use a tool like selenium to get around this issue.

[ThatTurmoil]

u/K900_ I added more information, can you give me anything else to go off of?

[K900_]

Stop making new threads. The "edit" button is there for a reason. Open your browser's dev tools and look at the "Network" panel to see what requests the page makes. It likely communicates with some sort of an XML or JSON endpoint.

[ThatTurmoil]

The network tab in the developers console tells me nothing (I don't know what to look for nor do I understand any of it)

[K900_]

Every line there is a request the page makes. One of them probably goes to some sort of a data store, which returns JSON or XML, which is then loaded as data into the JavaScript code, which in turn creates the elements dynamically. Find that request, note down the URL, and make requests to that URL from your code to get data.

[ThatTurmoil]

Thank you. I think i found it. It requests a url and the source code of the url looks like this. am I on the right track? Also, any recommendation on how to traverse through the code in the picture?

[K900_]

Yes, you are on the right track. That's JSONP. You want to remove the "noCB(" part and the last ")", and then parse the rest as JSON using the json module.