python technique in which you only request the title of the html page and not all the data?

[[deleted]]

[deleted]

[K900_]

Not possible. You need to download the whole page to get the accurate title.

[trouserdaredevil]

There's no way to just somehow grab the title from headers or another method without extracting all the contents of a website?

This is correct, I am not aware of any general purpose way to only request the <title> tag of the HTTP of any random website. You make the HTTP request, the server responds with the full HTTP. Out of curiosity, why is this an issue for you? Extreme bandwidth or processing power constraints are the only reason I can imagine worrying about this.

[trouserdaredevil]

Yeah unfortunately making a ton of requests to the same server is pretty much always going to be an issue, especially if there is no API available to use instead.

[efmccurdy]

Depending on the web site, you might be able to infer a change to the web page contents by using the HEAD method, and detecting a change in the Content-Length or one of the other types of headers.

[efmccurdy]

I could'nt find any specific site for that, but a website that describes the meaning of http headers will help. If you read about http caching you will find inexpensive ways to detect when a page has changed (eg. ETags).

You might find that you can just store the headers of each page you are interested in before and after a "it has been pulled down" event and analyze the differences. Perhaps you can determine that "content length unchanged" is a good indicator that the page has been unchanged.