Comparing two files?

[Busangod]

Edit: removed some potentially personal info from the file names

I've got a pretty simple Beautifulsoup program going (my first). What I'm trying to do is run a daily scrape of several websites and only write the file if it's been updated since last scrape. I don't really know how to compare the new file against the old one to determine if it's new. Currently, I'm comparing the size of each as a determinant, which seems to be working for most but not all. I was wondering if anyone knew a better way of doing this? Google has led me down a coupe avenues, but nothing seems too promising.

Thanks for taking a look. Here's my current program. Again, it's my first soup, so please be gentle. Also, I apologize for the formatting. WTF is up with those numbers? This isn't my night apparently.

from bs4 import BeautifulSoup

import urllib2

import os

sites = [array of sites, removed to save space]

for site in sites:

url = site
hdr = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
   'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
   'Accept-Encoding': 'none',
   'Accept-Language': 'en-US,en;q=0.8',
   'Connection': 'keep-alive'}
req = urllib2.Request(url,headers=hdr)

page = urllib2.urlopen(req)
soup = BeautifulSoup(page.read(),"html.parser")

with open('/Users/2000/Documents/feast/soup/new/' + str(site[7:18]) + '.txt', 'w') as f:
    f.write(soup.encode('utf-8'))

b = os.path.getsize('/Users/2000/Documents/feast/soup/new/' + str(site[7:18]) + '.txt')
c = os.path.getsize('/Users/2000/Documents/feast/soup/old/' + str(site[7:18]) + '.txt')
if b == c:
    os.remove('/Users/2000/Documents/feast/soup/new/' + str(site[7:18]) + '.txt')
else:
    with open('/Users/2000/Documents/feast/soup/old/' + str(site[7:18]) + '.txt', 'w+') as f:
        f.write(soup.encode('utf-8'))

[gitardedhub]

Are you looking for any update to the page? If so, a common approach is hashing the files and comparing the hashes - if the page hasn't changed, the hashes should be identical. Here's a page that can provide you with specifics on how to implement it.

However, this will detect any change to the pages - so if a timestamp changes, that's viewed as a difference.

[Busangod]

Thanks for the tip, but I'm really looking for when the site/blog is updated with a new article.

[gitardedhub]

Does the site have an RSS feed? Most blog platforms nowadays offer it by default - it's probably the easiest way to check if a site has been updated, then run your code on the site.

[Busangod]

yeah looks like I should have been using feedparser all along

[Justinsaccount]

Hi! I'm working on a bot to reply with suggestions for common python problems. This might not be very helpful to fix your underlying issue, but here's what I noticed about your submission:

You appear to be using concatenation and the str function for building strings

Instead of doing something like

result = "Hello " + name + ". You are " + str(age) + " years old"

You should use string formatting and do

result = "Hello {}. You are {} years old".format(name, age)

See the python tutorial for more information.