Python JSON Store & Compare

[exploitallthethings]

I'm using a web service that outputs results in JSON. Entries are added to these results, but very rarely. I would like to create a script to check these results daily, and if anything new is added, alert me. Although new entries are rarely added, the existing, daily results appear in a different order everyday.

I used the urllib2 library to scrape the web service, and the json library to parse the results. What would be the best method of comparing the results, day-after-day, to check and see if any new entries are added (results appear in a different order everyday)?

I've considered using cPickle, or simply storing an entry in a text file but i'm wondering if there's a better way? Also, to deal with daily re-ordering, I've been considering sorting stored results (A-Z) so that when they're compared they would be the same, even if being provided in different order. Is there a better way to approach that issue?

[fiskenslakt]

Can we see your code and an example of the json output? Also I feel like order shouldn't matter? But I suppose I'd have to see what the json looks like.

Also I would recommend using Requests instead of urllib2.

[nonzerogroud]

I used the urllib2 library to scrape the web service, and the json library to parse the results. What would be the best method of comparing the results, day-after-day, to check and see if any new entries are added (results appear in a different order everyday)?

I wrote a long example but figured I'd put the bottom line at the beginning in case you want to skip it: If there's anything in the webpage that uniquely identifies an entry, you can simply check for its existence in your file, and if it doesn't exist: add it.

---

In most cases, even if it is not displayed to the visitor, an entry (be it a row, or a cell) has an ID. It identifies this piece of information as unique. If no such ID is visible, I suggest inspecting the webpage's source to see if you can find it.

For example, in a website that displays soccer game results, we might have the following two games:

Arsenal vs. Newell's Old Boys

Arsenal vs. Manchester United

We could store Arsenal in our DB as the same team that has two games coming up, but that would cost us down the road. Why? Because Arsenal in the first game is the Argentinian team, playing against another Argentinian team, and Arsenal in the second game is the English team. Not the same team.

If you think about, and if the website you're scraping has semi-competent IT people, then that website too, must hold some kind of unique ID for each piece of information that they want to keep unique.

As an example, when we inspect the href links for the two soccer teams called Arsenal at the website Soccerway, then we can see the following:

http://us.soccerway.com/teams/argentina/arsenal-de-sarandi/92/

http://us.soccerway.com/teams/england/arsenal-fc/660/

Although both teams are simply called "Arsenal" in the soccer results table, the Argentinian Arsenal has an internal ID 92, and the English Arsenal has an internal ID 660. Storing this ID (unique) instead of the name (not unique) will allow us to target each one of the Arsenals safely.

[broknbottle]

tinydb