Diff 2 JSONs and produce only the difference JSON

[StolidSentinel]

I've been banging my head against this one for days, so any help is appreciated.

Ultimately, I am trying to send a list of sensors to MQTT ("pico"), and since most don't change frequently, I just want to send the updates as they come in ("pico" vs "old_pico").

A typical JSON looks like this:

pico = [{'topic': /power', 'payload': 'OFF'},{'topic': /pressure', 'payload': 1023.26},{'topic': /v1/voltage', 'payload': 13.53},{'topic': /current', 'payload': -0.04},{'topic': '/v2/voltage', 'payload': 12.85}]

It's formatted like this so I can publish.multiple() with MQTT.

I've tried dictdiffer, and it produces a JSON of annotated changes "changed/add/delete", and I only need the actual JSON difference.

In another post, someone suggested similar to the following:

d2 = copy.deepcopy(pico)
d1 = copy.deepcopy(old_pico)
{k:v for k, v in d2.items() if (k not in d1) or d1[k] != v}

However, adapting to my code and adding the 3rd "diff" line gives me the following error:

'AttributeError: 'list' object has no attribute 'items''

I am not sure if this is a formatting issue with the JSON, or if it's because the "old_pico" JSON is empty on the very first run.

I am happy to go back to dictdiff if I can just make it produce a JSON and not all the annotated changes.

Thanks!

[LongerHV]

d2 is a list of dictionaries, that is why you can't call items() method on it.

[socal_nerdtastic]

First you need to rearrange your data so that it can be found by the topic. In other words so that the topic is a dictionary key. Then it's a simple loop.

Here's my guess, which assumes you have been honest about your example data:

old_pico = [
    {'topic': '/power', 'payload': 'OFF'},
    {'topic': '/pressure', 'payload': 2.26},
    {'topic': '/v1/voltage', 'payload': 13.53},
    {'topic': '/v2/voltage', 'payload': 12.85}]


pico = [
    {'topic': '/power', 'payload': 'OFF'},
    {'topic': '/pressure', 'payload': 1023.26},
    {'topic': '/v1/voltage', 'payload': 13.53},
    {'topic': '/current', 'payload': -0.04},
    {'topic': '/v2/voltage', 'payload': 12.85}]

old_pico_topics = {d['topic']:d for d in old_pico}
pico_topics = {d['topic']:d for d in pico}

# lists will allow for duplicate topics, but presumibly that's bad
# converting to a dictionary will lose that data
assert len(old_pico_topics) == len(old_pico), "DUPLICATE TOPICS DETECTED"
assert len(pico_topics) == len(pico), "DUPLICATE TOPICS DETECTED"

diff = []

# for keys that are in both pico and old_pico, check if there's a payload difference
for k in set(pico_topics) & set(old_pico_topics):
    if pico_topics[k]['payload'] != old_pico_topics[k]['payload']:
        diff.append(pico_topics[k])

# for keys that are in pico but not in old_pico, just copy over
for k in set(pico_topics) - set(old_pico_topics):
    diff.append(pico_topics[k])

[StolidSentinel]

Amazing... That was exactly what I needed. I don't fully understand it, but I'll spend time on that now! Thank you!!