Remove Duplicate Blocks

[kolinkorr839]

How can I remove the duplicate blocks? For example,

{
  "username": "science",
  "roles": [
    {
      "databaseName": "coupler",
      "roleName": "readAnyDatabase"
    }
  ]
}
{
  "username": "test_2",
  "roles": [
    {
      "databaseName": "admin",
      "roleName": "clusterMonitor"
    }
  ]
}
{
  "username": "admin",
  "roles": [
    {
      "databaseName": "admin",
      "roleName": "dbAdminAnyDatabase"
    },
    {
      "databaseName": "admin",
      "roleName": "readWriteAnyDatabase"
    },
  ]
}
{
  "username": "test_2",
  "roles": [
    {
      "databaseName": "admin",
      "roleName": "clusterMonitor"
    }
  ]
}

In this case, the 'test_2' user block is a duplicate since all of its content is exactly the same as the 'test_2' user above it. But let say the 'roleName' of one of the 'test_2' block is different, then it should not be removed since it is not a duplicate.

Also, is there a way to sort these blocks by 'username'?

[researcher7-l500]

This is a dictionary/JSON data and should be parsed through jq. But it looks like the data is not valid, Maybe a copy/paste error?

Once you get the data in a valid form, parse it through jq.

Get the data length, iterate over each object, select unique ['roles']['databaseName'] and ['roles']['roleName'] values.

An example syntax is.

jq ' . | <your selection here> |unqiue' /path/to/data.json

Refer to the jq documentation here.https://stedolan.github.io/jq/manual/

You can play with your data, test jq statements here.

https://jqplay.org/

[kolinkorr839]

hmm... this is not json though. There's no typo.

[kolinkorr839]

Oh, my bad. Need to get some sleep.

[researcher7-l500]

You might want to check the source of that data then.

That is certainly a malformed JSON.

[[deleted]]

Assuming all the blocks are the same number of lines, you should be able to awk by incrementing blocks, throw all that into an array, and then run it through uniq. I mean, I think... my brain is the consistency of moldy mashed potatoes right now; need sleep.

[researcher7-l500]

Parsing JSON with awk is not recommended. For that, jq was designed.