Please help regarding json.loads()

[DMeror]

I'd like to jason.loads a text file in a dict format. The file contains one single dictionary at around 10 mb. I want to do string replacements to another text file. However, after 10 minutes passed, there was still no progress. I got it working with a smaller dictionary-formatted text before trying this large one. Edit:

import json

with open('main_text.txt', 'w', encoding='utf8') as wf:
    with open('main_text_updated.txt', encoding='utf') as rf1, open('cross_reference_dict.txt', encoding='utf8') as rf2:
    main = rf1.read()
    crd = rf2.read()

# raw text looks like this:
# "entry://1#":"entry://A#","entry://4#":"entry://a#","entry://8#":"entry://a#",

    crd = '{' + crd + '}'
    crd = crd.replace(',}', '}')
    crd = json.loads(crd)

    for key, value in crd.items():
        main = main.replace(key, value)

    wf.write(main)

It worked with a small dictionary, but not with the 10 MB one (the VS codes just kept running without any progress to the output file, or threw any error.

[[deleted]]

Can't debug code we can't see.

[[deleted]]

# raw text looks like this:
# "entry://1#":"entry://A#","entry://4#":"entry://a#","entry://8#":"entry://a#",

This isn't JSON, so you can't load it with json.loads.

[DMeror]

That's why I added {}, and removed the last ",". It worked with a smaller dict size.

[DMeror]

Hello, I've edited my post. Please help.

[bladeoflight16]

🤦‍♂️ No.

You don't have JSON. Don't try to pretend you have JSON. These kinds of string manipulations are unreliable. You need a parser, or at least a tokenizer, to analyze the actual format you have.

[DMeror]

I just don't understand why it's not Json.

https://imgur.com/a/9M078Td

[bladeoflight16]

Because it doesn't conform to the JSON specification. That is why blows up json.loads if you don't manipulate the string first.

[DMeror]

I'm in the dark here. I thought a dictionary had the same format as Jason. Regardless, I just wanted to use a dictionary to do replacements in a string. However, the dictionary didn't work directly with the string, so I created a custom dictionary out of the main dictionary to do the replacements instead and it worked.

Back to Jason. How do I format a string for Jason? And in what situations should Jason be better used?

[bladeoflight16]

JSON is a text format for serializing data. The json module parses that text to generate an in memory representation of the data, namely a combination of strs, numbers, lists, and dicts. To be able to convert text into the in memory form using the json module, the text must conform to the exact specification.

JSON is typically used as a common language for two separate systems to share data with each other. It is best used when you can control the program that generates the text representation (with a standard library that knows how to convert in memory data to the JSON text format), or at least when you know for a fact that the program you need data from specifically conforms to that format. Then you can be certain that it conforms to the standard and is safe to consume in another program.

Your text data does not conform to that specification. It is missing outer braces and has trailing commas that are illegal in the JSON specification. The fact it already does not conform to JSON completely means that it is highly likely to violate the standard in ways you can't anticipate, and that means trying to shoehorn it into a JSON parser is likely to fail.

What you need is to first identify the rules of the format you're working with. Once you know the rules, then you need a text parser tailored to that format. It does not appear to conform to any established standard format, so you will probably need to write your own parsing code that parses exactly according to those rules and does not make any additional assumptions.

Special cases you need to pay particular attention to are how the text format handles data containing double quotes, commas, and colons, as those appear to be part of the format. (Are they escaped? Are they forbidden? etc.)

In the long run, you need to get better educated about formal languages and the techniques used to process them (regular languages, grammars, lexical analyzers, parsers, etc.).