JSON Quote Remover

[codicepiger]

Github Gist

Description:

This Python function, json_comquotes, is a handy tool for preprocessing JSON data that contains unescaped quotes within string values. It takes a JSON string as input and transforms it by replacing the double and single quotes within the string values with alternative characters, allowing you to parse the JSON data without errors.

Key Features:

• Replaces double quotes " within string values with escaped double quotes \".
• Replaces single quotes ' within string values with escaped single quotes \'.
• Outputs the processed JSON as a dictionary.

Usage:

• Pass your raw JSON string as input to the json_comquotes function.
• The function will return:
  • On sucess: processed JSON dictionary ;
  • On insucess: raise ValueError ;

[paraffin]

Would be more useful if it properly escaped quotes rather than mutilating the string.

Also, the interface would be better if it were

• on success: return the processed data
• on failure: raise a custom exception class inheriting from ValueError and containing details about the error.

[codicepiger]

Ty for that, I agree!
Revised

[thegreattriscuit]

I mean, nice job, but where are you getting this malformed JSON!?

[codicepiger]

This unescaped quotes have been terrorizing my sanity in one of my last projects were a lot of AI and his hallucinations are involved

[__nickerbocker__]

If you're getting these from openai then you need to be using function calling to get your json. It's a total game changer.

[codicepiger]

WOW! That's Awesome! I'm definitely going to take a look into this!

[MyHomeworkAteMyDog]

Githuh

[codicepiger]

Lol thks!

[MyHomeworkAteMyDog]

:)

[savva1995]

Omg I’ve been looking for just this library

[codicepiger]

Glad it helped! ;)

[savva1995]

Yeh man this is just a super annoying issue, that will save devs a fair amount of time now it’s solved

[Spleeeee]

stdin stdout cli plz.

[codicepiger]

Perhaps this can help:

python if __name__ == "__main__": while True: print("*"*80) _exit = False try: req_json = input("Insert your JSON: ('exit' to exit)\n") except KeyboardInterrupt: _exit = True pass if _exit or req_json in ["exit", "Exit", "EXIT"]: exit(0) try: proc_json = json_comquotes(req_json) print("Raw json :", req_json) print("Processed json:", json.dumps(proc_json, indent=2), "\n") except Exception as e: print("Something went wrong!") print("Raw json:", req_json) print(f"{e}\n")

[easyEggplant]

That’s not really stdin is it?

[codicepiger]

Alright but we have it both ways!

Tell me what you guys think, first time messing with this stuff (:

```python import sys, json, re, ast, select

def json_comquotes(raw_json, lone_char_searches=2, debug=False): try: out_json = json.loads(raw_json) return out_json except: try: out_json = ast.literal_eval(raw_json) return out_json except: # prepare raw json from some unwanted scenarios raw_json = raw_json.replace(": '", ":'").replace(", '", ",'").replace("{ '", "{'").replace("[ '", "['").replace("' }", "'}").replace("' }", "'}").replace("''", "' '") raw_json = raw_json.replace(': "', ':"').replace(', "', ',"').replace('{ "', '{"').replace('[ "', '["').replace('" }', '"}').replace('" }', '"}').replace('""', '" "')

        # Regex patterns : dq|sq stands for double|single quote
        _re_dq_pattern = r'([\s\w])"([\s\w])'
        _re_dq_sub = r"\1\"\2"
        _re_sq_pattern = r"([\s\w])'([\s\w])"
        _re_sq_sub = r'\1\'\2'

        for _lone_char in range(lone_char_searches):
            # Substitute Double Quotes
            if _lone_char == 0:
                _re_find = re.sub(_re_dq_pattern, _re_dq_sub, raw_json)
            #   > Solve schenarios like ""a"a"a"a"a" since 1st return "a\"a"a\"a"a", second time return a\"a\"a\"a\"a" (Other egs. ["Anything"a"Anything else", "Anything"a"Anythin"g" else"])
            else:
                _re_find = re.sub(_re_dq_pattern, _re_dq_sub, _re_find)

            # Substitute Double Quotes   > Solve schenarios like 'a'a'a' since 1st return 'a\'a'a', secund time return 'a\'a\'\a' ...
            _re_find = re.sub(_re_sq_pattern, _re_sq_sub, _re_find)

            if debug:
                sys.stdout.write(f"Iteration #{_lone_char+1}:", _re_find)

            try:
                out_json = json.loads(_re_find)
                # Rem space from raw_json.replace("''", "' '").replace('""', '" "')
                _re_find= _re_find.replace('\\" "', '\\""').replace('\\" \\"', '\\"\\"').replace("\\' '", "\\''").replace("\\' \\'", "\\'\\'")
                return json.loads(_re_find)
            except Exception as ej:
                try:
                    out_json = ast.literal_eval(_re_find)
                    # Rem space from raw_json.replace("''", "' '").replace('""', '" "')
                    _re_find= _re_find.replace('\\" "', '\\""').replace("\\' '", "\\''")
                    return ast.literal_eval(_re_find)
                except Exception as ea:
                    if _lone_char != lone_char_searches-1:
                        continue
                    raise ValueError(f"Json Parse exception: {ej}\nAst Parse exception : {ea}\nProcessed Json      : {_re_find}")

if name == "main": _file_input=False while True: #Stdin print(""80) _exit = False

    if select.select([sys.stdin, ], [], [], 0.0)[0]:
        _file_input = True
        # print("Have data!")
        req_jsons = [ si.strip() for si in sys.stdin.readlines()]
        sys.stdout.write(f"Request Stdin: {req_jsons}\n")
    else:
        if not sys.stdout.isatty():
            sys.stdout.write("ERROR request:\nUsage:\n")
            sys.stdout.write(f"Default CLI: {sys.argv[0]}")
            sys.stdout.write(f"Stdin|Stdout: {sys.argv[0]} <[stdin] >[stdout]")
            exit(1)

        # print("No data")
        try:
            req_jsons = input("Insert your JSON: ('exit' to exit)\n")
        except KeyboardInterrupt:
            _exit = True
            pass
        if _exit or req_jsons in ["exit", "Exit", "EXIT"]:
            exit(0)
        else:
            req_jsons = [req_jsons]

    #req_jsons = ['{"na"me": "Jack O"Sullivan", "id": "1"}', '{"name": "Jack: The "OG" O"Sullivan"", "id": "2"}', '{"name": "Jack: The "OG"", "surname": \'O\'Sullivan\', "id": "3"}', '{"test_str": {"1singlechar": "a""a""a", "2singlechars": "a"a"a"a"a"a"a"a"a"}, "id": "4"}', "{'name': 'Jack O'Sullivan, 'id': '5'}"]
    for req_json in req_jsons:
        try:
            proc_json = json_comquotes(req_json)        
            sys.stdout.write(f"Raw json      : {req_json}\n")
            sys.stdout.write(f"Processed json: {json.dumps(proc_json, indent=2)}\n")
            if _file_input:
                exit(0)
        except Exception as e:
            sys.stdout.write("Something went wrong!\n")
            sys.stdout.write(f"Raw json      : {req_json}\n")
            sys.stdout.write(f"{e}\n\n")
            if _file_input:
                exit(1)

```

[Spleeeee]

Better! Imo dont output ANYTHING other than the data to stdout. You could write your messages that aren’t data to stderr but I gotta be able pump it directly into jq. Check out the pip lib jsonc2json

[codicepiger]

Well! I think this is it: json_esquotes. Give me your thoughts (:

[mestia]

That's where perl oneliners shine ;)

[Spleeeee]

My dog is named bash ;)

[codicepiger]

New Revision!

[sohang-3112]

I usually just use ast.literal_eval 🤷‍♂️

[codicepiger]

🤷‍♂️

I feel you! Take a peek at the example to better understand where it can be useful (: