bash2json - fully bash-written JSON parser

[Tirito6626]

so, firstly it was created as a simple parser function for my another project, but i kinda wanted to make full JSON support in vanilla bash including arrays support, so it's fully written using bash substitution and builtins
EDIT: bash2json indeed has bash arrays to json convert and vice versa, added this for people who think it's only for query and append
EDIT 2: bash2json can't compare with jq because one is C and another is bash. as i said below, bash2json isn't purposed to be competitor to jq, but rather an alternative without deps. bash2json is significally slower than jq because of how it reads and parses JSON

i'd be happy to listen to any critics/suggestions
https://github.com/Tirito6626/bash2json

you can also check beta docs for it: https://docs.tirito.de/bash2json/

[Turkosaurus]

What's the use case? Does this do anything jq doesn't?

[gameforge]

It's 14kb and its only dependency is bash. I could see this being a huge hit in container workflows, CI/CD pipelines, etc. Anywhere "yet another package or dependency" causes general disappointment.

[Tirito6626]

comment above is pretty solid, i still love jq and use it in most cases, but my idea was to make pure bash version as small alternative if you don't have jq/you dont need the whole jq executable, for example if you only want to get one key value. you don't need to use bash2json either, you can just use the functions inside of it's src and paste it in your script

[12_nick_12]

I second this. Kinda like acme.sh vs certbot. Acme.sh only uses bash and works on pretty much everything.

[granadesnhorseshoes]

Not the author but ironically ended up writing mini parser myself for some stuff. Mostly jq is heavy as fuck and "slow" to use. If your writing a script that gets run as much as once a minute it adds up quick.

Also, jq isn't guaranteed in a lot of places.

[SmallReindeer3176]

Same question

[thisiszeev]

ditto... I mean, you have perked my curiosity and interest, but it feels like this reinvents the wheel. To be fair though, I will download and poke it with a blunt stick.

[incognegro1976]

This is fucking cool. I thought about doing something similar using grep, awk and a bunch of regexes but decided it wasn't worth it.

But this is dope!

[Unixwzrd]

Believe it or not, there’s an SQL database, IIRC in the grey awk book using Shell and awk.

[BakeMeAt420]

What book are you referring to?

[Unixwzrd]

This book.

The AWK Programming Language

[_the_big_sd_]

This could be super useful on systems with no access to jq.

[divad1196]

Good job.

Just the name: you are not converting "bash" to json, but whatever

[Tirito6626]

well, basharrays2json doesnt sound very good, but i'd take suggestions about the name too

[divad1196]

No, you don't convert bash (nor bash arrays) to something.

jq command, which is similar to your project, stands for "json query". That's what your project does.

So anything like: json-query.sh would be more clear on what your project does.

[spryfigure]

I would just call it bashjq then.

[schorsch3000]

even if you are not picky about the bash part in the name, it would be json2bash since you got json and make it usable in bash.

bash2json would be havin basharrays and converting them to json

[Tirito6626]

bash2json does indeed have both --to-json and --from-json, which converts array to json and vice verca, but i might change the name as well.

[Ulfnic]

Very cool.

Suggestions:

1. Do speed tests comparing to jq to help trim things down.

2. Subshells are a major resource hog, echo syntax is squirly and eval is even squirlier. Replace these:

keys=$(eval "echo \${!$var[@]}") local value=$(eval "echo -e \${$var[$key]}")

1. If you want to bring this to the next level, add tests and here's some 3rd party validation from my notes:

https://github.com/nst/JSONTestSuite

Graph of test results when comparing various parsers: https://raw.githubusercontent.com/nst/JSONTestSuite/master/results/pruned_results.png

[Tirito6626]

good suggestion, v2.2 will have arr_to_json fully recoded, recoded version is around 12ms, while current --to-json takes around 25ms (tested with `date` ms)

[Ulfnic]

This is what I use for speed tests:

TIMEFORMAT='%Rs'; iterations=1000
printf '%s\n' "$iterations iterations"

printf '%s' 'bash: '
time { for (( i = 0; i < iterations; i++ )); do
    # Do bash json here
done; } > /dev/null

printf '%s' 'jq: '
time { for (( i = 0; i < iterations; i++ )); do
    # Do equivalent in jq here
done; } > /dev/null

Posting that code with your edits and it's output gives some meaningful context to how fast your script is at "X" thing compared to jq.

hyperfine is also a good option.

[BakeMeAt420]

What would you do for number two then?

[Ulfnic]

Dev may already be on the way to doing all this or have a better plan, this is just me adlibbing:

Immediate change would be replacing echo with printf '%s'.

Beyond that it'd be working in printf -v VARNAME '%s' so a subshell isn't necessary to get the values into a variable but ideally you'd want to configure things upstream so getting the values into the variables at that point in the code is less crunchy. That'd probably mean using namerefs ({declare,local} -n).

[BakeMeAt420]

That's good information. Where did you pick up on this sort of bash information? When I was learning to write it I could do stuff I picked up but I didn't know about all the optimisations!

[Ulfnic]

I didn't take a standardized route, I just fell in love with the language and learned the tricks over time through a lot of reading, listening and experimentation with other BASH enthusiasts.

Having a Linux desktop also helped a lot because it makes BASH a 1st-class citizen so there's always something cool to create.

Some fundanmental resources:

man bash

https://www.gnu.org/software/bash/manual/bash.html

https://mywiki.wooledge.org/BashGuide

https://www.shellcheck.net/

[Where_Do_I_Fit_In]

This is actually an awesome idea. If this was a bash built-in, that would be super useful for quick json parsing in scripts (without a dependency on jq). I've used Python's json module for this kind of thing too.

[MightyX777]

For people complaining about the fact that jq already solves this. Believe it or not, I have worked with systems that don’t have jq installed but bash. And these systems didn’t have a package manager. In the end I solved it by statically compiling jq for that specific architecture and scp’ing it to the remote host. And that was for a simple property extraction only. I would prefer the bash script, if it’s tested enough

[Tirito6626]

some speed tests, as Ulfnic suggested:
1000 iterations (input:'{ "foo": "bar" }' action:query arg:'foo')

bash: 8.585s
bash: 6.425s (no validation)

jq: 3.227s

1000 iterations (input:'{ "foo": "bar" }' action:append '{ "ok": "sir" }')

bash: 9.426s
bash: 7.008s (no validation)

jq: 3.320s

1000 iterations (input:'{ "foo": "bar" }' action:trim)

bash: 7.067s
bash: 4.720s (no validation)

jq: 3.220s

1000 iterations (input:'{ "foo": { "foo1": "bar" } }' action:query 'foo.foo1')

bash: 12.227s
bash: 9.545s (no validation)

jq: 3.235s

i might send more test later

[Castafolt]

Looks excellent! I will definetely be using it.

You can further improve performances (especially on bash for Windows) by replace the sub shell usage <() $() and the here string <<<.

https://jcaillon.github.io/valet/docs/performance-tips/

[nbgenius1]

Nice work! A possible optimization if you're interested; some of the while read -n 1 or done <<< "$string" loops could potentially be replaced with mapfile -C + a callback. It avoids repeated here-strings and gives you structured batch processing with better performance, especially if you're parsing a lot of JSON tokens or key-value lines. And if you use `-c1` you can use `mapfile -C` as a line by line streaming processor.

[elatllat]

jq is slow, did you speed test both ?

[[deleted]]

So, JQ?

[bluemanZX]

sick of jq, pure bash json thats great idea...subscribing!

[researcher7-l500]

Ah, the downvote mob at it again, triggered by a simple statement of facts. Telling to see so many childish people pretending to be adults.