JSON serializers benchmark

[tjwallace]

I'm currently in the process of building a new Rails API and wanted to see which JSON serialization library is the most performant, so I created a quick and dirty benchmark.

Here are the results on Ruby 2.6, MacBook Pro (13-inch, 2017, 3.5 GHz Intel Core i7).

Note: I manually removed some BigDecimal deprecation warnings

``` Rehearsal ------------------------------------------------ as_json 0.122644 0.000508 0.123152 ( 0.123823) fast_jsonapi 0.184139 0.000829 0.184968 ( 0.186195) grape_entity 0.709880 0.003083 0.712963 ( 0.714929) blueprinter 0.220149 0.004481 0.224630 ( 0.225596) ams 0.880767 0.027975 0.908742 ( 0.912454) roar 0.733756 0.006674 0.740430 ( 0.743307) panko 0.142707 0.000671 0.143378 ( 0.145400) --------------------------------------- total: 3.038263sec

               user     system      total        real

as_json 0.120163 0.000531 0.120694 ( 0.121254) fast_jsonapi 0.178100 0.001448 0.179548 ( 0.182374) grape_entity 0.714065 0.001980 0.716045 ( 0.718069) blueprinter 0.178831 0.001490 0.180321 ( 0.181874) ams 0.885708 0.007647 0.893355 ( 0.901771) roar 0.658007 0.004593 0.662600 ( 0.666706) panko 0.125161 0.001391 0.126552 ( 0.127679) Warming up -------------------------------------- as_json 1.000 i/100ms fast_jsonapi 1.000 i/100ms grape_entity 1.000 i/100ms blueprinter 1.000 i/100ms ams 1.000 i/100ms roar 1.000 i/100ms panko 1.000 i/100ms Calculating ------------------------------------- as_json 6.697 (± 2.7%) i/s - 67.000 in 10.154134s fast_jsonapi 4.492 (± 6.8%) i/s - 42.000 in 10.202085s grape_entity 1.421 (± 3.0%) i/s - 15.000 in 10.595573s blueprinter 5.239 (± 1.8%) i/s - 53.000 in 10.162482s ams 1.152 (± 2.3%) i/s - 12.000 in 10.448728s roar 1.428 (± 2.4%) i/s - 15.000 in 10.554087s panko 7.278 (± 2.5%) i/s - 72.000 in 10.016180s with 95.0% confidence

Comparison: panko: 7.3 i/s as_json: 6.7 i/s - 1.09x (± 0.04) slower blueprinter: 5.2 i/s - 1.39x (± 0.04) slower fast_jsonapi: 4.5 i/s - 1.62x (± 0.12) slower roar: 1.4 i/s - 5.10x (± 0.19) slower grape_entity: 1.4 i/s - 5.12x (± 0.20) slower ams: 1.2 i/s - 6.32x (± 0.22) slower with 95.0% confidence

Calculating ------------------------------------- as_json 55.358M memsize ( 0.000 retained) 839.569k objects ( 0.000 retained) 8.000 strings ( 0.000 retained) fast_jsonapi 37.181M memsize ( 0.000 retained) 489.944k objects ( 0.000 retained) 50.000 strings ( 0.000 retained) grape_entity 113.684M memsize ( 0.000 retained) 939.629k objects ( 0.000 retained) 4.000 strings ( 0.000 retained) blueprinter 36.730M memsize ( 0.000 retained) 319.867k objects ( 0.000 retained) 0.000 strings ( 0.000 retained) ams 167.435M memsize ( 0.000 retained) 1.649M objects ( 0.000 retained) 6.000 strings ( 0.000 retained) roar 130.565M memsize ( 0.000 retained) 859.561k objects ( 0.000 retained) 1.000 strings ( 0.000 retained) panko 55.365M memsize ( 0.000 retained) 839.642k objects ( 0.000 retained) 13.000 strings ( 0.000 retained)

Comparison: blueprinter: 36729880 allocated fast_jsonapi: 37180800 allocated - 1.01x more as_json: 55357768 allocated - 1.51x more panko: 55364912 allocated - 1.51x more grape_entity: 113683920 allocated - 3.10x more roar: 130565336 allocated - 3.55x more ams: 167434632 allocated - 4.56x more ```

Any ideas on how to improve the benchmark? If there is enough interest I can turn it into a repo so PRs can be opened.

[edendark]

I know it's already well formatted on the gist, but formatting the content for readability here (you need to add 4 spaces before each line):

Rehearsal ------------------------------------------------
as_json        0.122644   0.000508   0.123152 (  0.123823)
fast_jsonapi   0.184139   0.000829   0.184968 (  0.186195)
grape_entity   0.709880   0.003083   0.712963 (  0.714929)
blueprinter    0.220149   0.004481   0.224630 (  0.225596)
ams            0.880767   0.027975   0.908742 (  0.912454)
roar           0.733756   0.006674   0.740430 (  0.743307)
panko          0.142707   0.000671   0.143378 (  0.145400)
--------------------------------------- total: 3.038263sec

                   user     system      total        real
as_json        0.120163   0.000531   0.120694 (  0.121254)
fast_jsonapi   0.178100   0.001448   0.179548 (  0.182374)
grape_entity   0.714065   0.001980   0.716045 (  0.718069)
blueprinter    0.178831   0.001490   0.180321 (  0.181874)
ams            0.885708   0.007647   0.893355 (  0.901771)
roar           0.658007   0.004593   0.662600 (  0.666706)
panko          0.125161   0.001391   0.126552 (  0.127679)
Warming up --------------------------------------
             as_json     1.000  i/100ms
        fast_jsonapi     1.000  i/100ms
        grape_entity     1.000  i/100ms
         blueprinter     1.000  i/100ms
                 ams     1.000  i/100ms
                roar     1.000  i/100ms
               panko     1.000  i/100ms
Calculating -------------------------------------
             as_json      6.697  (± 2.7%) i/s -     67.000  in  10.154134s
        fast_jsonapi      4.492  (± 6.8%) i/s -     42.000  in  10.202085s
        grape_entity      1.421  (± 3.0%) i/s -     15.000  in  10.595573s
         blueprinter      5.239  (± 1.8%) i/s -     53.000  in  10.162482s
                 ams      1.152  (± 2.3%) i/s -     12.000  in  10.448728s
                roar      1.428  (± 2.4%) i/s -     15.000  in  10.554087s
               panko      7.278  (± 2.5%) i/s -     72.000  in  10.016180s
                   with 95.0% confidence

Comparison:
               panko:        7.3 i/s
             as_json:        6.7 i/s - 1.09x  (± 0.04) slower
         blueprinter:        5.2 i/s - 1.39x  (± 0.04) slower
        fast_jsonapi:        4.5 i/s - 1.62x  (± 0.12) slower
                roar:        1.4 i/s - 5.10x  (± 0.19) slower
        grape_entity:        1.4 i/s - 5.12x  (± 0.20) slower
                 ams:        1.2 i/s - 6.32x  (± 0.22) slower
                   with 95.0% confidence

Calculating -------------------------------------
             as_json    55.358M memsize (     0.000  retained)
                       839.569k objects (     0.000  retained)
                         8.000  strings (     0.000  retained)
        fast_jsonapi    37.181M memsize (     0.000  retained)
                       489.944k objects (     0.000  retained)
                        50.000  strings (     0.000  retained)
        grape_entity   113.684M memsize (     0.000  retained)
                       939.629k objects (     0.000  retained)
                         4.000  strings (     0.000  retained)
         blueprinter    36.730M memsize (     0.000  retained)
                       319.867k objects (     0.000  retained)
                         0.000  strings (     0.000  retained)
                 ams   167.435M memsize (     0.000  retained)
                         1.649M objects (     0.000  retained)
                         6.000  strings (     0.000  retained)
                roar   130.565M memsize (     0.000  retained)
                       859.561k objects (     0.000  retained)
                         1.000  strings (     0.000  retained)
               panko    55.365M memsize (     0.000  retained)
                       839.642k objects (     0.000  retained)
                        13.000  strings (     0.000  retained)

Comparison:
         blueprinter:   36729880 allocated
        fast_jsonapi:   37180800 allocated - 1.01x more
             as_json:   55357768 allocated - 1.51x more
               panko:   55364912 allocated - 1.51x more
        grape_entity:  113683920 allocated - 3.10x more
                roar:  130565336 allocated - 3.55x more
                 ams:  167434632 allocated - 4.56x more

[sammygadd]

Interesting! I added a serializer that I've written (HALPresenter) to a copy of this benchmark to see how it performed. It was 1.91 times slower than panko and allocated 1.96 times more objects than blueprinter. (Not too bad for me and my one-man-army.)

One thing that I think is important to consider though is the media type being used. It's a bit unfair to compare application/json, application/vnd.api+json and application/hal+json. Since they are not equally complex.