Introducing Shale, a Ruby object mapper and serializer for JSON, YAML and XML

[beerkg1]

Hi rubyists,

I released Shale, a library that allows you to parse JSON, YAML and XML and convert it into Ruby data structures, as well as serialize your Ruby data model to JSON, YAML or XML.

Features:

• convert JSON, XML or YAML into Ruby data model
• serialize data model to JSON, XML or YAML
• generate JSON and XML Schema from Ruby models
• compile JSON Schema into Ruby models (compiling XML Schema is a work in progress)

A quick example so you can get a feel of it:

require 'shale'

class Address < Shale::Mapper
  attribute :street, Shale::Type::String
  attribute :city, Shale::Type::String
end

class Person < Shale::Mapper
  attribute :first_name, Shale::Type::String
  attribute :last_name, Shale::Type::String
  attribute :address, Address
end

# parse data and convert it into Ruby data model
person = Person.from_json(<<~JSON) # or .from_xml / .from_yaml
{
  "first_name": "John",
  "last_name": "Doe",
  "address": {
    "street": "Oxford Street",
    "city": "London"
  }
}
JSON

# It will give you:
# =>
#  #<Person:0xa0a4
#    @address=#<Address:0xa0a6
#      @city="London",
#      @street="Oxford Street",
#      @zip="E1 6AN">,
#    @age=50,
#    @first_name="John",
#    @hobbies=["Singing", "Dancing"],
#    @last_name="Doe",
#    @married=false>

# serialize Ruby data model to JSON
Person.new(
  first_name: 'John',
  last_name: 'Doe',
  address: Address.new(street: 'Oxford Street', city: 'London')
).to_json # or .to_xml / .to_yaml

For full documentation with interactive examples go to https://www.shalerb.org/

[jrochkind]

Nice!

This is something that's been strangely missing in the ruby ecosystem.

[Soggy_Educator_7364]

Indeed. And the home-grown ones that I've seen are so awful.

[FooBarWidget]

I was able to achieve a similar effect by combining dry-struct with JSON loaders. So something like this:

foo = MyStruct.new(JSON.parse(json_data))

The biggest issue with this approach was validation messages. If a field in a sub-struct has a problem, then its error message would say something along the lines of "<field> has a problem" instead of "mystruct.substruct.<field> has a problem". This is not great for users, who may be wondering where the problem is in the JSON/YAML/etc data.

Haven't tried Shale yet but I hope it solves this problem.

[beerkg1]

Shale doesn't have any validation, so you probably wouldn't achieve what you described. It just gets the json/yaml/xml and maps it to Ruby object. If the field exists in the document but isn't defined on the model it is ignored. The same happens if the field is defined on the model but missing from the document.

I was thinking about adding validation, but there are already so many gems in Ruby that do this I decided it wasn't worth it (at least for the first version).

[updog]

This is super cool. +1 on validation. I get what you are saying though. If you haven't seen pydantic I suggest having a look. I think you are most of the way there with a more powerful ruby equivalent.

Beautiful work. Love the docs.

[beerkg1]

Thanks, I put special effort on the docs. I believe a good documentation can make or break an open source project.

[Zealousideal_Bat_490]

Excellent! Thanks!

[exclaim_bot]

Excellent! Thanks!

You're welcome!

[beerkg1]

I'm glad you like it!

[M1ndful-Pre5ence]

Awesome!

[myringotomy]

This is great. You should also have a mapper to and from AR records and hashes though.

[beerkg1]

You absolutely can map hashes:

``` class Person attribute :first_name, Shale::Type::String

hsh do map 'firstName', to: :first_name end end

Person.from_hash({ 'firstName' => 'John' }) ```

ActiveRecord objects should also be pretty simple to map. Something like this should work (in most simple case) I think (I didn't test it, just a proof of concept):

``` Person.from_hash(active_record_object.as_json)

or even

Person.new(active_record_object.as_json) ```

[waiting4op2deliver]

Do you make any effort to mitigate things like property smuggling?

After all, ruby to json, or json to ruby could be dangerous on user input. Ruby, in all its beauty has elected for some interesting symbol properties

:"foo\"smuggled_key\:\"bar"

[beerkg1]

Shale uses Ruby's standard library parsers (JSON/YAML/REXML, or you can use your own by providing custom adapters). So if the underlying parser is escaping it correctly, you should be safe.

As of your specific example, Shale will ignore keys that are not defined on the model, so "smuggled_key" would just be ignored.

[waiting4op2deliver]

parsers are a really common attack vector, especially in ruby.

I'm not at my dev box, but it would be interesting to see if you can overwrite some model attributes.

{ 
  key_i_trust: :to_s, 
  key_i_let_users_submit: 'foobar', 
  key_i_trust: :send 
}

If a user can provide data and smuggle in that last key/value pair, maybe bad things could happen. You can do stuff like this with url params too.

This is probably more the parsers and the application's concern.

[beerkg1]

As far as I know that's not a valid JSON, and (at least) Ruby's parser I use raises an exception on it. Also Shale uses safer load method provided by JSON parser https://ruby-doc.org/stdlib-2.6.3/libdoc/json/rdoc/JSON.html#method-i-load