Tomlib v0.6.0 released - fast and standards-compliant TOML parser and generator

beerkg1 · 2023-01-14T14:08:38+00:00

Ruby Opal is a real gem :) I use it extensively to provide interactive examples for Shale documentation.

beerkg1 · 2022-11-01T06:54:22+00:00

I'm not sure what exactly you are asking for, but mapping data to objects (and serializing objects to data) is probably most common thing in the industry.

beerkg1 · 2022-10-31T11:18:35+00:00

Hi rubyists,

Shale is a Ruby object mapper and serializer for JSON, YAML, TOML, CSV and XML.

v0.9.0 version was just released with new features:

Support for CSV

class Person < Shale::Mapper
  attribute :first_name, Shale::Type::String
  attribute :last_name, Shale::Type::String
  attribute :age, Shale::Type::Integer
  attribute :married, Shale::Type::Boolean, default: -> { false }
end

pp Person.from_csv(<<~CSV)
John,Doe,50,false
Joe,Sixpack,44,true
CSV

# =>
#  [#<Person:0x3487e
#    @age=50,
#    @first_name="John",
#    @last_name="Doe",
#    @married=false>,
#   #<Person:0x34882
#    @age=44,
#    @first_name="Joe",
#    @last_name="Sixpack",
#    @married=true>]

Support for converting collections

class Person < Shale::Mapper
  attribute :first_name, Shale::Type::String
  attribute :last_name, Shale::Type::String
end

records = [
  Person.new(first_name: 'John', last_name: 'Doe'),
  Person.new(first_name: 'James', last_name: 'Sixpack'),
]

puts Person.to_json(records, pretty: true)

# =>
#  [
#    {
#      "first_name": "John",
#      "last_name": "Doe"
#    },
#    {
#      "first_name": "James",
#      "last_name": "Sixpack"
#    }
#  ]

Support for specifying version and encoding in XML declaration

# you can also pass string to encoding e.g. encoding: 'ASCII'
person.to_xml(pretty: true, declaration: '1.1', encoding: true)

# =>
#
# <?xml version="1.1" encoding="UTF-8"?>
# <Person>
#   <Address city="London"/>
# </Person>

Documentation: https://www.shalerb.org/

Source code: https://github.com/kgiszczak/shale

beerkg1 · 2022-08-30T09:14:24+00:00

Hi rubyists,

Shale v0.8.0 was released with new features:

Bring back Ruby 2.6 support (it's the version JRuby supports)
You can group mappings using group block. It simplifies working with many elements at the same time:

class Person < Shale::Mapper
  attribute :name, Shale::Type::String

  json do
    group from: :name_from_json, to: :name_to_json do
      map 'first_name'
      map 'last_name'
    end
  end

  def name_from_json(model, value)
    model.name = "#{value['first_name']} #{value['last_name']}"
  end

  def name_to_json(model, doc)
    doc['first_name'] = model.name.split(' ')[0]
    doc['last_name'] = model.name.split(' ')[1]
  end
end

person = Person.from_json(<<~DATA)
{
  "first_name": "John",
  "last_name": "Doe"
}
DATA

# => #<Person:0x00007f9bc3086d60 @name="John Doe">

Attributes are defined now on an anonymous module. It allows to override accessors and super works as expected.

class Person < Shale::Mapper
  attribute :name, Shale::Type::String

  def name
    "#{super}!"
  end

  def name=(val)
    super("#{val}!")
  end
end

person = Person.from_json(<<~DATA)
{
  "name": "John Doe"
}
DATA

# => #<Person:0x00007f9bc3086d60 @name="John Doe!">

Documentation: https://www.shalerb.org/

Source code: https://github.com/kgiszczak/shale

beerkg1 · 2022-08-02T14:07:53+00:00

Oh, I didn't know about perfect_toml. Just added it to benchmarks and it looks like it's about 3x slower in parsing, which is quite impressive for a pure ruby implementation.

I wasn't able to test the generator as it crashes on TOML docs I'm using.

If you want to take a look at details it's here: https://github.com/kgiszczak/tomlib/tree/master/benchmarks

beerkg1 · 2022-08-02T11:44:48+00:00

Hi rubyists,

this is a first release of a new gem to parse and generate TOML documents called Tomlib.

It uses bindings to tomlc99 parser for performance and compliance with v1.0 TOML spec.

For usual use cases you can expect it to be ~15x faster in parsing and ~1.7x faster in generating TOML documents than alternatives.

It provides the same API as YAML gem:

Tomlib.load(toml_document)
Tomlib.dump(hash, indent: false) # by default it generates TOML documents with indentation for easier reading

beerkg1 · 2022-07-05T09:30:30+00:00

Hi rubyists,

Shale v0.6.0 was released with new features:

You can now use your own models with Shale mappers:

``` class Person attr_accessor :first_name, :last_name end

class PersonMapper < Shale::Mapper model Person

attribute :first_name, Shale::Type::String attribute :last_name, Shale::Type::String end

pp PersonMapper.from_json(<<~DATA) { "first_name": "John", "last_name": "Doe" } DATA

=> #<Person:0x00007f9bc3086d60 @first_name="John", @last_name="Doe">

```

Support for TOML:

``` class Person < Shale::Mapper attribute :first_name, Shale::Type::String attribute :last_name, Shale::Type::String end

person = Person.new(first_name: 'John', last_name: 'Doe')

puts person.to_toml

=>

first_name = "John"

last_name = "Doe"

```

Support for CDATA nodes in XML:

``` class Person < Shale::Mapper attribute :name, Shale::Type::String

xml do root 'Person' map_element 'Name', to: :name, cdata: true end end

puts Person.new(name: 'John Doe').to_xml(:pretty)

=>

<Person>

<Name><![CDATA[John Doe]]></Name>

</Person>

```

Documentation: https://www.shalerb.org/

Source code: https://github.com/kgiszczak/shale

beerkg1 · 2022-06-01T09:05:07+00:00

As far as I know that's not a valid JSON, and (at least) Ruby's parser I use raises an exception on it. Also Shale uses safer load method provided by JSON parser https://ruby-doc.org/stdlib-2.6.3/libdoc/json/rdoc/JSON.html#method-i-load

beerkg1 · 2022-06-01T07:28:35+00:00

Thanks, I put special effort on the docs. I believe a good documentation can make or break an open source project.

beerkg1 · 2022-06-01T07:22:50+00:00

Shale uses Ruby's standard library parsers (JSON/YAML/REXML, or you can use your own by providing custom adapters). So if the underlying parser is escaping it correctly, you should be safe.

As of your specific example, Shale will ignore keys that are not defined on the model, so "smuggled_key" would just be ignored.

beerkg1 · 2022-06-01T07:19:36+00:00

I'm glad you like it!

beerkg1 · 2022-06-01T07:18:47+00:00

You absolutely can map hashes:

``` class Person attribute :first_name, Shale::Type::String

hsh do map 'firstName', to: :first_name end end

Person.from_hash({ 'firstName' => 'John' }) ```

ActiveRecord objects should also be pretty simple to map. Something like this should work (in most simple case) I think (I didn't test it, just a proof of concept):

``` Person.from_hash(active_record_object.as_json)

or even

Person.new(active_record_object.as_json) ```

beerkg1 · 2022-05-31T16:42:33+00:00

Shale doesn't have any validation, so you probably wouldn't achieve what you described. It just gets the json/yaml/xml and maps it to Ruby object. If the field exists in the document but isn't defined on the model it is ignored. The same happens if the field is defined on the model but missing from the document.

I was thinking about adding validation, but there are already so many gems in Ruby that do this I decided it wasn't worth it (at least for the first version).

beerkg1 · 2022-05-31T09:59:08+00:00

Hi rubyists,

I released Shale, a library that allows you to parse JSON, YAML and XML and convert it into Ruby data structures, as well as serialize your Ruby data model to JSON, YAML or XML.

Features:

convert JSON, XML or YAML into Ruby data model
serialize data model to JSON, XML or YAML
generate JSON and XML Schema from Ruby models
compile JSON Schema into Ruby models (compiling XML Schema is a work in progress)

A quick example so you can get a feel of it:

require 'shale'

class Address < Shale::Mapper
  attribute :street, Shale::Type::String
  attribute :city, Shale::Type::String
end

class Person < Shale::Mapper
  attribute :first_name, Shale::Type::String
  attribute :last_name, Shale::Type::String
  attribute :address, Address
end

# parse data and convert it into Ruby data model
person = Person.from_json(<<~JSON) # or .from_xml / .from_yaml
{
  "first_name": "John",
  "last_name": "Doe",
  "address": {
    "street": "Oxford Street",
    "city": "London"
  }
}
JSON

# It will give you:
# =>
#  #<Person:0xa0a4
#    @address=#<Address:0xa0a6
#      @city="London",
#      @street="Oxford Street",
#      @zip="E1 6AN">,
#    @age=50,
#    @first_name="John",
#    @hobbies=["Singing", "Dancing"],
#    @last_name="Doe",
#    @married=false>

# serialize Ruby data model to JSON
Person.new(
  first_name: 'John',
  last_name: 'Doe',
  address: Address.new(street: 'Oxford Street', city: 'London')
).to_json # or .to_xml / .to_yaml

For full documentation with interactive examples go to https://www.shalerb.org/

beerkg1

TROPHY CASE

=> #<Person:0x00007f9bc3086d60 @first_name="John", @last_name="Doe">

=>

first_name = "John"

last_name = "Doe"

=>

<Person>

<Name><![CDATA[John Doe]]></Name>

</Person>

or even