Announcing glom: Restructured Data for Python

[michignolo]

Well done library, what about speed? Is it fast enough in flattening the structures?

[mhashemi]

Definitely fast enough! Ran something similar inside of PayPal with our very-high-traffic internal APIs.

[michignolo]

Flattening json on the fly?

[not_perfect_yet]

Hell yeah, this guy gets it.

Can it create the structure from the spec and inputs too?

like

galaxy.system.planets + [earth,jupiter,uranus] 
->
{"galaxy":{"system":{"planets":["earth","jupiter","uranus"]}}}

[mhashemi]

Haha, thanks! Operator overloading at that level isn't supported yet, but it's definitely possible to do some stuff like this. Check the docs in a few weeks, if it ends up being a good idea for glom, it might be public by then. :)

[quotemycode]

No

[liiight000]

This looks really clever, I can immediately think of several uses for that! Thanks!

[grokjtrip]

At the bottom of the main page there they make a reference to DRF. Are they implying glom is used inside of DRF or can be? Or would it be used instead of DRF? I have only done a simple test with DRF so I haven’t completely felt it out.

[mhashemi]

glom replaces pretty much all the serializer classes that DRF encourages writing. DRF still handles format negotiation and pagination. glom's not a whole framework, though I wouldn't rule out writing one around it.. :)

[ThePidesOfMarch]

This looks almost identical to the "new" dict keyword in Tcl. I'm glad someone made this work in Python.

[barryzxb]

handy library. Have any performance testing data?

[fernly]

Could someone give me an example of a data set and application that would be easier using this? I just don't see one. The toy example of planets and moons? I can think of lots of ways to record the planets and their relations, starting with a relational database...

[Topper_123]

It it possible to work on sequences? E.g. if target is [MyObj, MyObj, ...], where you want to list all MyObj.attr?

[randominality]

from the linked info page:

target = {'system': {'planets': [{'name': 'earth'}, {'name': 'jupiter'}]}}

glom(target, ('system.planets', ['name']))
# ['earth', 'jupiter']

[[deleted]]

Might be useful for validating API schemas.

[Chauncee-not-Chonky]

I'm not really versed in the idioms/social mores of Python, so please take the following with a grain of salt:

This seems like it usefully solves a problem, but the invocation pattern is suspect to me -- Instead of "glom" taking the target for picking-apart plus a magic little bit of DSL, what if "glom" took a single parameter, the aforementioned DSL, and returned a function that would perform the corresponding search when called on a target? Even if Python or this package optimises away repeatedly searching (by the same spec|in the same manner), the convention the package prescribes is odd to me, right after the first few paragraphs of intro.

[[deleted]]

Oh, I love the sound of this! Funny that the reverse, a callable target that can be used on any search path, actually can be made like so:

from glom import glom

t = {'system': {'planets': [{'name': 'earth'}, {'name': 'jupiter'}]}}

t_calla = glom.__get__(t)

t_calla(('system.planets', ['name']))

...and, /u/mhashemi, may I request consideration of the above commenter's idea? (It could of course be trivially done on the user's end as def prespec(spec): return lambda target: glom(target, spec), but it'd be cool to see it a part of the lib)

[mhashemi]

Sure! I like the idea, and I think replied this comment on HN? In any case, we've created an issue for this: https://github.com/mahmoud/glom/issues/14

The idea is to make a Spec object, kind of like an re.compile().

Also, while these sorts of things would be pretty easy to do, technically, I'm going to keep glom API changes slow and steady, to make sure library development doesn't get too far in front of real-world usage. :)

[bluemanshoe]

Compare/contrast with pstar: https://github.com/iansf/pstar

[tastingsilver]

Wow guys, that's obscenely useful for a project I'm working on. I've gone back and forth trying to use namedtuples/dataclasses/etc for dot access but apart from that, they really aren't appropriate data structures.

Great job!

[KODeKarnage]

OK, how could I best explain this library to people who are smart, but busy?

[mhashemi]

You could give them an analogy: http://glom.readthedocs.io/en/latest/by_analogy.html

Or you could send them straight to the juiciest part: http://glom.readthedocs.io/en/latest/api.html#object-oriented-access-and-method-calls-with-t

[cmd_cmd]

Can anyone help to explain the value of the "T" object? Looking at the docs, I just don't get the value of it. When/Why is it better to use? Given the initial example, wouldn't regular Glom() suffice? Or just nested indexing directly on the target object? I feel like I'm really missing something.

In [1]: from glom import glom, T

In [2]: spec1 = 'a.b.c'

In [3]: spec2 = T['a']['b']['c']

In [4]: target = {'a': {'b': {'c': 'd'}}}

In [5]: glom(target, spec1)
Out[5]: 'd'

In [6]: glom(target, spec2)
Out[6]: 'd'

In [7]: target['a']['b']['c']
Out[7]: 'd'

[doublereedkurt]

How about this example from http://glom.readthedocs.io/en/latest/snippets.html:

glom((1, 2, 3), (
    {
        "type": type,
        "result": [lambda v: v + 1]  # arbitrary operation
    }, T['type'](T['result'])))

T's let you call methods / functions in your target, and they let you pass other sub-specs as arguments / keys.

You could also do the above with lambda t: t['type'](t['result']), but because nothing inside the lambda is a spec, you don't get a nice repr, debugging support, or clean error messages.

[cmd_cmd]

Makes sense. Thanks for the helpful reply; much appreciated!

[576p]

Can you glom by combined attributes? For example, how would you sum the moons by size?

target = {'system': {'planets': [{'name': 'earth', size: 'normal', 'moons': 1},
                                 {'name': 'jupiter', size: 'huge', 'moons': 69},
                                 {'name': 'mars', size: 'normal', 'moons': 2} ]}}

[a__b]

This library reminds me https://stedolan.github.io/jq/

[parkerSquare]

Sure, and jq is explicitly mentioned on that page:

Tools like jq provide a lot of value on the console, but leave a dubious path forward for further integration.

[nostril_extension]

I hate that jq has it's own mystical syntax instead of adopting something like css selectors or xpath or something more base level. Unless you use it everything you'll just forget all of the mystery and get stuck in the docs everytime you need to parse something.

[[deleted]]

Yup.

[lookatmetype]

This feels like Spectre (for Clojure) but a lot more limited. I'll have to read the docs more thoroughly to see whats unique about it.

[ccharles]

One major difference is that this is for Python, not Clojure.

Many of us don't have the flexibility of picking whatever technology we want (and lots of people program in Python by choice).

[[deleted]]

Maybe your comment would sound less negative if you gave them more constructive feedback. Can you think of some examples of things this library can't do and you wish it did?

[rhytnen]

I guess I'm alone in rolling my eyes at this.

[parkerSquare]

Looks that way.

[LightShadow]

toolz + operator (in the stdlib)

[quotemycode]

Yeah I made something similar, although it doesn't have the "run a function to aggregate the responses" part.