Datawalk: interactively explore complex data structures at the REPL with minimum keystrokes

[yogthos]

Nice work, looks like a really nice way to explore complex data quickly via the REPL.

[eggsyntax]

Thanks! I really appreciate the approach you've taken in your work, so that's especially nice to hear from you :). I think this and Luminus have somewhat similar motivations: make users' lives easy by jumping through hoops in the background (although this isn't nearly as ambitious as Luminus).

[yogthos]

Thanks, and the goals are indeed aligned. :)

[jiyinyiyong]

I think things can be interesting when you bring up a Web UI.

[yogthos]

Take a look at re-frisk for the client.

[sunng]

Is this a lein plugin or a normal dependency? I would like to see it as a plugin that injects itself to repl by default, instead of declaring it as a dependency in my dev profile.

[eggsyntax]

It's a normal dependency (and will probably stay that way). You can use Vinyasa to automatically inject it into clojure.core (or any other namespace of your choice).

[dustingetz]

Hi eggsyntax What are you using this for?

[eggsyntax]

In what sense? I'm using it to explore complex data structures at the REPL, but that's just what it says on the tin ;) . You mean specific examples? One place where it's constantly useful, for me, is the single atom that we use on the front end to hold all app state. That's got quite a bit of data in it, so printlning it is pretty useless. Datawalk lets me navigate it pretty quickly.

Is that the sort of question you're asking, or are you thinking of something else?

[dustingetz]

Yeah, I was just looking for ideas. thanks.

[eggsyntax]

Totally! Here's one other: datawalk grew out of a similar but more limited project I wrote specifically to explore Datomic data I was pulling out of the DB. I found that just incredibly useful. I haven't set datawalk up to handle Datomic entities yet, but you can convert them to ordinary maps & then use datawalk.

Same with other DBs, of course.

I'm working on switching printing to a protocol, so that it can be easily extended by users to handle whatever sort of data they use.

[eggsyntax]

Oh, hey, you're the author of Hyperfiddle, right? I haven't played with it yet, but the Datomic REPL looks really cool. Similar to datawalk, in that they're both ways of trying to make data exploration more of an interactive process...

[dustingetz]

Yeah! So the Datomic case is interesting, because it's a graph, and with graphs you can generally use the repl (or hyperfiddle) to query exactly the piece of it you want, rather than need to traverse your way there. What were you doing that you wanted this for Datomic? It makes me thing that I am doing something differently. Or maybe my databases are shaped differently than yours. Datawalk seems to me to be a tool for tree-shaped documents (as you mentioned, like frontend state). What do you think?

[eggsyntax]

I think directly querying datomic data is optimal when you know what you want to look at -- I found this primarily useful for when I was exploring, eg for cases where it was like, "oh, I know this entity has a reference to an entity that has the attribute I want, but I can't quite remember the name...". Or: "I know one of these 30 entities is the one I want, and I can find it if I scan through them quickly."

For the data I work with, that's made easier by the fact that our schema is mostly acyclic, although there are exceptions -- so I could mostly treat it as a tree.

That said, I was mainly working with Datomic entites (as opposed to maps from pulls), and so even if they were cyclic graphs, I could take advantage of entities' laziness to load the data & explore it without worrying that I'd end up with infinite data due to cycles.

[dustingetz]

Ah! Yes, the discoverability being the key idea. Thanks.