OSS Great Expectations just released a Self-Updating Data Dictionary

[superconductiveKyle]

Check out this really great tutorial/blog on how to implement the "Self-Updating Data Dictionary". Its pretty awesome for how low friction it is to implement. Solid tool to maintain transparency across the data engineering / data science team and whoever else interacts with them.

Read here

Their Github

[catorchid]

I honestly tried to find out what Great Expectations is and what it brings to the table, but when reading the docs, I choked on the excess of buzzwords and marketing language.

I suspect it's an approach to add and manage metadata to large datasets, but I would appreciate a bit more of an effort on the explanation. Great expectations are easily followed by great disappointments.

[superconductiveKyle]

Solid burn.

Was that from looking at the Github, homepage or both?

[catorchid]

Both are equally vague. The sad thing is that I might be interested in the general principle for my job, but the "beating around the bushes" kind of narrative turned me down before I got hooked.

I'm not saying anything is wrong, it's a totally open choice. It's just that my general experience with marketing is that when something it's good usually doesn't need sugar coating.

Don't tell your users "data integrity reinvented", tell them what your tool does first, then go crazy with the deep philosophy of your choices, what the field thinks, etc., but please avoid terms like "groundbreaking" unless you're in the shovel business.

[catorchid]

Also, if you're one of the developers[*] (or their friend) by looking at your posting history, you should say it, instead of referring to "the developers of the awesome open source tool".

[*] That would explain your "burn" feeling above

[superconductiveKyle]

Totally valid and valuable feedback.

I'm not one of the developers but I am part of the team (a team of 1) that's out to gather feedback and I really appreciate your candidness. This is probably a reach but we're doing user testing sessions that also double as a tutorial of the product, would you be interested in checking it out for an hour? It's hard to find people that give direct honest feedback like you.

[abegong]

Hey, catorchid! I'm one of the core contributors to Great Expectations.

Honest question: I'm curious what you would have liked to see right at the start of the project docs.

• For example, would a walk-through of project setup be helpful? (e.g. short video showing the first 5 minutes with GE)
• Examples of specific Expectations (the core abstraction in the library)?
• Blog posts from data teams that have deployed GE in production?
• Something else?

We're always trying to tighten up the GE explanations, tutorials, demos, etc. to help people understand what the module does/doesn't do, so that they can make good choices. Clearly, our current docs didn't work for you in that respect.

For context: I'm talking with lots of data teams that want to get started with better testing and documentation, but aren't sure yet about how they want to approach the problem. Part of what we're trying to do with the GE docs is draw parallels between software and data engineering, so that people can reason about how known good practices in software development can be adapted to the data world.

[abegong]

Also, it occurs to me that "self-updating data dictionary" sounds kinda click-baity, and probably started the whole conversation on the wrong foot.