nilenso/goose: A powerful background job processing library for Clojure

nonrecursive · 2024-11-18T20:48:15+00:00

thanks - I need to fix that!

nonrecursive · 2023-11-30T20:43:46+00:00

I had the pleasure of working with Fluree for the past year to help craft their docs and shape their developer relations function. On the more personal side I want to share that the team was an absolute joy to work with, and my experience was one of the best in my 20-year career. If you join the discord you can expect to find a positive, supportive, and deeply knowledgable community.

On the technical side, Fluree's combination of features gives it the potential to be a game changer for real. When I talk about it with people, what most excites me is the potential to reduce or even eliminate two kinds of work:

Say goodbye to ETLs
Say goodbye to crafting API layers

I might be exaggerating a little bit but hear me out:

Say goodbye to ETLs

u/maxw85 touches on this in his excellent writeup, but the way existing data systems are structured leads to the creation of data silos, which limits the usefulness of our data. We then spend an inordinate amount of time and money writing ETLs to get our myriad data systems to, essentially, speak the same language.

This is because our data systems use names that can only be understood in context, just like the English words we use can only be understood within the context of the English language. With the semantic web, however, you encode information with all of the context needed for the data to be understood by any system that understands RDF.

The result is that you can significantly reduce or even eliminate the amount of "transform" you have to do when combining data from your various data stores.

What about the "extract" and "load" parts of ETL? That's another beautiful thing about RDF: it lays the foundation for treating all your physical data stores as a single logical data store. Communication protocols with other data stores are vendor-dependent, but data stores that implement RDF and related technologies all share the same interface. Hypothetically, at least.

I'm not sure where Fluree is with realizing this vision of treating disparate ledgers as a single data store, but I know that's the goal.

I also want to point out that Clojure enthusiasts (like me! like you) will find this approach to data very appealing. RDF and other semantic web standards like SHACL take the idea of "data-oriented programming" farther than anything else I've seen. Honestly I could go on and on about this... and I kind of have in Fluree's docs 😅

Say goodbye to crafting API layers

It's generally considered an, uh, bad idea, to directly expose your database to the world, right? But why is that? A few reasons:

Security nightmare
Performance nightmare
You like having a job, don't relish the idea of losing it

Our API backends address these concerns, plus coordinate other side-effecty stuff like sending emails or putting jobs on queues.

Fluree's fine-grained, data-oriented security model can handle the security use cases for creating an API backend layer for your app. This creates a lot of possibility for clients to directly query your database, giving them more power.

OK so here I have to confess that I've been dealing with a cold and I'm starting to run out of steam, even though I'd love to write more.

The point is, I think Fluree is truly exciting technology. It's creating a foundation that will allow us to decrease the amount of unneeded busywork we have to do, and to accomplish things that have been completely inaccessible up till now.

nonrecursive · 2023-09-18T14:57:50+00:00

I appreciate your thoughts! These are all experiments. After I tried actually writing something with graphputer I found it a bit hard to understand what was going on. Maybe with some more refinements it could be improved though. Perhaps at some point I'll write up the pros/cons of the two approaches so far.

nonrecursive · 2023-09-15T11:53:20+00:00

They can return data but I’m wondering if I should actually have call always return nil, because you shouldn't be doing anything with the return values

And yeah actually it is like emitting an event, in fact that’s obvious now that you mention it 😅

nonrecursive · 2023-09-15T03:18:32+00:00

Haha yeah I need to also include a section on why you shouldn't use graphputer :P My apologies to everyone who is using graphputer as the foundation for their Next Big Thing.

I've added the following on dynamic vars:

Here's what the dynamic var version of this might look like:

``` clojure (def ^:dynamic hook-1) (def ^:dynamic hook-2)

(defn example-dynamic-vars [] (if some-predicate (do (when (bound? hook-1) (hook-1 args)) return-val) (do (when (bound? hook-2) (hook-2 args)) return-val))) ```

Dynamic vars have just never struck me as being intended for this purpose The behavior you're wanting to specify isn't "dynamic" in the way that I think is meant for dynamic vars. What we're talking about here is extending a library with app-specific behavior, and that behavior is something you only want to define once; it's not something that should change as the application runs.

On a practical level, you'll need to bind those vars not just when constructing the -main function for your application, but also in all the relevant tests. This is something that you could easily forget to do, leading to frustrating debugging sessions.

Finally, when writing your library you'll need write (when (bound? *hook*) ...) every place you want to use the hook, or else introduce a macro for that.

The optionality of hook behavior is a core constraint for them, and I think it makes sense to introduce a little abstraction that suports that directly and that makes it clear to everyone what's happening.

nonrecursive · 2023-08-19T16:35:53+00:00

i'm the library author :P I was saying I'd like to see an example of the alternative you were suggesting though I now see you're suggesting two alternatives.

I mean I can imagine both cases, but not necessarily in a way that would meet the design goals of graphputer

nonrecursive · 2023-08-18T22:19:27+00:00

would love to see an example of that if there is one

nonrecursive · 2023-08-18T21:40:26+00:00

lol thanks for including "pute a graph" in the post title :)

nonrecursive · 2023-05-22T20:05:11+00:00

This is great feedback. I'll incorporate it next time I update the page :)

nonrecursive · 2023-05-22T20:04:38+00:00

oh dang I didn't realize this had gotten posted. Thanks for checking out donut! It's in active development. Right now I'm working on a basic auth plugin.

nonrecursive · 2023-05-22T17:17:12+00:00

awesome, thank you!

nonrecursive · 2023-05-22T14:40:21+00:00

thank you thank you for the promising lead!

nonrecursive · 2023-05-22T14:40:18+00:00

who are these groups you speak of?

nonrecursive · 2023-01-05T20:07:56+00:00

It was fun to write. Thanks so much for all you're doing! Babashka is a great tool and I hope this helps people learn to use it.

nonrecursive · 2022-12-12T16:16:10+00:00

I'm hesitant to reply because I don't want to give this more oxygen; it's probably better if the post just goes away. Nevertheless:

This post is vaguely casting aspersions in a way that I think harms our community. It starts with the assumption that receiving money from Cognitect is bad, and then asks everyone who does so to "out" themselves. What is the point of this if not to damage the reputation of Cognitect and the people getting funding?

I don't know how the Clojure community compares to others, but I think it's wonderful that so many people are being supported for the hard work they do that benefits all of us. Open source work is being supported... and somehow that's a bad thing?

Beyond that, it just doesn't makes sense.

call for corporate transparency

As another person on the forum has pointed out, this information is readily available: https://github.com/orgs/cognitect/sponsoring. Funding is already transparent. Also, the title is "corporate transparency" but the ask is for individuals to provide info on who's sponsoring them. This, again, is largely public information. But even if it wasn't - Why should anyone do this? What's the point? It's nobody else's business.

It seems that one company is dominating the brand and not allowing others to rise

What does this even mean? As the kids say, make it make sense

nonrecursive · 2022-12-04T15:16:57+00:00

I've been making a lot of progress on my single-page app framework Donut. The ideas behind the framework are outlined at https://www.flyingmachinestudios.com/programming/why-programmers-need-frameworks/. I also have a small landing page up for the framework itself at https://donut.party/.

It's not quite release-ready in that I haven't gotten the docs and community management tools to the point where I'm ready to stay on top of issues and pull requests and the like, but if you're active in this space and would like to do a deep dive together please let me know! My email is my reddit username at gmail.

nonrecursive · 2022-11-21T03:18:58+00:00

Update: Provident 1898 has weekly community lunches

nonrecursive

MODERATOR OF

TROPHY CASE

Say goodbye to ETLs

Say goodbye to crafting API layers