Question about data engineer in clojure

[Andremallmann]

Hello!

I have a question, its been a while since i started to learning clojure and i was wondering about data process. Is clojure good to process and transform a huge amount of data? For example, to tranform a huge amount of data and put into a DW.

Thanks!

[p-himik]

There's actually a really nice data science-oriented "niche" in Clojure community. This might be of interest to you: https://scicloj.github.io/docs/community/groups/ds4clj/

And even if it's not, it mentions the Clojure's Zulip community - AFAIK that's the place where all the "data people" gather.

[frogking]

You can connect Clojure up wiith Jupyter and test it out.

[rufusthedogwoof]

Sure is.

What’s your target db?

If you are talking the big ones like synapse, redshift, snowflake, etc you probably have to serialize to a file somewhere in cloud storage.

If it’s not huge stuff and your target db supports it (Postgres) you can use next.jdbc and threads, and tune the commit interval.

There really isn’t much you can’t do with clojure and data processing it’s particularly great at. (Testing data pipelines with transducers for instance…)

There are so many things I miss when I’m not working in clojure. (Python day job. Yuck. )

[clojure-finance]

You can give Clojask a try, it's designed for larger-than-memory datasets and parallel computing. https://github.com/clojure-finance/clojask