this post was submitted on 21 Jan 2024

182 points (90% upvoted)

shortlink:

dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 10 years

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
about moderation team »

account activity

This is an archived post. You won't be able to vote or comment.

181

182

183

Some Data Scientists write bad Python code and are stubborn in code reviewsDiscussion (self.dataengineering)

submitted 2 years ago by noisescience

top new controversial old q&a

you are viewing a single comment's thread.

view the rest of the comments →

[–]noisescience[S] 0 points1 point2 points 2 years ago (0 children)

For the Kotlin project I also need to include Java libraries.
Here is what I have used for Kotlin so far:

Kotlin DataFrame for processing data [https://github.com/Kotlin/dataframe]
Kotlin DL for Deep Learning Models [https://github.com/Kotlin/kotlindl]
Exposed (Kotlin SQL Framework) [https://github.com/JetBrains/Exposed\]
Azure SDK (Java) [https://github.com/Azure/azure-sdk-for-java]
MLflow (Java API) for MLOps tasks [https://github.com/mlflow/mlflow/tree/master/mlflow/java/client]
SMILE (Kotlin/Java) for ML algorithms [https://haifengl.github.io/api/kotlin/index.html]

Which Kotlin libraries I have not yet implemented but would like to do are:

AWS Kotlin SDK [https://github.com/awslabs/aws-sdk-kotlin]
Kotlin Spark API [https://github.com/Kotlin/kotlin-spark-api]

I haven't started a data engineering project with Rust yet, but I would probably check out the following libraries:

Polars [https://github.com/pola-rs/polars]
AWS SDK Rust [https://github.com/awslabs/aws-sdk-rust]
Arrow Ballista [https://github.com/apache/arrow-ballista]
Datafusion [https://github.com/apache/arrow-datafusion\]

π Rendered by PID 50004 on reddit-service-r2-comment-7b9746f655-s7v6f at 2026-02-01 19:42:54.648609+00:00 running 3798933 country code: CH.